Emulating expensive decision choices with
application to computer models of complex physical
systems
Jonathan A Cumming and Michael Goldstein
Department of Mathematical Sciences, Durham University
July 2012
1 / 33
Outline
1
The decision problem
2
Role of the computer model
3
Methodology
4
Comments
2 / 33
The decision problem
The decision problem
3 / 33
The decision problem
Motivation
Well test analysis
Well test analysis is a technique used to reveal information about the
geometry and geological features of an underground hydrocarbon
reservoir
In practice, a well test involves producing fluid from a reservoir at a known
and controlled rate and observing the change in pressure during and
after this period of production
4 / 33
The decision problem
Motivation
0
Rate
400 800
3800
Pressure
4400
5000
Well test analysis
0
10
20
30
40
50
60
70
Time
4 / 33
The decision problem
Motivation
Well test analysis
4 / 33
The decision problem
Motivation
Well test analysis
Well test analysis is a technique used to reveal information about the
geometry and geological features of an underground hydrocarbon
reservoir
In practice, a well test involves producing fluid from a reservoir at a known
and controlled rate and observing the change in pressure during and
after this period of production
Pressure and rates are linked by a convolution relationship with a third
function, usually called the deconvolved derivative
The behaviour of this derivative function follows common patterns which
are attributed to particular reservoir characteristics
4 / 33
The decision problem
Motivation
−1.0
−1.5
Deconvoled Derivative
−0.5
Well test analysis
Infinite
Fault
Channel
Closed
−2
−1
0
1
2
log10(T)
4 / 33
The decision problem
Motivation
Well test analysis
Well test analysis is a technique used to reveal information about the
geometry and geological features of an underground hydrocarbon
reservoir
In practice, a well test involves producing fluid from a reservoir at a known
and controlled rate and observing the change in pressure during and
after this period of production
Pressure and rates are linked by a convolution relationship with a third
function, usually called the deconvolved derivative
The behaviour of this derivative function follows common patterns which
are attributed to particular reservoir characteristics
The object of the well test is to make an inference about the class of the
reservoir
4 / 33
The decision problem
Motivation
The Problem
We have a reservoir of interest, which can belong to one of four different
classes
We have to make a decision about how to perform the well test – what
rate to produce, how long for, how long to observe pressure
We then enact that decision (under error) and observe the pressures and
rates, and estimate the deconvolved derivative function
We then make an inference about the class of the reservoir using the
observations
On the basis of this inference, we then take a further action about
whether to develop or abandon this reservoir
Depending on the decisions we made and the nature of the reservoir, we
will receive some reward (or penalty)
We have a computer model which simulates the results of the well test
Are there any generally good well test regimes, and if so what are they?
5 / 33
The decision problem
General Problem Description
General Problem
We are interested in some complex physical system
Denote the portion of the system that is observable (and interesting) as y,
and let the state of the complex system be θ
We want to make a decision, δ, that may or may not affect the system
We enact that decision, and take observations of the system z
Using what we learn from z, we take a further decision (or action) α
Depending on δ, α, and the system θ, we receive a reward expressed as
a utility U [δ, α, θ]
We want to find good choices of δ
6 / 33
The decision problem
Decision Tree
The problem as a decision tree
U [δ, α, θ]
θ
System
P[
, z]
θ|δ
α
Action
z
P[
]
z|δ
Observation
δ
Decision
7 / 33
The decision problem
Decision Tree
Solving the tree
To solve the decision tree (and the decision problem) we apply backward
induction:
1
2
3
4
Find U [δ, α, z] = E [U [δ, α, θ]] with respect to P [θ|δ, z]
Find U [δ, z] = U [δ, α∗ , z] where the action α∗ maximises U [δ, α, z]
Find U [δ] = E [U [δ, z]] with respect to P [z|δ]
Find the decision δ∗ which maximises U [δ]
Comments:
1
2
3
The distribution P [θ|δ, z] required for step (1) is the result of a calibration
The maximisation operations can be handled in a variety of ways depending
on whether the variables are discrete, continuous, and high- or
low-dimensioinal
If we can’t get the distributions we need, we can always approximate the
expectations by Monte Carlo
8 / 33
Role of the computer model
Role of the computer model
9 / 33
Role of the computer model
The computer model
The computer model f mimics the behaviour of the system y
By adding observational error and model discrepancy we can use it to
mimic the observations z
The simulator takes two sets of parameters
1
2
x, the input parameters which encode the state of the system
d, the input parameters which represent the decision in the model
We can explore the behaviour of f (or z) over d and x (corresponding to δ,
θ) by making many evaluations of the model
If f is expensive or if δ has a substantial effect on the behaviour of the
system, we may choose to emulate it at this stage.
Alternatively, we could push our model runs through the decision process
and explore U [·] directly
At the end, we get values of the complex utility function U [δ, α, θ] over δ,
α, and θ
10 / 33
Role of the computer model
Influence Diagram Mk. II - Now With More Arcs!
Real World
Observations
Real Decision
θ
δ
y
z
α
U [δ, α, θ]
f (d, x)
z(d, x)
a
U [δ, α, θ]
U [δ]
U∗
d
x
Computer Model
Synthetic Observations
Decision Model
11 / 33
Role of the computer model
Using the computer model
The main use of the computer model will be provide information to
perform the two key calculations in the backward induction
Take expectations over θ
We need P [θ|δ, z] – which is effectively a calibration – and can be expressed
as
P [θ|δ, z] ∝ P [z|δ, θ] P [θ] ,
(1)
We assume we have a prior P [θ], and we can use the computer model to
generate samples of P [z|δ, θ] to build an approximate empirical distribution
Hence we can use the computer model to help us approximate these
probabilities which we can use to find the required expectation
Take expectations over z
Here we need P [z|δ] to find the expectation
We could repeat the above approach
Or we can use the computer model to generate many samples of z|δ, and
throw Monte Carlo at it
12 / 33
Methodology
Methodology
13 / 33
Methodology
Key steps
Our methodology is composed of the following stages:
1
Design
2
Evaluation of the computer model
3
Calibration
4
Generation of utilities
5
Solution of the decision tree
6
Emulation of the utility
7
Optimisation of the utility
8
Validation of the result
14 / 33
Methodology
Key steps
Our methodology is composed of the following stages:
1
Design — we construct two designs of (δ, θ) pairs: the first to explore
the decision space of δ, and the second to provide information required
for the calibration step.
2
Evaluation of the computer model
3
Calibration
4
Generation of utilities
5
Solution of the decision tree
6
Emulation of the utility
7
Optimisation of the utility
8
Validation of the result
14 / 33
Methodology
Key steps
Our methodology is composed of the following stages:
1
Design
2
Evaluation of the computer model — for every (δ, θ) point in the
design, we generate multiple consistent (d, x) pairs, each of which is
evaluated by the computer model, and then combined with noise to
generate and ensemble of synthetic observations.
3
Calibration
4
Generation of utilities
5
Solution of the decision tree
6
Emulation of the utility
7
Optimisation of the utility
8
Validation of the result
14 / 33
Methodology
Key steps
Our methodology is composed of the following stages:
1
Design
2
Evaluation of the computer model
3
Calibration — using information from the calibration runs, for each of our
synthetic observations z(d, x) we assess P [θ|δ, z(d, x)] for each possible
state θ.
4
Generation of utilities
5
Solution of the decision tree
6
Emulation of the utility
7
Optimisation of the utility
8
Validation of the result
14 / 33
Methodology
Key steps
Our methodology is composed of the following stages:
1
Design
2
Evaluation of the computer model
3
Calibration
4
Generation of utilities — for each of our synthetic observations z(d, x)
we calculate the utility U [δ, α, θ] for all possible actions α.
5
Solution of the decision tree
6
Emulation of the utility
7
Optimisation of the utility
8
Validation of the result
14 / 33
Methodology
Key steps
Our methodology is composed of the following stages:
1
Design
2
Evaluation of the computer model
3
Calibration
4
Generation of utilities
5
Solution of the decision tree — by using the distributions and utility
values obtained above, we can now calculate a single U [δ] for every δ in
our original design by collapsing the decision tree.
6
Emulation of the utility
7
Optimisation of the utility
8
Validation of the result
14 / 33
Methodology
Key steps
Our methodology is composed of the following stages:
1
Design
2
Evaluation of the computer model
3
Calibration
4
Generation of utilities
5
Solution of the decision tree
6
Emulation of the utility — given the collection of values of U [δ] for the
values of δ in our original design, we construct an emulator to model the
behaviour of U [·] over the entire decision space.
7
Optimisation of the utility
8
Validation of the result
14 / 33
Methodology
Key steps
Our methodology is composed of the following stages:
1
Design
2
Evaluation of the computer model
3
Calibration
4
Generation of utilities
5
Solution of the decision tree
6
Emulation of the utility
7
8
Optimisation of the utility — we maximise the emulator of U [δ] and
identify regions of good decisions, and possible a unique optimal decision
δ∗ .
Validation of the result
14 / 33
Methodology
Key steps
Our methodology is composed of the following stages:
1
Design
2
Evaluation of the computer model
3
Calibration
4
Generation of utilities
5
Solution of the decision tree
6
Emulation of the utility
7
Optimisation of the utility
8
Validation of the result — assess the quality of the final solution to
determine whether the result is satisfactory or whether additional
evaluations should be performed.
14 / 33
Methodology
Design
We can design for and evaluate the computer model as usual, however
we must be mindful of how we construct the design
We need two sets of design points:
1
2
Points to be used for space exploration to be used to build the final emulator
— no/minimal replication, primarily space-filling
Points to be used to understand P [z|δ, θ] — requires replicates for the same
δ values
How we allocate our runs to each of these two groups under a fixed
budget is an interesting problem
15 / 33
Methodology
Results: Design
For the main design, we constructed a design of 40,000 runs of the
computer model composed of:
Each of 4 possible reservoir classes θ
For each θ, we choose 1000 values of δ in a Latin hypercube over the
3-dimensional decision space
For each (θ, δ), we choose 10 different values of the simulator input x in a
further Latin-hypercube over the 9-dimensional x space
This gave 40,000 runs of the computer model requiring 11.52 hours to
complete.
16 / 33
Methodology
-5
-10
Z
0
Results: Design
-6
-4
-2
0
2
4
6
T
16 / 33
Methodology
Results: Design
For the main design, we constructed a design of 40,000 runs of the
computer model composed of:
Each of 4 possible reservoir classes θ
For each θ, we choose 1000 values of δ in a Latin hypercube over the
3-dimensional decision space
For each (θ, δ), we choose 10 different values of the simulator input x in a
further Latin-hypercube over the 9-dimensional x space
This gave 40,000 runs of the computer model requiring 11.52 hours to
complete.
For the calibration design, we chose 250 values of δ for each of the 4
reservoir classes which was replicated 100 times
16 / 33
Methodology
Evaluation
Given the designs, the next stage is to perform the evaluations of the
computer model and generate the synthetic observations.
Each design point corresponds to a single combination of (δ, θ), but the
input to the computer model is in terms of the parameters (d, x)
Unfortunately, we could associate many possible (d, x) with any given
choice of (δ, θ) due to:
Decision uncertainty — we cannot guarantee δ is performed exactly in the
real world
Representation uncertainty — our general system state θ (or decision δ)
could have a non-unique representation in computer model terms
Therefore to evaluate the model at (δ, θ), we must sample a compatible
(d, x) before we can evaluate the computer model (or many if we need
replicates)
We then evaluate f (d, x) and combine it with some observational error
and model discrepancy to generate a synthetic observation z(d, x)
17 / 33
Methodology
Parameters and Evaluation
θ is a 4-valued categorical variable representing the reservoir classes
x is a 9-dimensional vector of numerical parameters uniquely specifying a
particular deconvolution function
δ are the parameters used to specify our well test:
1
2
3
Dur – the duration of production from the well
Rate – the rate of production during the well test
Buildup – the duration for which we observe pressures in the reservoir after
production has terminated
Rates are notoriously poorly observed and controlled, so we expect
substantial uncertainty on enacting a given decision
For each (x, δ) in our design, we generate the true-but-unknown
deconvolution function and a series of rates. These are then combined
(convolved) to produce the pressures, adding noise at various points. The
pressures and the rates are deconvolved, and this is our model output.
18 / 33
Methodology
Calibration
To collapse the end node of the decision tree, we require for every run of
the model the probabilities P [θ|δ, z]
Being Bayesians, we exploit the relation
P [θ|δ, z] ∝ P [z|δ, θ] P [θ]
Arbitrarily, we use the prior that P [θk ] = 41 for k = 1, . . . , 4
So all we need is P [z|δ, θ] to complete the calculation
We adopt a sampling approach to approximate P [z|δ, θ] and then use that
to approximately find the required probabilities
Using the evaluations of the calibration design and its many replicates,
we can approximate the empirical distribution of P [z|δ, θ]
19 / 33
Methodology
Calibration
If we find that P [z|δ, θ] is independent of (or at worst weakly dependent
on) δ we can choose to ignore the effects of δ on P [z|δ, θ] and calculate
P [z|θ] – which is can be simply represented as a collection of tables of
probabilities
If not, we will have to consider more sophisticated approaches such as
emulating the distribution as a function of δ itself
Another complication in this approach occurs when the dimension of z is
too large that P [z|θ, δ] becomes too complex an object to estimate easily
In which case, we reduce dimensionality by replacing z with a summary
statistic, g(z, θ).
Pragmatically, if we believe that this statistic has similar qualitative
behaviour with respect to the system state θ as the original observations,
i.e.
P [θ|δ, g(z, θ)] ' P [θ|δ, z] ,
(2)
then we can we can then replace z in the calibration by g(z, θ) and expect
to obtain similar results.
Then, given each run of the computer model z(d, x) we can now
approximately find P [θk |δ, z(d, x)] for each θk
20 / 33
Methodology
Results: Classification
*
1.0
0.8
0.0
-3.5
0.2
-3.0
0.4
-2.5
0.6
-1.5
-2.0
500
-6
-4
-2
0
2
4
6
1
2
3
4
t
0.00
0.00
0.15
Model 2 : -1.32
0.15
Model 1 : 0.14
-0.1
-0.05
0
0.05
0.1
0.15
0.2
-0.25 -0.2 -0.15 -0.1 -0.05
0
0.05
0.1
0.00
0.15
Model 4 : -5.11
0.15
Model 3 : -2.56
0.00
z
248,2183,79
-0.3
-0.2
-0.1
0
0.1
-0.3
-0.2
-0.1
0
0.1
0.2
21 / 33
Methodology
Generation of utilities
The last ingredient required to start solving the decision problem is the
utility value for each run of the model z(d, x).
Given any z(d, x), we must evaluate the utility function U [δ, α, θ] for all
possible actions, α, and states, θ, thereby attaching a utility value to
every leaf of the decision tree.
If the dimension of α or θ are large or the utility function is expensive to
evaluate, then it will be infeasible to consider such an exhaustive
evaluation.
Instead, we could sample from α and θ, evaluated the utility function for
this sample, and then use the sample of utilities as a training sample for
an emulator of the utility function
22 / 33
Methodology
Results: Utilities
Our utility for the combination (δ, α, θ) is a combination of the rewards for
taking actions (δ, α) when the system is in state z(d, x) and the cost for
taking the decisions (δ, α)
The reward we receive for taking these actions depends on the true state
of the reservoir
Large Small Abandon
Infinite
100
75
0
Fault
50
75
0
Channel
50
75
0
Closed
-10
-5
100
We use a simple linear cost function over the decisions δ with coefficients
Intercept
50
log(Duration) 0.035
0.03
log(Buildup)
Rate 0.003
23 / 33
Methodology
Solving the tree
U [δ, α, θ]
θ
System
P[
, z]
θ|δ
α
Action
z
P[
]
z|δ
Observation
δ
Decision
24 / 33
Methodology
Solving the tree
The first steps in solving the decision tree via the computer model are
then:
1
2
3
Find U [δ, α, z(d, x)] = E [U [δ, α, θ]] with respect to P [θ|δ, z(d, x)] — this is
why we needed the calibration
Find U [δ, z(d, x)] = U [δ, α∗ , z(d, x)] where the action α∗ maximises
U [δ, α, z(d, x)]
Find U [δ] = E [U [δ, z(d, x)]] with respect to P [z(d, x)|δ]
To perform step (3), we use the collection of evaluations associated with
each δ to make a simple Monte Carlo assessment of the expectation
25 / 33
Methodology
Emulating the utility
We have now reduced the problem to a sample of values of (δ, U [δ])
where U [·] is an expensive and poorly-understood function
Since U [δ] is a complex function about which we only have partial
knowledge over the space of δ, we apply standard emulation techniques
to approximate its behaviour.
Emulation revealed that the Rate decision variable was not active, and so
did not contribute to variation in the U [δ] surface
This left Duration and Buildup as the driving variables for the U [δ] surface
26 / 33
Methodology
Results: U [δ] (costless)
27 / 33
Methodology
Results: U [δ] including costs
28 / 33
Methodology
Optimisation
Since we have emulated the utility it is now substantially more efficient to
investigate a new choice of δ than it is to repeat the decision analysis.
Thus we can probably optimise by brute force for problems of
small-dimension
For problems of higher dimension we will require more sophisticated
optimisation routines (e.g. gradient ascent should be easy given an
emulator with well-explained by its mean function)
Additionally, larger problems would require more fancy methods of
visualisation (e.g. as in implausibility plots)
For the example, we could grid the space to sufficient detail to find the
maximum by brute force
The unique maximum of the emulated utility surface occurs at
Duration= 65.7hrs and Buildup= 139.6hrs.
However there is a region of (almost equally) good decisions when
surrounding this point.
29 / 33
Methodology
Validation
There are many validation questions we should ask of our results:
Are the empirical likelihoods used in the calibration step adequate?
Does our utility emulator capture the behaviour of the utility surface?
Is our Monte Carlo estimate for the expected utility adequate?
Is the ‘best’ decision really the best when accounting for all the
uncertainties?
30 / 33
Comments
Comments
31 / 33
Comments
Comments
There will be a paper soonTM
32 / 33
Comments
FIN
33 / 33