Sensitivity Analysis for
Complex Models
Jeremy Oakley &
Anthony O’Hagan
University of Sheffield, UK.
Introduction
• An economic model is to be used to predict the costeffectiveness of a particular treatment(s).
• Values of some or all of the input parameters required
for the model are uncertain.
• This implies the output of the model, the costeffectiveness of the treatment is also uncertain.
• We wish to identify which input parameters are the most
influential in driving this output uncertainty.
• Should we learn more about these parameters before
making a decision?
Introduction
• Various measures of importance for input variables have
been proposed, such as the partial expected value of
perfect information (partial EVPI) (Klaxton et al, 2001).
• Computing the values of these measures is
conventionally done using Monte Carlo techniques.
These invariably require a very large numbers of runs of
the economic model.
• For computationally expensive models, this can be
completely impractical.
• We present an efficient alternative to Monte Carlo.
Sensitivity analysis: partial EVPI
• We work with net benefit: utility of treatment is
K x efficacy – cost
With K the monetary value of a unit increase in
efficacy.
• The economic model states the utilities of various
treatment as a function of a set of input variables X.
u = f (X )
• There is uncertainty regarding the true values of the
input variables X.
• Decision maker chooses treatment with highest
expected utility, U *.
Value of learning an input variable
• Denote Y to be the input variable in the model under
question.
• Expected value to the decision maker of learning Y
before choosing a treatment is U Y.
• Value of learning Y, the partial EVPI of Y is
U Y- U *.
• Partial EVPI is zero if the same decision is made for all
values of Y.
Improving on Monte Carlo
Use each run of the model to learn
about the entire model.
Efficient computation using an
emulator
• We can compute partial EVPIs more efficiently through
the use of an emulator.
• An emulator is a statistical model of the original
economic model which can then be used as a fast
approximation to the model itself.
• An approach used by Sacks et al (1989) for dealing with
computationally expensive computer models.
• Any regression technique can be used. We employ a
nonparametric regression technique based on Gaussian
processes (O’Hagan, 1978).
Gaussian processes
• The gaussian process model for the function f (X ) is
non-parametric; the only assumption made about
f (X ) is that it is a continuous function.
• The interpolation is exact; given sufficient runs of the
economic model, the gaussian process emulator
`becomes’ the economic model.
• It is also possible to allow for uncertainty in the
emulator; we can say how our estimate might change if
we were to obtain more runs from the economic model.
• Additional features can be exploited to speed up
computation.
Example: GERD model
• The GERD model, presented in Briggs et al (2002)
predicts the cost-effectiveness of a range of treatment
strategies for gastroesophageal reflux disease.
• Various uncertain inputs in the model related to
treatment efficacies, resource uses by patients.
• Model outputs mean number of weeks free of GERD
symptoms, and mean cost of treatment for a particular
strategy.
• Distributions for uncertain inputs detailed in Briggs et al
(2002).
GERD example
•
We consider a choice between three treatment
strategies:
›
›
›
Acute treatment with proton pump inhibitors (PPIs) for 8
weeks, then continuous maintenance treatment with PPIs at
the same dose.
Acute treatment with proton pump inhibitors (PPIs) for 8
weeks, then continuous maintenance treatment with hydrogen
receptor antagonists (H2RAs).
Acute treatment with proton pump inhibitors (PPIs) for 8
weeks, then continuous maintenance treatment with PPIs at
the a lower dose.
GERD example
• There are 23 uncertain input variables.
• We estimate the partial EVPI for each input variable,
based on 600 runs of the GERD model.
• We assume a value of $250 for each week free of GERD
symptoms. (It is straightforward to repeat our analysis
for alternative values).
• The GERD model is computationally cheap, and so we
can compare our estimates with Monte Carlo estimates
based on large samples.
GERD Results
Conclusions.
• The use of the Gaussian process emulator allows partial
EVPIs to be computed considerably more efficiently.
• Sensitivity analysis feasible for computationally
expensive models.
• Can also be extended to value of sample information
calculations.
Acknowledgement
We would like to thank Andrew Briggs, Ron Goeree,
Gord Blackhouse and Bernie O’Brien for providing us
with the GERD model and input distributions.