Quantitative methods to manage
uncertainty in science
by
Andrea Saltelli, Stefano Tarantola and
Michela Saisana, Joint Research
Centre of the European Communities in
Ispra (I), [email protected]
Mini-symposium
“The management of uncertainty in
risk science and policy”,
World Congress on Risk
Brussels, 22-25 June 2003.
http://www.jrc.cec.eu.int/uasa
1
Models mimic systems
Decoding
N
F
Natural
system
Formal
system
Entailment
Entailment
Rosen’s
formalisation
of the
modelling
process
Encoding
http://www.jrc.cec.eu.int/uasa
2
Models mimic systems (Rosen)
“World” (the natural system) and “Model” (the
formal system) are internally entailed driven by a causal structure.
Nothing entails with one another “World” and
“Model”; the association is hence the result of a
craftsmanship.
But this does not apply to natural systems only:
give 10 engineers the blueprint of the same plant
and they will return you 10 model based risk
assessments for the same plant.
http://www.jrc.cec.eu.int/uasa
3
Models mimic systems (Rosen)
It can help the craftsman that the uncertainty in the information
provided by the model (the substance of use for the decoding
exercise) is carefully apportioned to the uncertainty associated
with the encoding process.
http://www.jrc.cec.eu.int/uasa
4
Models maps assumptions onto inferences ...
but often too narrowly
<<[…] most simulation models will be complex, with many
parameters, state-variables and non linear relations. Under
the best circumstances, such models have many degrees of
freedom and, with judicious fiddling, can be made to
produce virtually any desired behaviour, often with both
plausible structure and parameter values.>>, HORNBERGER
and Spear (1981)
<<Cynics say that models can be made to conclude anything
provided that suitable assumptions are fed into them.>>,
The Economist, 1998.
KONIKOV and Bredehoeft, 1992  Oreskes et al. 1994.
http://www.jrc.cec.eu.int/uasa
5
Use of models in the scientific discourse
But yet models are used ...
… and a legitimate question is the following:
“If we had mapped the space of uncertain
assumptions honestly and judiciously, would the
space of inference still be of use1?”
1Read: do we still have peak around some useful inference (e.g.
YES or NO, safe or unsafe, hypothesis accepted or rejected,
policy effective or ineffective etc. ) or do we have as many
YES as NO etc.?
http://www.jrc.cec.eu.int/uasa
6
Models maps assumptions onto inferences …
<<I have proposed a form of organised sensitivity analysis that I
call “global sensitivity analysis” in which a neighborhood of
alternative assumptions is selected and the corresponding interval
of inferences is identified. Conclusions are judged to be sturdy
only if the neighborhood of assumptions is wide enough to be
credible and the corresponding interval of inferences is narrow
enough to be useful.>>
Leamer, “Sensitivity Analysis would help”, 1990
http://www.jrc.cec.eu.int/uasa
7
Models maps assumptions onto inferences …
Leamer’s view of global Sensitivity Analysis (SA)
Space of
estimated
parameters
Simulation
inference
Space of plausible
models space
...
Other
assumptions
http://www.jrc.cec.eu.int/uasa
8
Models maps assumptions onto inferences …
(Parametric bootstrap version of UA/SA )
Input data
Model
(Estimation)
Estimated
parameters
Uncertainty
and
sensitivity
analysis
(Parametric bootstrap:
we sample from the
posterior parameter
probability)
Inference
http://www.jrc.cec.eu.int/uasa
9
Bootstrapping-of-the-modelling-process
version of UA/SA, after Chatfield, 1995
Model
(Model
Identification)
Loop on bootreplica of the
input data
(Estimation)
(Bootstrap of the
modelling process)
Estimation
of
parameters
Inference
http://www.jrc.cec.eu.int/uasa
10
Bayesian Uncertainty and Sensitivity Analysis
(Draper 1995, Planas and Depoutot 2000)
Posterior
of Model(s)
Prior of
Model(s)
Model
Data
(Sampling)
Prior of
Parameters
Inference
Posterior
of
Parameters
http://www.jrc.cec.eu.int/uasa
11
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
The space of the model induced choices (the
inference) swells and shrinks by our swelling and
shrinking the space of the input assumptions. How
many of the assumptions are relevant at all for the
choice? And those that are relevant, how do they act
on the outcome; singularly or in more or less complex
combinations?
http://www.jrc.cec.eu.int/uasa
12
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
I desire to have a given degree of robustness in
the choice, what factor/assumptions should be
tested more rigorously? (=> look at how much
“fixing” any given f/a can potentially reduce the
variance of the output)
Can I confidently “fix” a subset of the input
factors/assumptions? The Beck and Ravetz
“relevance” issue. How do I find these factors?
http://www.jrc.cec.eu.int/uasa
13
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
Suggested tools :
V (Y )
Vx i (Y | X i  X i* )
E x* (Vx i (Y | X i ))
i
“Reduced” variance
Expected reduced variance
– it is small if the factor is
important.
http://www.jrc.cec.eu.int/uasa
14
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
Big if factor
important
Small if factor
important
Vx* ( Ex i (Y | X i  X i* ))  E x* (Vx i (Y | X i ))  V (Y )
i
i
sigm a(0.2) sigm a(0,3)
sigm a(0,4)
1%
1%
1%
f(0,1)
6%
rest
9%
hence :
Vx* ( Ex i (Y | X i  X i* ))
i
V (Y )
f(0,2)
14%
First order
effect
f(0,4)
38%
f(0,3)
30%
http://www.jrc.cec.eu.int/uasa
15
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
Also used is the total effect term:
Ex i (Vx* (Y | X i ))
This is the expected fractional value of the
variance that would be left if all factor but
Xi were fixed.
i
V (Y )
The use of different sensitivity measures
should be seen as the answer to a rigorous
question concerning the relative importance
of input factors.
http://www.jrc.cec.eu.int/uasa
16
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
One can thus relate the total effect term
Ex i (Vx* (Y | X i ))
to a question relative to the possibility to
fix factor(s), (Factor Fixing Setting),
while the first order effect
i
V (Y )
Vx ( Ex i (Y | X i ))
i
V (Y )
frames into the Factors’ Prioritisation
Setting.
http://www.jrc.cec.eu.int/uasa
17
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
Other setting (questions) can easily be
imagined. Settings to frame the uncertaitny
and sensitivity analyses are crucial.
The alternative would be to have different
SA methods suggesting different factors
relative imortance.
Settings should be audited! = Let us agree
on what “importance” means before we
engage in the analysis.
http://www.jrc.cec.eu.int/uasa
18
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
Is the model-induced choice weak (non robust)
because there is an insufficient number of
observations, or because the experts cannot
agree on an accepted theory?
http://www.jrc.cec.eu.int/uasa
19
Useful inference versus falsification
of the analysis
Example: imagine the inference is Y = the logarithm
of the ratio between the two pressure-on-decision
indices (Tarantola et als. 2000).
Region where
Incineration
is preferred
Region where
Landfill
is preferred
Y=Log(PI 1/PI 2)
http://www.jrc.cec.eu.int/uasa
20
Useful inference versus falsification
of the analysis
http://www.jrc.cec.eu.int/uasa
21
Use of models in the scientific discourse
… and role of uncertainty - sensitivity analysis
What happens if I address the space of the
policy options?
http://www.jrc.cec.eu.int/uasa
22
Gauging the leverage of the policy
options latitude
desired
target bounds
errors
Policy Options
model structures
Simulation
Model
data
resolution
levels
uncertainty analysis
model
output
sensitivity analysis
parameters
feedbacks on input data and model factors
23
Conclusions
The output from global uncertainty and sensitivity analyses can
feed back into the extended peer review process via e.g.
- refocusing of the critical issues/factor,
- (re-assignment of weights for multiple criteria, or)
- inference falsification
- identification of policy relevance/ irrelevance
Note: EC Guidelines for Extended Impact Assessment inlcude
explicit and detailed indication for global SA!
http://www.jrc.cec.eu.int/uasa
24
References
ROSEN R., Life Itself - A Comprehensive Inquiry into Nature, Origin, and
Fabrication of Life. Columbia University Press 1991.
HORNBERGER G.M., and R. C. Spear (1981) An approach to the preliminary
analysis of environmental systems. Journal of Environmental management, 12, 718.
KONIKOV and Bredehoeft, 1992, "Groundwater models cannot be validated"
Advances in Water Resources 15(1), 75-83.
ORESKES, N. , Shrader-Frechette K., Belitz, K., 1994, Verification, Validation,
and Confirmation of Numerical Models in the Earth Sciences, SCIENCE, 263,
641-646
Edward E. Leamer, “Sensitivity Analysis would help”, in Modelling Economic
Series, Edited by CWJ Granger, 1990, Clarendon Press, Oxford.
CHATFIELD C., Model uncertainty , data mining and statistical inference,
J. R. Statist. Soc. A, 158 (3) , 419-466, 1993
http://www.jrc.cec.eu.int/uasa
25
Further reading on SA
Papers - Saltelli et als., Statistical Science, 2000; Saltelli and
Tarantola, JASA, 2002
Book - Saltelli et al. Eds., Sensitivity Analysis, 2000, John Wiley
& Sons publishers, Probability and Statistics series
Book - A primer (Sensitivity Analysis in Practice) will appear by
end 2003, with Wiley.
A forum - http://sensitivity-analysis.jrc.cec.eu.int/
Presentations of the mini-symposium on www.nusap.net
http://www.jrc.cec.eu.int/uasa
26