DG Joint Research Centre
Formal and informal
approaches to the quality of
information in integrated
assessment
Stefano Tarantola
January 24-25, 2002
Laxenburg, Austria
http://www.jrc.cec.eu.int/uasa
1
European Commission
Tools for Extended quality assurance
Information used as input to policy-making is complex,
uncertain and disputed.
Established guidelines: eg
NUSAP and model Pedigree
schemes for the quality
assurance of the decision
process.
[email protected]
http://www.jrc.cec.eu.int/uasa
2
European Commission
To set the frame
[Leamer, 1990 ] (economist):
“I propose a form of organised sensitivity analysis in
which a neighborhood of alternative assumptions is
selected and the corresponding interval of inferences
Chair in Man. Bus. Econ., UCLA
is identified.
http://www.jrc.cec.eu.int/uasa
3
European Commission
To set the frame
“Conclusions are judged to be sturdy only if the neighborhood of
assumptions is wide enough to be credible and the corresponding
interval of inferences is narrow enough to be useful.”
Edward E. Leamer, 1990 “Sensitivity Analysis
would help”, in Modelling Economic Series,
Edited by CWJ Granger, Clarendon Press,
Oxford. Chair in Man. Bus. Econ., UCLA
http://www.jrc.cec.eu.int/uasa
4
European Commission
To set the frame
We move one step further: after characterising
the interval of inferences (using e.g. the
statistical variance),
we apportion such variability to its constituents
(the input factors) in the space
of the assumptions (or input space).
Input factors should be
Decomposition of
model prediction
uncertainty
interpreted in sensu lato:
- alternative assumptions,
- choice of model,
- algorithmic alternatives,
- poorly-known data...
http://www.jrc.cec.eu.int/uasa
5
European Commission
The Case Study:
incineration vs. landfill (Austria 1994)
Robustness
assessment fails:
the interval of the
inference
is too wide
No defensible choice
is
possible given the
uncertainties.
Y
http://www.jrc.cec.eu.int/uasa
6
European Commission
The Case Study:
incineration vs. landfill (Austria 1994)
A
B
E/F
58%
Tu
Data
E/F
GWP
W_E
Data
16%
GWP
9%
Tu
3%
STH
4%
EEC
5%
W_E
5%
http://www.jrc.cec.eu.int/uasa
EEC
STH
7
European Commission
Settings for the sensitivity analysis
To validate or invalidate assessments
GSA used to show that the uncertainty in the decision on
whether to burn or dispose solid waste depends on the choice
of the system of indicators, and not on the quality of the
available data.
Money should not be spent to improve quality in data, but
to reach a consensus on the proper system of indicators.
V(Y)=V[E(Y|Xi)]+E[V(Y|Xi)]
Tarantola et al., in Saltelli
et al. Eds, (2000)
Sensitivity Analysis John
Wiley
http://www.jrc.cec.eu.int/uasa
8
European Commission
Settings for the sensitivity analysis
Problem simplification and dialogue optimisation
We look for those uncertain factors that have negligible
influence on the output.
These can be fixed to the most plausible value within
their range of variation.
The dimensionality of the input space is then reduced.
E/F
58%
Tu
Data
Useless discussing about the use of
different architectures to build the
composite indicator, when these do not
influence the result.
http://www.jrc.cec.eu.int/uasa
E/F
GWP
W_E
Data
16%
GWP
9%
Tu
3%
STH
4%
EEC
5%
EEC
STH
W_E
5%
9
European Commission
Settings for the sensitivity analysis
Output uncertainty reduction
Joint use of UA and GSA (iterative procedure).
Perform UA and get a confidence interval for the output
If it is unacceptably large, acquire better knowledge on the
most important factors. Perform UA again to check ...
It the output quality exceeds the requirements,
the specifications on the input quality can be relaxed,
starting from the less important factors. Crosetto and Tarantola (2001)
Int J Geogr Inf Science
http://www.jrc.cec.eu.int/uasa
10
European Commission
Bibliography
[1] Saltelli, A., K. Chan, M. Scott, Editors, 2000, Sensitivity analysis,
John Wiley & Sons publishers, Probability and Statistics series.
[2] Saltelli, A., Chan, K., Scott, M. Eds., 1999, Special Issue on sensitivity
analysis, Computer Physics Communications, 117.
[3] Saltelli A., Tarantola S., and Chan K., 1999, A quantitative, model
independent method for global sensitivity analysis of model output,
Technometrics, 41(1), 39-56.
[4] Saltelli A., Tarantola S., Campolongo F., 2001, Sensitivity analysis as
an ingredient of modelling, Statistical Science, 15(4), 377-395.
http://www.jrc.cec.eu.int/uasa
11
European Commission