Working
Report
2006-31
Sensitivity Analysis Methods and a Biosphere
Test Case Implemented in EIKOS
Per-Anders
Ekström
Robert Broed
May 2006
POSIVA
OY
FI-27160 OLKILUOTO, FINLAND
Tel
+358-2-8372 31
Fax +358-2-8372 3709
Working Report 2006-31
Sensitivity Analysis Methods and a Biosphere
Test Case Implemented in EIKOS
Per-Anders Ekström
Robert Broed
Facilia AB
May
2006
Base maps: ©National Land Survey, permission 41/MYY/06
Working Reports contain information on work in progress
or pending completion.
The conclusions and viewpoints presented in the report
are those of author(s) and do not necessarily
coincide with those of Posiva.
ABSTRACT
Computer-based models can be used to approximate real life processes. These models
are usually based on mathematical equations, which are dependent on several variables.
The predictive capability of models is therefore limited by the uncertainty in the value
of these.
Sensitivity analysis is used to apportion the relative importance each uncertain input parameter has on the output variation. Sensitivity analysis is therefore an essential tool in
simulation modelling and for performing risk assessments.
Simple sensitivity analysis techniques based on fitting the output to a linear equation are
often used, for example correlation or linear regression coefficients. These methods
work well for linear models, but for non-linear models their sensitivity estimations are
not accurate. Usually models of complex natural systems are non-linear.
Within the scope of this work, various sensitivity analysis methods, which can cope
with linear, non-linear, as well as non-monotone problems, have been implemented, in a
software package, EIKOS, written in Matlab language. The following sensitivity analysis methods are supported by EIKOS: Pearson product moment correlation coefficient
(CC), Spearman Rank Correlation Coefficient (RCC), Partial (Rank) Correlation Coefficients (PCC), Standardized (Rank) Regression Coefficients (SRC), Sobol' method,
Jansen's alternative, Extended Fourier Amplitude Sensitivity Test (EFAST) as well as
the classical FAST method and the Smirnov and the Cramér-von Mises tests. A graphical user interface has also been developed, from which the user easily can load or call
the model and perform a sensitivity analysis as well as uncertainty analysis.
The implemented sensitivity analysis methods has been benchmarked with well-known
test functions and compared with other sensitivity analysis software, with successful
results.
An illustration of the applicability of EIKOS is added to the report. The test case used is
a landscape model consisting of several linked biosphere compartment models. The
model was created in the tool Pandora [PG+05]. The test case serves as an example of
how to apply the different sensitivity analysis methods to a model using EIKOS, and
also presents results from actual performed model simulations.
Keywords: models, uncertainty, sensitivity analysis, non-linear, non-monotone,
EIKOS, software
HERKKYYSANALYYSIMENETELMÄT JA BIOSFÄÄRIMALLI TESTAUSKÄYTTÖÖN TOTEUTETTUNA EIKOS-TYÖKALULLA
TIIVISTELMÄ
Tietokonepohjaisia malleja käytetään arvioimaan todellisen maailman prosesseja. Nämä
mallit pohjautuvat useimmiten useasta muuttujasta riippuviin matemaattisiin yhtälöihin,
joten niiden ennustamiskykyä rajoittavat näiden muuttujien arvojen epävarmuudet.
Herkkyystarkasteluja käytetään kunkin epävarman syöttöparametrin suhteellisen merkityksen selvittämiseen, mikä tekee niistä välttämättömän työkalun simulaatiomallinnuksessa ja laadittaessa riskiarvioita.
Usein käytetään yksinkertaisia herkkyystarkastelumenetelmiä, jotka perustuvat lineaarisen yhtälön sovitukseen, esimerkiksi korrelaatio- tai lineaarisia regressiokertoimia. Nämä menetelmät toimivat tietenkin hyvin lineaarisille malleille, mutta epälineaarisissa
tapauksissa ne antavat epätarkkoja tuloksia. Toisaalta monesti monimutkaisten luonnollisten ilmiöiden mallit ovat juuri epälineaarisia.
Tämän työn puitteissa toteutettiin Matlab-kielellä useita eri herkkyystarkastelumenetelmiä, jotka pystyvät käsittelemään lineaarisia, epälineaarisia ja ei-monotonisia malleja.
Menetelmistä muodostettiin yhtenäisen graafisen käyttöliittymän sekä mallien lataus- ja
kutsutoimintojen avulla ohjelmistopaketti, joka sai nimen EIKOS. Siinä on toteutettuna
seuraavat menetelmät: Pearsonin tulomomenttikorrelaatiokerroin (CC), Spearmanin järjestyskorrelaatiokerroin (RCC), osittaiset (järjestys)korrelaatiokertoimet (PCC), normitetut (järjestys)regressiokertoimet (SRC), Sobolin menetelmä, Jansenin vaihtoehto,
laajennettu Fourier-amplituditesti (EFAST) sekä klassinen FAST menetelmä ja Smirnovin ja Cramér-von-Misesin testit. Näitä käyttämällä EIKOS-työkalulla voidaan suorittaa
kattavasti niin herkkyys- kuin epävarmuusanalyysitkin. Toteutettujen menetelmien toiminta on varmistettu onnistuneesti vertailemalla hyvin tunnettuihin testifunktioihin ja
toisiin herkkyysanalyysiohjelmistoihin.
Raportin toinen osa esittelee EIKOS-työkalun käyttöä erään Olkiluodon biosfäärilaskuharjoitusmallin avulla. Tämä malli käsittää useita toisiinsa kytkettyjä kulkeutumismalleja, ja siten myös runsaasti parametreja, ja se on toteutettu käyttämällä Pandora-työkalua [PG+05]. Raportissa esitetään miten EIKOS:lla voidaan analysoida eri menetelmiä
käyttäen valmiiksi kehitettyä mallia sekä lisäksi tässä testitapauksessa saadut mallinnustulokset.
Avainsanat: mallinnus, epävarmuus, herkkyysanalyysi, epälineaarisuus, ei-monotonisuus, EIKOS, ohjelmisto
1
TABLE OF CONTENTS
ABSTRACT
TIIVISTELMÄ
TABLE OF CONTENTS...........................................................................................
1
LIST OF ABBREVIATIONS .....................................................................................
3
1
INTRODUCTION .........................................................................................
5
2
ANALYSING SENSITIVITY AND UNCERTAINTY ......................................
2.1
Uncertainty analysis .........................................................................
2.2
Sensitivity analysis ...........................................................................
7
7
8
3
SCREENING METHODS............................................................................. 11
3.1
Morris method .................................................................................. 12
3.2
Remarks on screening methods ...................................................... 14
4
SAMPLING-BASED METHODS ..................................................................
4.1
Graphical methods ...........................................................................
4.2
Regression analysis.........................................................................
4.3
Correlation coefficients ....................................................................
4.4
Rank transformations .......................................................................
4.5
Two-sample tests .............................................................................
4.6
Remarks on sampling-based methods ............................................
17
17
18
20
21
21
23
5
VARIANCE-BASED METHODS ..................................................................
5.1
Sobol' indices ...................................................................................
5.2
Jansen (Winding Stairs)...................................................................
5.3
Fourier amplitude sensitivity test......................................................
5.4
Extended Fourier amplitude sensitivity test .....................................
5.5
Remarks on variance-based methods .............................................
25
27
29
31
33
34
6
SUMMARY OF SENSITIVITY ANALYSIS METHODS ................................ 35
7
EXAMPLE OF APPLICATION OF EIKOS ...................................................
7.1
The landscape model used in the study ..........................................
7.2
Input parameters ..............................................................................
7.3
The studied output variables............................................................
7.4
Results of the sensitivity and uncertainty analysis ...........................
7.4.1 Results of screening study ...................................................
7.4.2 Results of the sampling based methods ..............................
7.4.3 Results of the Variance based methods ..............................
7.4.4 Conclusion of sensitivity analysis.........................................
8
CONCLUDING REMARKS .......................................................................... 59
REFERENCES
37
37
39
40
42
42
46
51
56
............................................................................................. 61
APPENDIX 1: COAST MODEL APPLIED ............................................................... 65
APPENDIX 2: LAKE MODEL APPLIED................................................................... 67
APPENDIX 3: RIVER MODEL APPLIED................................................................. 69
2
APPENDIX 4: FOREST MODEL APPLIED ............................................................. 73
APPENDIX 5: REPRODUCING THE RESULTS OBTAINED IN SECTION 7. ........ 75
3
LIST OF ABBREVIATIONS
UA
SA
MC
CC
RCC
SRC
SRRC
PCC
PRCC
FAST
EFAST
PDF
LHS
OAT
WS
Uncertainty Analysis
Sensitivity Analysis
Monte Carlo
Pearson Product Moment Correlation Coefficient
Spearman Rank Correlation Coefficient
Standardized Regression Coefficient
Standardized Rank Regression Coefficient
Partial Correlation Coefficient
Partial Rank Correlation Coefficient
Fourier Amplitude Sensitivity Test
Extended Fourier Amplitude Sensitivity Test
Probability Distribution Function
Latin Hypercube Sampling
One-at-a-time
Winding Stairs
eei
R2
Si
STi
Elementary effects (for parameter Xi)
Model coefficient of determination
First order sensitivity index (for parameter Xi)
Total order sensitivity index (for parameter Xi)
4
5
1
INTRODUCTION
In the safety assessments and safety case for spent nuclear fuel repositories simulation
modelling and environmental risk assessment is applied. Because of the complexity of
such models, not to mention the nature itself, analyses of significance of different contributors to the uncertainties in predictions with the models are needed. The modern
simulation models are usually too complex to allow this kind of assessment been done
directly. Thus, software tools capable of the task are needed.
EIKOS software is available from Facilia AB. Its development of EIKOS has been
sponsored by the Norwegian Radiation Protection Authority (NRPA) and Posiva. The
Pandora software [PG+05] used to construct the landscape model for this report has
been developed for Posiva and SKB (Svensk Kärnbränslehantering AB) jointly.
For testing the compliance of EIKOS with Pandora, testing the workflow and also to
gain experience on landscape modelling, i.e. connected ecosystem-specific biosphere
model blocks on the basis of terrain and ecosystem development forecasts, a modelling
exercise was also conducted to be reported as a test case for EIKOS/Pandora system.
This report has been organised into two main parts. The first part begins with a brief
introduction to sensitivity analysis, followed by a theoretical text about main sensitivity
analysis techniques incorporated in EIKOS, these methods are divided into
• Screening methods
• Sampling-based methods
• Variance-based methods
In the second part a biosphere test case implemented in Pandora and analysed using
EIKOS is presented and thoroughly discussed.
6
7
2
ANALYSING SENSITIVITY AND UNCERTAINTY
A simulation modelling and risk assessment tool, such as Pandora [PG+05], should include procedures for sensitivity analysis, which can be used to assess the influence of
different model parameters on simulation endpoints. Such assessments are needed for
ranking the parameters in order of importance serving various purposes, for example:
•
•
to identify major contributors to the uncertainties in the predictions with a
model,
to identify research priorities to improve risk assessments with a model.
In the following text, the model is seen as a black box function f with k input parameters
x=(x1, x2,…, xk) and one single scalar output y, i.e.
y = f( x1 , x 2 ,..., x k ) .
(1)
In an application y may have a higher dimensionality, for example it could be a vector
instead of a scalar. The term input parameter is used to denote any quantity that can be
changed in the model prior to its execution. It may be, for example, a parameter, an initial value, a variable or a module of the model.
2.1
Uncertainty analysis
The input parameters of models are not always known with a sufficient degree of certainty. Input uncertainty can be caused by natural variability as well as by errors and
uncertainties associated with measurements.
In this context, the model is assumed to be deterministic, i.e. the same input data would
produce the same output if the model was run twice. Therefore, the input uncertainties
are the only uncertainty propagated through the model affecting the output uncertainty.
The uncertainty of input parameters is often expressed in terms of probability distributions; which can also be derived from samples of measured values, i.e. empirical probability distributions. The different input parameters may have dependencies on each
other, i.e. they may be correlated.
Generally, the main reason of performing an uncertainty analysis is to assess the uncertainty in the model output that derives from uncertainty in the inputs. The question to be
investigated is: How does y vary when x varies according to some assumed joint probability distributions?
8
2.2
Sensitivity analysis
Andrea Saltelli [Sal00] states that Sensitivity analysis (SA) is the study of how the variation in the output of a model (numerical or otherwise) can be apportioned, qualitatively
or quantitatively, to different sources of variation, and how the given model depends
upon the information fed into it.
Saltelli also lists a set of reasons why modellers should carry out a sensitivity analysis,
these are to determine:
a)
b)
c)
d)
if a model resembles the system or processes under study;
the parameters that mostly contribute to the output variability;
the model parameters (or parts of the model itself) that are insignificant;
if there is some region in the space of input parameters for which the model
variation is maximal;
e) the optimal regions within the space of the parameters for use in a subsequent
calibration study;
f) if and which (group of) parameters interact with each other.
Local and global sensitivity analysis
Sensitivity analysis aims at determining how sensitive the model output is to changes in
model inputs. When input parameters are relatively certain, we can look at the partial
derivative of the output function with respect to the input parameters. This sensitivity
measure can easily be computed numerically by performing multiple simulations varying input-parameters around a nominal value. We will find out the local impact of the
parameters on the model output and therefore techniques like these are called local sensitivity analysis. For environmental and health risk assessments, input parameters will
often be uncertain and therefore local sensitivity analysis techniques will not be usable
for a quantitative analysis. We want to find out which of the uncertain input parameters
are more important in determining the uncertainty in the output of interest. To find this
we need to consider global sensitivity analysis, which are usually implemented using
Monte Carlo (MC) simulation and are, therefore, called sampling-based methods.
Choice of an appropriate method
Different sensitivity analysis techniques will do well on different types of model problems. At an initial phase, for models with a large amount of uncertain input parameters,
a screening method could be used to qualitatively find out which the most important parameters are and which are not important. The screening method implemented in
EIKOS is the Morris design [Mor91]. A natural starting point in the analysis with sampling-based methods would be to examine scatter plots. With these, the modeller can
graphically find out nonlinearities, nonmonotonicity and correlations between the inputoutput parameters.
9
For linear models, linear relationship measures like Pearson product moment correlation
coefficient (CC), Partial Correlation Coefficients (PCC) and Standardized Regression
Coefficients (SRC) will be adequate.
For non-linear but monotonic models, measures based on rank transforms like Spearman Rank Correlation Coefficient (RCC), Partial Rank Regression Coefficient (PRCC)
and Standardized Rank Regression Coefficients (SRRC) will perform well.
For non-linear non-monotonic models, methods based on decomposing the variance are
the best choice. Examples of these methods are the Sobol' method, Jansen's alternative,
the Fourier Amplitude Sensitivity Test (FAST) and the Extended Fourier Amplitude
Sensitivity Test (EFAST).
Methods of partitioning the empirical input distributions according to quantiles1 (or
other restrictions) of the output distribution are called Monte Carlo filtering. Measures
of their difference are called two-sample-tests. Two non-parametric2 two-sample-tests
are implemented: the Smirnov and the Cramér-von Mises tests.
1
The q-quantile of a random variable X is any value x such that the probability P(X≤x)=q.
Non-parametric -- minimal or no assumptions are made about the probability distributions of the parameters being assessed.
2
10
11
3
SCREENING METHODS
When dealing with computationally laborious models, containing large amounts of uncertain input parameters, screening methods can be used to isolate the set of parameters
that has the strongest effect on the output variability by performing only a few model
runs. This way the number of uncertain input parameters to examine might be reduced.
It is often the case that the number of significant input parameters is quite small compared to the total number of input parameters in a model.
The most appealing property of the screening methods is their low computational costs
i.e. the required number of model runs. A drawback of this feature is that the sensitivity
measure is only qualitative in the sense that the input parameters are ranked in order of
significance, but their absolute contribution is not quantified.
There are various screening techniques described in the literature. One of the simplest is
the one-parameter-at-a-time design (OAT). In this design the input parameters are varied in turn and the effect each has on the output is measured. Normally, the parameters
that are not varied are fixed at nominal values. A maximum and minimum value is often
used representing the range of likely values for each parameter. Usually the nominal
value is chosen to be mean of these extremes.
The OAT design can be used to compute the local impact of the input parameters on the
model outputs thus this method is often referred to as local sensitivity analysis. It is
usually carried out by computing partial derivatives of the output functions with respect
to the input variables. The approach often seen in the literature is, instead of computing
derivatives, to vary the input parameters in a small interval around the nominal value.
The interval is usually a fixed (e.g. 5%) fraction of the nominal value and is not related
to the uncertainty in the value of the parameters.
In general, the number of model runs required for an OAT design is of the order O(k)
(often, 2k+1), k being the number of parameters examined.
An elementary OAT design for computing the local impact of the input parameters is
implemented in EIKOS. In this design one choose a nominal value and extremes for
each input parameter, or assign a fraction of variation of the nominal value. EIKOS then
varies each parameter in turn to compute local sensitivities (partial derivatives) if the
extremes are not too far away from the nominal values and the model relationship is linear this approximation will be accurate.
12
X1: 5%
X2: 14%
X3
Unexplained: 49%
X2
X3: 32%
X1
−0.3562
0
0.3562
Figure 3.1. Example of results from performing a local sensitivity analysis, illustrated
as a tornado and a pie chart. The result clearly indicates that parameter X3 has the
most impact on the model prediction.
3.1
Morris method
Morris [Mor91] came up with an experimental plan that is composed of individually
randomised OAT designs. The data analysis is then based on the so-called elementary
effects, the changes in an output due to changes in a particular input parameter in the
OAT design. The method is global in the sense that it does vary the input values over
their whole range of uncertainty. The Morris method can determine if the effect of the
input parameter xi on the output y is negligible, linear and additive, nonlinear or involved in interactions with other input parameters x~i3.
90
80
←X4
←X1
Standard Deviations (σ)
70
←X3
60
←X2
←X6
←X5
50
40
30
←X17
10
0
−30
←X7
←X14
←X20
←X19
←X18
←X12
←X16
←X15
←X11
←X13
20
−20
−10
0
10
20
Estimated means (µ)
←X9
←X8
30
40
←X10
50
Figure 3.2. Example of results from performing a sensitivity analysis using Morris
method
3
xi is the input parameter under consideration; x~i is all input parameters, except the input parameter under consideration.
13
According to Morris if:
a) f(xi+∆, x~i) - f(x) is nonzero, then xi affects the output.
b) f(xi+∆, x~i) - f(x) varies as xi varies, then xi affects the output nonlinearly.
c) f(xi+∆, x~i) - f(x) varies as x~i varies, then xi affects the output with interactions
with other inputs.
where ∆ is the variation size.
The input parameter space is discretised and the possible input parameter values will be
restricted to be inside a regular k-dimensional p-level grid, where p is the number of
levels of the design. The elementary effect of a given value xi of input parameter Xi is
defined as a finite-difference derivative approximation:
eei (x) = [f( x1 , x 2 ,..., xi −1 , xi + ∆, xi +1 ,..., x k ) − f(x)] / ∆
(2)
for any xi between 0 and 1-∆ where x x ∈ {0,1 /( p − 1),2 /( p − 1),...,1} , and ∆ is a predetermined multiple of 1/(p-1). The influence of xi is then evaluated by computing several
elementary effects at randomly selected values of xi and x~i.
If all samples of the elementary effect of the i'th input parameter are zero, then xi does
not have any effect on the output y, the sample mean and standard deviation will both be
zero. If all elementary effects have the same value, then y is a linear function of xi. The
standard deviation of the elementary effects will then equal zero. For more complex interactions, due to interactions between parameters and nonlinearity, Morris state
[Mor91] that if the mean of the elementary effects is relatively large and the standard
deviation is relatively small, the effect of xi on y is mildly nonlinear. If the opposite, i.e.
the mean is relatively small and the standard deviation is relatively large, then the effect
is supposed to be strongly nonlinear. As a rule of thumb:
a) a high mean of elementary effects indicates a parameter with an important overall influence on the output and
b) a high standard deviation in elementary effects indicates that either the parameter is interacting with other parameters or the parameter has nonlinear effects on
the output.
To compute r elementary effects of the k inputs we need to do 2rk model evaluations.
With the use of the Morris randomized OAT design the number of evaluations are reduced to r(k+1).
The design that Morris proposed [Mor91] is based on the construction of a (k+1)×k orientation matrix B*. Rows in B* represent input vectors x=x(1), x(2),…, x(k+1) that define a
trajectory4 in the input parameter space, for which the corresponding experiment pro4
Trajectory -- A sequence of points starting from a random base vector in which two consecutive elements differ only for one component.
14
vides k elementary effects, one for each input parameter. The algorithm of Morris design is:
a) Randomly choose a base value x* for x, sampled from the set {0,1/(p-1),2/(p1),…,1-∆}.
b) One or more of the k elements in x are increased by ∆.
c) The estimated elementary effect of the i'th component of x(1) is (if changed by ∆)
eei(x(1))=[f(x1(1),x2(1),…,xi-1(1),xi(1)+∆,xi+1(1),…,xk(1))-f(x(1))]/∆
if x(1) has been increased by ∆, or
eei(x(1))=[f(x(1))-f(x1(1),x2(1),…,xi-1(1),xi(1)+∆,xi+1(1),…,xk(1))]/∆
if x(1) has been decreased by ∆
d) Let x(2) be the new vector (x1(1),x2(1),…,xi-1(1),xi(1)±∆,xi+1(1),…,xk(1)) defined above.
Select a new vector x(3) such that x(3) differs from x(2) for only one component j:
either xj(3)=xj(2)+∆ or xj(3)=xj(2)-∆, j≠i. Estimated elementary effect of parameter j
is then eej(x(2))=(f(x(3))-f(x(2)))/∆, if ∆ > 0, or eej(x(2))=(f(x(2))-f(x(3)))/∆, otherwise.
e) The previous step is then repeated such that a succession of k+1 input vectors
x(1),x(2),…,x(k+1) is produced with two consecutive vectors only differing in one
component.
To produce the randomised orientation matrix B* containing the k+1 input vectors, Morris proposed:
B * = (J k +1,1 x * + (∆ / 2)((2B − J k +1,k )D* + J k +1,k )P * ) ,
(3)
where Jk+1,k is a (k+1)×k matrix of ones, B is a (k+1)×k sampling matrix which has the
property that for every of its column i=1, 2,…, k, there are two rows of B that differ
only in their i'th entries. D* is a k-dimensional diagonal matrix with elements ±1 and
finally P* is a k×k random permutation matrix, in which each column contains one element equal to 1 and all the others equal to 0, and no two columns have ones in the same
position.
B* provides one elementary effect per parameter that is randomly selected, r different
orientation matrices B* have then to be selected in order to provide an r×k-dimensional
sample.
3.2
Remarks on screening methods
The main advantage of the Morris design is the relatively low computational cost. The
design requires only about one model run per computed elementary effect.
One drawback with the Morris design is that it only gives an overall measure of the interactions, indicating whether interactions exists, but it does not reveal which are the
most significant. Also it can only be used with a set of orthogonal input parameters, i.e.
correlations cannot be induced on the input parameters.
15
In an implementation, there is a need to think about the choice of the p levels among
which each input parameter is varied. In EIKOS, these levels correspond to quantiles of
the input parameter distributions, if the distributions are not uniform. For uniform distributions, the levels are obtained by dividing the interval into equidistant parts.
The choice of the sizes of the levels p and realizations r is also a problem; various experimenters have demonstrated that the choice of p=4 and r=10 produces good results
[STCR04].
16
17
4
SAMPLING-BASED METHODS
The sampling-based methods are among the most commonly used techniques in sensitivity analysis. They are computed on the basis of the mapping between the inputoutput-relationship generated by Monte Carlo simulation. Sampling-based methods are
sometimes called global since these methods evaluate the effect of xi while all other input parameters xj, j≠i, are varied simultaneously. All input parameters are varied over
their entire range. This in contrast to local perturbation approaches where the effect of xi
is evaluated when the others xj, j≠i, are kept constant at a nominal value.
In the rest of this section various common sampling-based sensitivity analysis methods
are described.
4.1
Graphical methods
Providing a means of visualizing the relationships between the output and input parameters, graphical methods play an important role in sensitivity analysis.
A plot of the points [Xij,Yj] for j=1,2,…,N, usually called a scatter plot Figure 4.1 can
reveal nonlinear or other unexpected relationships between the input parameter xi and
the output y.
0.99
0.98
0.97
Y
0.96
0.95
0.94
0.93
0.92
0.91
0.01
0.02
0.03
0.04
0.05
X
0.06
0.07
0.08
0.09
Figure 4.1. Example of a scatter plot with an overlay of a regression line.
18
Scatter plots are undoubtedly the simplest sensitivity analysis technique, and they are a
natural starting point in the analysis of a complex model. They facilitate the understanding of model behaviour and the planning of more sophisticated sensitivity analysis.
When only one or two inputs dominate the outcome, scatter plots alone often completely reveal the relationships between the model input X and output Y.
Using Latin hypercube sampling can be particularly revealing due to the full stratification over the range of each input variable.
A tornado graph Figure 4.2 is another type of plot often used to present the results of a
sensitivity study. It is a simple bar graph where the sensitivity statistics is visualized
vertically in order of descending absolute value. All of the global sensitivity measures
presented in this section can be presented with a tornado graph.
X2
X3
X5
X1
X7
X6
X4
−0.084
0
0.084
Figure 4.2. Example of a tornado chart.
4.2
Regression analysis
A sensitivity measure of a model can be obtained using a multiple regression to fit the
input data to a theoretical equation that could produce the output data with as small error as possible. The most common technique of regression in sensitivity analysis is the
least squares linear regression. Thus the objective is to fit the input data to a linear equation ( Yˆ = aX + b ) approximating the output Y, with the criterion that the sum of the
squared difference between the line and the data points in Y is minimized. A linear regression model of the N×k input sample X to the output Y takes the form:
19
k
Yi = β 0 + ∑ β j X ij + ε i ,
j =1
(4)
where β0 is the constant term, βj are the regression coefficients to be determined and εi is
the error due to the approximation, i.e. ε i = Yi − Yˆi .
A measure of the extent to which the regression model can match the observed data is
called the model coefficient of determination, R2, which is defined as:
R
2
∑
=
∑
N
i =1
N
(Yˆi − Y ) 2
(Yi − Y ) 2
i =1
,
(5)
where Yˆi is the approximated output obtained from the regression model and Yi and Y
are the original values and their mean respectively. If R2 is close to 1, then the regression model is accounting for most of the variability in Y. If on the other side R2 is low,
nonlinear behaviour is implicated and a linear approximation is therefore no good. Another method of analysis should therefore be used.
The regression coefficients βj, j=1,…,k, measure the linear relationship between the input parameters and the output. Their sign indicates whether the output increases (positive coefficient) or decreases (negative coefficient) as the corresponding input parameter
increases. Since the coefficients are dependent on the units in which X and Y are expressed, the normalized form of the regression model is used in sensitivity analysis:
k β s X − X
Yˆi − Yi
j ˆj
ij
j
=∑
,
sˆ
sˆ
sˆ j
j =1
(6)
where
⎡ N (Yi − Y ) 2 ⎤
sˆ = ⎢∑
⎥
⎣ i =1 N − 1 ⎦
1/ 2
⎡ N ( X ij − X j ) 2 ⎤
, sˆ j = ⎢∑
⎥
N − 1 ⎦⎥
⎣⎢ i =1
1/ 2
(7)
In sensitivity analysis, the standardized coefficients β j sˆ j / sˆ in Eq. (6), called standardized regression coefficients (SRCs), are used as a sensitivity measure.
If Xj are independent, SRCs provide a measure of the significance, based on the effect
of moving each variable away from its expected value by a fixed fraction of its standard
deviation while retaining all other variables at their expected values. Calculating SRCs
is equivalent to performing the regression analysis with the input and output variables
normalized to mean of zero and standard deviation of one.
20
4.3
Correlation coefficients
The correlation coefficients (CC) usually known as Pearson's product moment correlation coefficients, provide a measure of the strength of the linear relationship between
two variables. The CC between two N-dimensional vectors x and y is defined by
ρ xy =
[∑
∑
N
k =1
( x k − x )( y k − y )
N
( xk − x ) 2
k =1
] [∑
1/ 2
N
( yk − y) 2
k =1
]
1/ 2
,
(8)
where x and y are defined as the mean of x and y respectively. The CC could be reformulated as
ρ xy =
cov( x, y )
,
σ ( x)σ ( y )
(9)
where cov(x,y) is the covariance between the data sets x and y and σ(x) and σ(y) are the
sampled standard deviations.
Thus, the correlation coefficient is then the normalized covariance between the two data
sets and, as SRC, it produces a unitless index between -1 and +1. The CC is equal in
absolute value to the square root of the model coefficient of determination R2 associated
with the linear regression.
The CC only measures the linear relationship between two variables without considering the effect that other possible variables might have. So when more than one input parameters are under consideration, as it usually is, partial correlation coefficients (PCCs)
can be used instead to provide a measure of the linear relationships between two variables when all linear effects of other variables have been removed. The PCC between an
individual variable Xi and Y can be obtained from the use of a sequence of regression
models. The procedure begins with constructing the following regression models:
Xˆ i = c0 +
k
∑ c j X j and Yˆi = b0 +
j =1, j ≠ i
k
∑b
j =1, j ≠ i
j
Xj.
(10)
Then PCC is defined by the CC of X i − Xˆ i and Y − Yˆ . PCC can also be written in
terms of simple correlation coefficients by denoting the PCC of X1 and Y while holding
Z=X2,…, Xk fixed as ρX1Y|X2,…,Xk, then
ρ X Y |Z =
1
ρ X Y − ρ X Z ρ YZ
1
(1 − ρ
1
2
XiZ
2
)(1 − ρ YZ
)
.
(11)
21
Partial correlation coefficients characterize the strength of the linear relationship between two variables after a correction has been made for the linear effects of the other
variables in the analysis. Standardized regression coefficients on the other hand characterize the effect on the output variable that results from perturbing an input variable by a
fixed fraction of its standard deviation. Thus, PCC and SRC provide related, but not
identical, measures of the significance of the variable. When input parameters are uncorrelated, results from PCC and SRC are identical.
4.4
Rank transformations
Since the above methods are based on the assumption of linear relationships between
the input-output parameters, they will perform poorly if the relationships are nonlinear.
Rank transformation of the data can be used to transform a nonlinear but monotonic relationship to a linear relationship. When using rank transformation, the data is replaced
with their corresponding ranks. Ranks are defined by assigning 1 to the smallest value, 2
to the second smallest and so-on, until the largest value has been assigned the rank N. If
there are ties in the data set, then the average rank is assigned to them. The usual regression and correlation procedures are then performed on the ranks instead of the original
data values. Standardized rank regression coefficients (SRRC) are SRC calculated on
ranks, spearman rank correlation coefficient (RCC) are corresponding CC calculated on
ranks and partial rank correlation coefficients (PRCC) are PCC calculated on ranks.
The model coefficient of determination R2 in Eq. (5) can be computed with the ranked
data and measures then how well the model matches the ranked data.
Rank-transformed statistics are more robust, and provide a useful solution in the presence of long tailed input-output distributions. A rank-transformed model is not only
more linear, but it is also more additive. Thus the relative weight of the first-order terms
is increased on the expense of higher-order terms and interactions.
4.5
Two-sample tests
Two-sample tests were originally designed to check the hypothesis of two different
samples belonging to the same population [Con99]. In sensitivity analysis, two-sample
tests can be used together with Monte Carlo filtering. In Monte Carlo filtering the input
samples are partitioned into two sub-samples according to restrictions on the output distribution. The two-sample test is then performed on the two empirical cumulative distributions of the sub-samples, and measured if their difference is significant or not, i.e. if
the two distributions are different, it can be said that the parameter influences the output.
The Smirnov test is defined as the greatest vertical distance between the two empirical
cumulative distributions:
22
1
0.9
Prior
F (x |Bhat)
0.8
Fm(xi|B)
n
i
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Figure 4.3. Example of difference plot of two empirical samples.
smirnov(Y , X j ) = max X j | F1 ( X j ) − F2 ( X j ) |,
(12)
where F1(x) and F2(x) are the cumulative distributions of Xi estimated on the two subsamples and the difference is estimated at all the xij points, i=1,…,N.
A test similar to the Smirnov test, but with slightly more calculations, is the Cramér-von
Mises test, which is given by:
cramer(Y , X j ) =
N1 N 2
( N1 + N 2 ) 2
∑ (F ( X
1
Xj
2
i
) − F2 ( X i )) ,
(13)
where the squared difference in the summation is computed at each xij, i.e. the statistic
depends upon the total area between the two distributions.
23
4.6
Remarks on sampling-based methods
According to Saltelli et al. [SM90, SH91] the estimators PRCC and SRRC appears to
be, in general, the most robust and reliable methods of the ones described in this section,
followed by Spearman's RCC and the Smirnov test. Nevertheless, all above tests have
been included in EIKOS since they could be of use in different contexts.
The rankings of SRCC and PRCC are usually identical, so they could be considered redundant. Differences occur only when there are significant correlations amongst the input parameters.
Predictions using SRRC are strongly correlated to those of Spearman's RCC and the
Smirnov test. Therefore, the value of the model coefficient of determination R2 plays a
crucial role, as it indicates the degree of reliability of the regressed model in SRRCs as
well as the other techniques.
24
25
5
VARIANCE-BASED METHODS
The main idea of the variance-based methods is to quantify the amount of variance that
each input parameter Xi contributes on the unconditional variance of the output V(Y).
We are considering a model function of Y=f(X), where Y is the output and X=(X1, X2,...,
Xk) are k independent input parameters, each one varying over its own probability density function.
The aim is to rank the input parameters according to the amount of variance that would
disappear, if the true value xi* of a given input parameter Xi was known. The V(Y|Xi=xi*)
is the conditional variance of Y given Xi= xi* and obtained by taking the variance over
all parameters but Xi.
In most cases the true value xi* for each Xi is unknown. Therefore, the average of this
conditional variance for all possible values xi* of Xi, is used, i.e. E[V(Y|Xi)] is the expectation value over the whole variation interval of the input Xi. Having the unconditional
variance of the output V(Y), the above average and by using: V(Y) = V(E[Y|Xi]) +
E[V(Y|Xi)], we obtain the variance of the conditional expectation Vi = V(E[Y|Xi]). This
measure is sometimes called main effect and used as an indicator of the significance of
Xi on the variance of Y, i.e. the sensitivity of Y to Xi. Normalising the main effect Vi by
the unconditional variance of the output gives:
Si =
V(E[Y | X i ])
V(Y )
(14)
The ratio Si was named first order sensitivity index by Sobol [Sob93]. Various other
names for this ratio can be found in the literature: importance measure, correlation ratio
and first order effect.
The first order sensitivity index measures only the main effect contribution of each input parameter on the output variance. It does not take into account the interactions between input parameters. Two parameters are said to interact if their total effect on the
output is not equal to the sum of their first order effects. The effect of the interaction
between two orthogonal parameters Xi and Xj on the output Y, in terms of conditional
variances, takes the form of:
Vij = V(E[Y | X i , X j ]) − V(E[Y | X i ]) − V(E[Y | X j ]).
(15)
V(E[Y|Xi,Xj]) describes the joint effect of the pair (Xi,Xj) on Y. This effect is known as
the second-order effect. Higher-order effects can be computed in a similar fashion, i.e.
the variance of the third-order effect between the three orthogonal parameters Xi, Xj and
Xl would be:
26
Vijl = V(E[Y | X i , X j , X l ]) − Vij − Vil − V jl − Vi − V j − Vl .
(16)
A model without interactions is said to be additive, for example a linear model is always
additive. The first order indices sums up to one in an additive model with orthogonal
inputs. For additive models, the first order indices coincide with outputs of regression
methods (described in section 3.2). For non-additive models information from all interactions is seeked for, as well as the first order effect. For non-linear models the sum of
all first order indices can be very low. The sum of all the order effects that a parameter
accounts for is called the total effect [HS96]. So for an input Xi, the total sensitivity index STi is defined as the sum of all indices relating to Xi (first and higher order). Having
a model with three input parameters (k=3), the total sensitivity index for input parameter
X1 would then be
S T1 = S1 + S12 + S13 + S123 .
(17)
Computing all order-effects to obtain the total effect by brute force is not advisable
when the number of input parameters k increases, since the number of terms needed to
be evaluated are as many as 2k-1.
In the following subsections, four methods to obtain these sensitivity indices are described. These are the Sobol' indices, Jansen's Winding Stairs technique, Fourier Amplitude Sensitivity Test (FAST) and the Extended Fourier Amplitude Sensitivity Test
(EFAST). All four methods, except FAST, can obtain both the first and total order effect in an efficient way. The standard FAST method only computes the first order effect.
The Sobol method can achieve all order effects, but the need of model runs increases
too much to be practical.
The input parameter space Ωk is hereafter assumed to be the k-dimensional unit hypercube: Ωk = (X | 0 ≤ Xi ≤ 1, i = 1,…, k). This does not give any loss of generality since
the parameters can be deterministically transformed from the uniform distributions to a
general probability distribution function. The input parameters are also assumed to be
orthogonal against each other, thus no correlation structure can be induced on the input
parameters.
The expected value of the output E(Y) can be evaluated by the k-dimensional integral as
E(Y ) = ∫ k f( X) p( X)dX = ∫ k f( X)dX,
Ω
Ω
(18)
where p(X) is the joint probability density function assumed to be uniform for each input parameter.
27
5.1
Sobol' indices
Sobol [Sob93] introduced the first order sensitivity index by decomposing the model
function f into summands of increasing dimensionality:
k
k
k
f( X 1 ,..., X k ) = f 0 + ∑ f i ( X i ) + ∑
i =1
∑f
l =1 j = i +1
ij
( X i , X j ) + ... + f 1...k ( X 1 ,..., X k ).
(19)
This representation of the model function f(X), holds if f0 is a constant (f0 is the expectation of the output, i.e. E(Y)) and the integrals of every summand over any of its own
variables are zero, i.e.:
1
∫f
0
ii
( X i1 ,..., X is )dX ik = 0, if 1 ≤ k ≤ s .
(20)
As a consequence of this, all the summands are mutually orthogonal.
The total variance V(Y) is defined as
V(Y ) = ∫ k f 2 ( X)dX − f 02
Ω
(21)
and the partial variances are computed from each of the terms in Eq. (19)
1
1
0
0
Vi1 ...is = ∫ ...∫ f i12...is ( X i1 ,..., X is )dX i1 ,..., dX is ,
(22)
where 1 ≤ i1 <…< is ≤ k.
The sensitivity indices are then obtained by
S i1 ...is =
Vi1 ...is
V(Y )
,
(23)
for 1 ≤ i1 <…< is ≤ k.
The integrals in Eq. (21) and Eq. (22) can be computed with Monte Carlo methods. For
a given sample size N the Monte Carlo estimate of f0 is:
1
f0 =
N
N
∑ f(X
m =1
m
),
where Xm is a sampled point in the input space Ωk.
(24)
28
The Monte Carlo estimate of the output variance V(Y) is
V̂(Y ) =
1
N
N
∑f
2
m =1
( X m ) − f̂ 02 ,
(25)
The main effect of input parameter Xi is estimated as
V̂i =
1
N
N
∑ f(X
m =1
( M 1)
~ im
) f( X (~Mim2 ) , X im( M 1) ) − f̂ 02 .
(26)
where two sampling matrices, X(M1) and X(M2), are used; both of size N×k. X (~Mim1) identifies the full set of samples from X(M1) except the i'th one. Matrix X(M1) is usually called
the data base matrix while X(M2) is called the resampling matrix [CST00].
For formulas of computing partial variances of higher order than in Eq. (26), a separate
Monte Carlo integral is required to compute any effect, as further discussed in [HS96].
Counting the Monte Carlo integrals needed for computing f̂ 0 , a total of 2k Monte Carlo
integrals are therefore needed for a full characterization of the system.
In 1996, Homma and Saltelli proposed an extension for direct evaluation of the total
sensitivity index STi [HS96]: STi can be evaluated with just one Monte Carlo integral
instead of computing the 2k integrals. They suggested dividing the set of input parameters into two subsets, one containing the given variable Xi and the other containing its
complementary set Xci. The decomposition of f(X) would then become:
f( X) = f 0 + f i ( X i ) + f ci ( X ci ) + f i ,ci ( X i , X ci ) .
(27)
The total output variance V(Y) would be
V(Y ) = Vi + Vci + Vi ,ci ,
(28)
and the total effect sensitivity index STi
S Ti = S i + S i ,ci = 1 − S ci .
(29)
Thus, to obtain the total sensitivity index for variable Xi we only need to obtain its complementary index Sci=Vci/V(Y). Homma and Saltelli [HS96] shows that Vci can be estimated with just one Monte Carlo integral, as
V̂ci =
1
N
N
∑ f(X
m =1
( M 1)
~ im
, X im( M 1) ) f( X (~Mim1) , X im( M 2 ) ) − f̂ 02 .
(30)
29
The first and total order sensitivity indices can be computed as: S i = V̂i / V̂(Y ) and
S Ti = 1 − V̂ci /V̂(Y ) respectively.
To obtain both first and total order sensitivity indices for k parameters and N samples,
with Sobol' method, we need to make N(2k+1) model runs if either X(M1) or X(M2) is
used to compute the unconditional variance V(Y) as described in Eq. (25).
When the mean value (f0) is large, a loss of accuracy is induced in the computation of
the variances by the Monte Carlo methods presented above [Sob01]. If c0 is defined as
an approximation to the mean value f0, the new model function f(x)-c0 can be used instead of the original model function (f(x)) in the analysis. For the new model function,
the constant term will then be very small, and the loss of accuracy, due to a large mean
value, will disappear. This transformation is, however, not implemented in EIKOS.
When computing Sobol' indices, the standard Monte Carlo sampling schemes are usually not used, in favour of Sobol's LPτ sequences. LPτ sequences are quasi-random sequences used to produce points uniformly distributed in the unit hypercube. The difference between quasi-random numbers and uncorrelated pseudo-random numbers is that
the quasi-random numbers maintain a nearly uniform density of coverage of the domain
while pseudo-random numbers may have places that are relatively undersampled and
other places that have clusters of points. The main reason to use quasi-random numbers
instead of pseudo-random numbers in a Monte Carlo simulation is that the former converge faster [Sob01].
Algorithms to produce LPτ sequences have not yet been incorporated in EIKOS, thus
the Sobol' indices has to be computed on the basis of pseudo-random numbers.
5.2
Jansen (Winding Stairs)
Chan, Saltelli and Tarantola [CST00] proposed the use of a new sampling scheme to
compute both first and total order sensitivity indices in only N×k model runs. The sampling method used to measure the main effect was called Winding Stairs, developed by
Jansen, Rossing and Deemen in 1994 [JRD94]. The main effect is computed as
V i J = V( Y ) −
1
E[f( X i , X ~ i ) − f( X i , X '~ i )] 2 ,
2
(31)
and the total effect is computed as
VTJi =
1
E[f( X i , X ~i ) − f( X i' , X ~i )]2 .
2
(32)
Here (Xi,X´~i) denotes that all variables are resampled, except for the i'th one. Jansen's
method uses the squared differences of two sets of model outputs to compute the indices
30
whereas the Sobol' method uses the product [CTSS00]. It has been shown that the covariance’s in
1
E[f( X i , X ~i ) − f( X i , X '~i )]2 = V(Y ) − cov[f( X i , X ~i ), f( X i , X '~i )]
2
(33)
and
1
E[f( X i , X ~i ) − f( X i' , X ~i )]2 = V(Y ) − cov[f( X i , X ~i ), f( X i' , X ~i )]
2
(34)
are equivalent to those of the Sobol' first and total partial variances.
The Winding Stairs (WS) sampling scheme was designed to make multiple use of the
number of model runs. With a single series of N model evaluations, it can compute both
the first-order and the total sensitivity indices. The winding stairs method consists of
computing the model outputs after each drawing of a new value for an individual parameter and building up a so-called WS-matrix.
The WS-matrix is set up in such a way that no two observations within a column share
common input parameters. Therefore, the output within each column of the matrix is
independent and can be used to estimate the variance of the output. In total, k×(N+1)
input points are generated. [CST00] discusses further on the theory on how the WSmatrix cyclically is built up. An example of the output Winding Stairs matrix for k=3
and N=4:
⎡ y1
⎢y
⎢ 4
⎢ y7
⎢
⎣ y10
y 2 y 3 ⎤ ⎡ f( X 11 , X 21 , X 31 )
y 5 y 6 ⎥⎥ ⎢⎢f( X 12 , X 22 , X 32 )
=
y8 y 9 ⎥ ⎢ f( X 13 , X 23 , X 33 )
⎥ ⎢
y11 y12 ⎦ ⎣f( X 14 , X 24 , X 34 )
f( X 11 , X 22 , X 31 ) f( X 11 , X 22 , X 32 ) ⎤
f( X 12 , X 23 , X 32 ) f( X 12 , X 23 , X 33 ) ⎥⎥
f( X 13 , X 24 , X 33 ) f( X 13 , X 24 , X 34 )⎥
⎥
f( X 14 , X 25 , X 34 ) f( X 14 , X 25 , X 35 )⎦
(35)
The WS sample estimate of V(Y) is then computed as
V̂
WS
k ⎡ N
1
⎡1
(Y ) =
⎢∑ y 2 (m, i ) − ⎢
∑
k ( N − 1) i =1 ⎢⎣ m =1
⎣N
⎤
y (m, i )⎥
∑
m =1
⎦
N
Where y(m,i) is the (m,i)'th element in the WS-matrix.
Estimates of the main effect Vi are computed as
2
⎤
⎥,
⎥⎦
(36)
31
V̂iWS = V̂ WS (Y ) −
1
2N
N
∑[ y
j =i
k ( j −1+1)
− y kj +i −1 ] 2 ,
(37)
where yk is the k'th y as in the example WS-matrix in Eq. (35), circularly shifted if the
index is out of bounds.
Estimates of the complementary effect Vci is computed as
V̂ciWS
⎧ 1 N
2
if i = 1
⎪⎪ 2 N ∑ [ y jk − y jk +1 ] ,
j =i
WS
= V̂ (Y ) − ⎨
N
⎪ 1 ∑ [ y k ( j −1) +i −1 − y k ( j −1) +i ] 2 , if i ≠ 1
⎪⎩ 2 N j =i
(38)
As before, the first and total order sensitivity indices are computed as S i = V̂iWS /V̂ WS (Y )
and S Ti = 1 − V̂ciWS /V̂ WS (Y ) respectively.
5.3
Fourier amplitude sensitivity test
The Fourier Amplitude Sensitivity Test (FAST) was proposed already in the 1970's
[CSF+73, SS73, CSS75], and at the time it was successfully applied to two chemical
reaction systems involving sets of coupled, nonlinear rate equations.
The main idea underlying the FAST method is to convert the k-dimensional integral in
Eq. (18) into a one-dimensional integral, applying a theorem proposed by Weyl
[Wey38]. Each uncertain input parameter Xi is related to a frequency ωi and transformed
by Xi(s) = Gi(sin(ωis)), where Gi is a suitably defined parametric equation which allows
each parameter to be varied in its range, as the parameter s is varied. The set {ω1,…,ωk}
are linearly independent integer frequencies. The parametric equations define a curve
that systematically explores the whole input parameter space Ωk.
According to Chan et al. [CTSS00], the multidimensional integral in Eq. (18) can be
estimated by integrating over the curve as
Ê[Y ] =
1
2π
π
∫ π f(s)ds ,
−
where f(s) = f(G1(sin(ω1s)), G2(sin(ω2s)),…, Gk(sin(ωks))).
The output variance may be approximated by performing a Fourier analysis as
(39)
32
V̂ FAST (Y ) =
1
2π
π
∫πf
−
2
( s )ds − E 2 [Y ] ≈
≈
∞
∑(A
j = −∞
2
j
+ B 2j ) − ( A02 + B02 ) ≈
N
≈ 2∑ ( A 2j + B 2j ),
j =1
(40)
where Ai and Bi are the Fourier coefficients defined as
1 π
f( s ) cos( js )ds ,
2π ∫−π
1 π
Bj =
f( s ) sin( js )ds.
2π ∫−π
Aj =
(41)
Finally, the partial variances are approximated by
M
V̂iFAST = 2∑ ( A p2ωi + B p2ωi ),
p =1
(42)
where M is the maximum harmonic considered5, usually assigned the value 4 or 6.
An application would need to numerically evaluate the Fourier coefficients A2pωi and
B2pωi. McRae et al. [MTS82] proposed the following difference expressions for the Fourier coefficients that can be derived by a simple numerical quadrature technique:
⎧ 0,
⎪
q
Aj = ⎨ 1 ⎛
⎛ πjp ⎞ ⎞
⎜y +
( y + y − p ) cos⎜
⎟ ⎟⎟,
⎪ N ⎜⎝ 0 ∑ p =1 p
⎝ N ⎠⎠
⎩
⎧0,
⎪
Bj = ⎨ 1 q
⎛ πjp ⎞
( y − y − p ) sin ⎜
⎟,
⎪⎩ N ∑ p =1 p
⎝ N ⎠
if j is odd
if j is even
if j is even
if j is odd
(43)
where q=(N-1)/2.
Saltelli et al. [CTS99] recommend a suitable transformation parametric equation Gi defined as
5
M is the maximum number of Fourier coefficients that may be retained in calculating the partial variances without interferences between the assigned frequencies.
33
X i ( s ) = G i (sin(ω i s )) =
1 1
+ arcsin(sin(ω i s )).
2 π
(44)
According to Saltelli et al. this transformation better provides uniformly distributed
samples for each parameter Xi in the unit hypercube Ωk than has been proposed by various others, among them Cukier et al. [CFS+73].
Saltelli et al. [STC99] has shown that according to the Nyquist criterion, the minimum
sample size required to compute V̂iFAST is 2Mωmax+1, where ωmax is the largest frequency in the set {ω1,…,ωk}.
In 1998, Saltelli and Bolado [SB98] proved that the ratio V̂iFAST /V̂ FAST (Y ) computed
with the FAST method is equivalent to the first order sensitivity indices proposed by
Sobol [Sob93].
5.4
Extended Fourier amplitude sensitivity test
In 1999, Saltelli et al. [STC99] proposed an improvement of the FAST method. They
called it the Extended Fourier Amplitude Sensitivity Test (EFAST). With this method
they could estimate the total effect indices, as in the Sobol method, by estimating the
variance in the complementary set V̂ciFAST . This is done by assigning a frequency ωi for
the parameter Xi, usually high, and almost identical frequencies to the rest ω~i, usually
low). The partial variance of the complementary set is then computed as
M
V̂ciFAST = 2∑ A p2ω ~ i + B p2ω ~ i
p =1
(45)
A modification of the parametric equation in Eq. (44) was also introduced to get a more
flexible sampling scheme. Since Gi in Eq. (44) always returns exactly the same points in
Ωk, a random phase-shift ϕ i was added. The new equation now becomes
X i ( s ) = G i (sin(ω i s )) =
1 1
+ arcsin(sin(ω i s + ϕi )) .
2 π
(46)
Because of symmetry properties, the curve now must be sampled over (-π, π). The technique of using many phases generating different curves in Ωk and doing independent
Fourier analysis over them and finally taking the arithmetic means over the estimates is
called resampling.
34
A whole new set of model runs is needed to compute each of the k complementary variances V̂ciFAST , so the computational cost to obtain all first and total order indices are
k(2Mωmax+1)Nr, where Nr is the number of resamples that was done.
5.5
Remarks on variance-based methods
The variance-based methods described in this section are considered being quantitative
sensitivity analysis methods. All methods can compute the main effect contribution of
each input parameter to the output variance. The total sensitivity index can also be obtained by the Sobol', Jansen and EFAST methods. The total effect index is a more accurate measure of the influence of a parameter on the model output, since it takes into account all interaction effects involving that parameter.
To compute the main effect contribution, the FAST method only requires a single set of
model runs. EFAST needs k(2Mωmax+1)Nr model runs to compute both the main effect
as well as the total effect contribution. To compute the Sobol' indices, the required
model runs are N(2k+1) using the Sobol' method and only Nk runs using the WSsampling scheme designed to make multiple uses of the runs. Chan et al. [CTSS00] expressed a concern that the reduction of model runs in Jansen's method might affect the
accuracy of the obtained estimates.
Quasi-random sampling has not been implemented in EIKOS, and as it is the commonly
used sampling scheme in computing the Sobol' indices, the other methods might be
preferable.
35
6
SUMMARY OF SENSITIVITY ANALYSIS METHODS
In general, which sensitivity analysis method should be used? It depends on several parameters, such as how ”heavy” the needed model computations are, the number of uncertain input parameters and whether the model output depends linearly, monotonically
or non-monotonically on the input parameters under consideration.
The methods for parameter screening, for example the Morris method, are useful as a
first step in dealing with computationally laborious models containing large number of
input parameters. The parameters that control most of the output variability can be identified at low computational costs.
Local sensitivity analysis is used to investigate the impact of the input parameters on the
model, locally. It measures how sensitive the model is to small perturbations of the input parameters. When the model is nonlinear and various input parameters are affected
by uncertainties of different order of magnitude, local sensitivity analysis should not be
used.
If the model is nonlinear, correlation and regression coefficients can not be trusted. To
determine if there are nonlinearities in the model, the model coefficient of determination
can be examined. A R2-value that is lower than 0.6 indicates that the regression model
does not describe the dependency between the inputs and the outputs accurately enough.
Another way to detect nonlinearities in the model is to examine scatter plots of the inputs versus outputs.
The coefficients computed on the basis of rank-transformed data can handle non-linear
models that still are monotone. Using ranked data, the R2 value will be improved but a
drawback is that the model under analysis has been altered. The analysis based on rank
transformed data is more robust than for untransformed data, but for small sample sizes
the results are not as trustworthy, due to loss of information from the transformation.
According to Saltelli et al. [SM90] and [SH91], the most robust and reliable of the sampling-based methods are PRCC and SRRC. Therefore, the results obtained with these
methods should be trusted more, than results obtained with other sampling-based methods.
Latin Hypercube sampling can be used instead of simple random sampling in the Monte
Carlo analysis. Latin hypercube sampling forces the samples to be drawn from the full
range of the desired distribution functions. Thus, a lower number of samples can be
used to emulate the distribution functions. A drawback is that it can give a biased estimation of the variance of the distributions; therefore, Latin hypercube sampling should
not be used to generate sample distributions used in the variance-based methods.
Variance-based methods that can compute the total effect contribution of the input parameters on the model output should be used if the model is thought to be nonmonotone. These methods require more model evaluations, increasing with the number
of input parameters, than the other methods. For models with a moderate number of in-
36
put parameters and not too long model execution time, these methods are ideal. All
variance-based methods give the same type of results: first-order and total sensitivity
indices.
Different amount of model runs are required for the different methods. Sobol' method
requires N(2k+1) model evaluations while the Jansen method only requires Nk model
evaluations. For the Fourier amplitude sensitivity test the number of model evaluations
required is in the order of O(k2), while for the extended version k(2Mωmax+1)Nr model
evaluations are required. Drawbacks of the variance-based methods are their higher
computational costs and their assumption that all information about the uncertainty in
the output is captured by its variance.
Overall, sensitivity analysis provides a way to identify the model inputs that have the
strongest effect on the uncertainty in the model predictions. However, sensitivity analysis does not provide an explanation for such effects. This explanation must come from
the analysts involved and, of course, be based on the mathematical properties of the
model under consideration.
Sensitivity analysis methods are essential tools in risk assessment and simulation
modelling. Due to difficulties of implementing the methods, most sensitivity analyses
are today performed using local methods, correlation or regression coefficients. The
variance-based methods described here are recommended for studies of environmental
risk assessment models, since these are often non-linear and can be also non-monotonic.
Screening, preliminary step to
find the significant parameters
Morris method may reduce the
number of parameters
Model relationship is linear
Scatter plots show relationship
between inputs and outputs
Model relationship is nonlinear but still monotonic
Model relationship is nonmonotonic
Pearson product moment
correlation coefficients
Spearman rank correlation
coefficients
First and total order sensitivity
index (Si, STi)
Standardized regression
coefficients
Standardized ranked regression coefficients
Extended Fourier amplitude sensitivity test
Partial correlation coefficients
Partial ranked correlation
coefficients
Sobol’s Method
Fourier amplitude sensitivity
test (First order index Si)
Jansen’s method using
Winding Stairs sampling
Two-sample tests (Smirnov or
Cramér-von Mises test)
Figure 6.1. Flowchart of sensitivity analysis and selection of appropriate methods.
37
7
EXAMPLE OF APPLICATION OF EIKOS
To estimate the effect on man of radionuclide releases from a deep geological repository
it is necessary to know the distribution of radionuclides in the biosphere over time. By
identifying a likely release path from the discharge point, it is possible to develop a
landscape model consisting of a set of linked biosphere models. Such case was implemented in the tool Pandora [PG+05] and sensitivity and uncertainty analyses were carried out with EIKOS.
The objective of this example application was to study the relative performance of the
different methods implemented in EIKOS for this specific type of simulation modeling
problem. The model, input parameters and selected output variables used in the study
are in sections 7.1, 7.2 and 7.3 respectively.
The sensitivity analysis started with a screening of the input parameters (see section 3),
using Morris and the Local sensitivity methods, to identify which parameters should be
included in more detailed studies. Further sensitivity analyses were carried out using
sampling-based (see section 4) and variance-based methods (see section 5) in order to
compare the performance of the different methods available in EIKOS.
7.1
The landscape model used in the study
The model describes the future terrain and ecosystem development on the basis of sea
depth data and on the approximation that 2000 years after present there will be a land
uplift of 10 meters above the current depth and elevation values (Rautio et al. 2005). It
was assumed that there will be no significant sea level changes during this period. The
discharge was assumed to go directly to an area above the repository, which at the time
will be covered by forest. The succession of linked biosphere models was identified
from maps of the terrain and will also involve lakes, rivers and coastal areas, as illustrated in figure 7.1.
From the depressions remaining under the sea level, locations of lakes can be estimated.
Likely some of the lakes will be larger, or smaller, than the depression, but for testing
the tools and modelling methods these estimates were judged to be adequate.
Based on the type of current sea bottom sediments (Fig. 7.1), future forest types on the
area can be forecasted [RAI05]. The current prevailing forest types will prevail also in
the future, but somewhat more wetlands and deciduous forests around the water bodies
are predicted.
On the basis of the terrain and ecosystems forecast for the selected time period the corresponding landscape model (Fig. 7.2) was built using ecosystem-specific biosphere
modules taken from earlier assessments and modelling exercises (see Appendices 1-4).
A view of the implementation in Pandora [PG+05] of the interlinked ecosystem models
is shown in Fig. 7.3.
38
River 2
Lake 2
Baltic Sea
River 1
Lake 1
(Bothnian Sea)
River 0
Eurajoki
River
River 4
River 5
Lake 4
River 3
Lake 3
Lapinjoki
River
0
Sea 2000AP
Current coastline
Recent gyttja clay / mud
Washed sand layer
Lakes 2000AP
ONKALO UCRF
Litorina clay
Sand and gravel
Rivers 2000AP
Illustrative repository
Ancylus clay
Till
Glaciaquatic mixed sediment
Bedrock outcrop
1
2
4
km
´
ARIK / Oct 27, 2005
Figure 7.1. Surroundings of Olkiluoto at about 2000 AP (modified from Rautio et al.
2005).
´
Coast
River 2
Lake 1
Baltic Sea
(Bothnian Sea)
River 1
Lake 1
Forest
River 0
Eurajoki
River
River 4
River 5
Lake 4
River 3
Current coastline
Lake 3
Lapinjoki
River
0
1
2
4
km
ARIK / Oct 27, 2005
Figure 7.2. The area of Olkiluoto at about 2000 AP and the ecosystem models included
in the landscape model.
39
Outer Coast
Inner Coast
backflux soluble
outflux soluble
influx soluble
influx soluble
conc water
conc water
C_OuterCoas
C_OuterCoast
C_InnerCoas
C_InnerCoast
conc water
Cl-36 301000
Cs-137 30
I-129 15700000
Ni-59 76000
Pu-239 24110
Ra-226 1600
Tc-99 211000
Lake2
C_River2
C_River2
in soluble
out souluble
outflux water
River2
influx water
conc water
PANDORA
C_River1
C_River1
C_Lake2
River1
C_Lake2
Lake1
conc water
in soluble
outflux water
out souluble
influx water
conc water
C_Lake1
C_Lake1
conc soil
-C1 Bq/Yr Input
C_Forest
C_Forest
In1
outflux
Forest
Figure 7.3 The landscape model implemented in Pandora, the main level view.
The ecosystem models included in the landscape model were:
• a coast model taken from [KB00], see Appendix 1.
• a lake model taken from [KB00], see Appendix 2,
• a river model taken from [JE05], see Appendix 3,
• a forest model taken from [Avi06], see Appendix 4
7.2
Input parameters
The parameters of the landscape model were divided into three types: radionuclide specific (for example the distribution coefficients (Kd) and the soil-to-plant concentration
ratios CR), object specific and system specific. Object specific parameters are parameters that have specific values for each particular object, such as the mean depth of a
lake. System specific parameters are parameters that have the same value for all objects
of the same type.
40
The probability density functions assigned to the parameters (see Tables 7.1-7.3) were
taken from [KB00], except for the parameters of the forest model, which were taken
from [Avi06] and the parameters of the river model, which were taken from [JE05]. All
parameters were assigned the used triangular distribution, and the distributions are
specified by minimum, mode (most probable) and maximum values. Note that not much
attention was given to the uncertainty in the parameter data since the goal was to test
EIKOS performance, and get some first impressions of a real-case use of the tool on a
landscape model.
7.3
The studied output variables
The model supports calculations of time-dependent inventories and concentrations of
multiple radionuclides in several environmental media of the objects included in the
landscape (see Figure 7.3). For sensitivity analysis three output variables were considered:
• The Cl-36 concentration in the soil of the forest ecosystem after 1000 years of continuous release of 1 Bq/y into the forest.
• The Cl-36 concentration in the water of Lake 2 after 1000 years of continuous release of 1 Bq/y into the forest.
• The Cl-36 concentration in the water of outer coast after 1000 years of continuous
release of 1 Bq/y into the forest.
Table 7.1 The object-specific parameters used in the simulations. “GIS” refers to data
acquired from the geographical information described in section 7.1.
Parameter
Name in model
Unit
Min
Mode
Max
Ref.
Forest area
Forest catchment area
Bulk density of forest soil
Thickness of forest soil rooting layer
Yearly production of tree leaves
Yearly production of understorey plants
Yearly production of tree wood
Average tree lifetime
Volumetric water content in soil
Average depth of river 1
Length of river 1
Average depth of river 2
Length of river 2
Area of inner coast
Average depth of inner coast
Area of outer coast
Average depth of outer coast
Yearly average flowrate of Eurajoki River
Area of lake 1
Average depth of lake 1
Area of lake 2
Average depth of lake 2
area_Forest
Acatch_Forest
density_soil
h
LP
UP
WP
Tlife
theta
R1_Depth
R1_deltax
R2_Depth
R2_deltax
area_InnerCoast
IC_D
area_OuterCoast
OC_D
Flowrate_Eurajoki
L1Area
L1D
L2Area
L2D
m2
m2
Kg/m3
M
Kg dw/m2/yr
Kg dw/m2/yr
Kg dw/m2/yr
Year
m3/m3
M
M
M
M
m2
M
M2
M
m3/yr
m2
M
m2
M
5,0E+05
2,5E+05
7,0E+02
0.2
0.05
0.02
0.018
2,0E+01
0.1
0.1
1,0E+02
0.1
6,7E+01
1,0E+08
2.5
2,0E+11
6,0E+00
2,0E+08
2,6E+05
0.1
5,0E+04
0.1
2,2E+06
4,4E+06
1,2E+03
0.3
0.08
0.08
0.18
4,0E+01
0.2
0.3
4,1E+02
0.3
1,7E+03
2,0E+08
3,0E+00
2,3E+11
7,0E+00
3,0E+08
8,6E+05
0.3
1,4E+05
0.3
4,4E+06
7,0E+06
1,5E+03
0.5
1.7
0.25
1.8
6,0E+01
0.5
1,0E+00
1,5E+03
1,0E+00
2,9E+03
3,0E+08
3.5
2,6E+11
8,0E+00
4,0E+08
1,3E+06
0.9
2,2E+05
0.6
GIS
GIS
[Avi06]
[Avi06]
[Avi06]
[Avi06]
[Avi06]
[Avi06]
[Avi06]
GIS
GIS
GIS
GIS
GIS
[KB00]
GIS
[KB00]
GIS
GIS
[KB00]
GIS
[KB00]
41
Table 7.2 The system-specific parameters used in the simulations.
Parameter
Name in model
Unit
Min
Mode
Max
Ref.
Plant biomass in rivers
Cross-sectional angle in rivers
Diffusion coefficient in rivers
Particle size distribution, rivers (50%
percentile)
Particle size distribution, rivers (90%
percentile)
Resuspension of surface sediment
(coast)
Water retention time, inner coast
Specific density in rivers
Gross sedimentation rate, inner coast
Gross sedimentation rate, outer coast
Gross sedimentation rate, lakes
Suspended matter, inner coast
Suspended matter, outer coast
Suspended matter, lakes
Sedimentation velocity in rivers
Advective transport velocity in bed
siediment
Suspended particulate matter in
stream water
Depth of surface sediment in rivers
Depth of deep sediment in rivers
Sediment porosity in rivers
Sediment density in rivers
Dry mass of surface sediment, inner
coast
M_biomass
Alpha
D
D_50
Kg·fw/m2
Rad
m2/yr
m
4,0E+00
0.5
0.0079
0.0005
5,0E+00
0.7
0.0158
0.0007
6,0E+00
0.9
0.0237
0.0009
[JE05]
[JE05]
[JE05]
[JE05]
D_90
m
0.0017
0.0019
0.0021
[JE05]
Resusp
Year-1
0.1
0.2
0.3
[KB00]
RetTime_IC
S
SR_Coast
SR_Sea
SR_lake
Susp_Coast
Susp_Sea
Susp_lake
V_partsed
Vz
Year
[-]
Kg·dw/(m2·yr)
Kg·dw/(m2·yr)
Kg·dw/(m2·yr)
Kg·dw/m3
Kg·dw/m3
Kg·dw/m3
m/yr
m/yr
0.0014
2.4
0.5
0.05
0.5
0.003
0.0005
0.005
3,0E+02
157.68
0.002
2.6
1.1
0.2
1.1
0.009
0.001
0.01
4,0E+02
315.36
0.0027
2.8
1.5
0.4
1.5
0.03
0.002
0.06
5,0E+02
473.04
[KB00]
[JE05]
[KB00]
[KB00]
[KB00]
[KB00]
[KB00]
[KB00]
[JE05]
[JE05]
cp
Kg/m3
0.01
0.02
0.03
[JE05]
deltaz1
deltaz2
porosity_sed
rho_sed
rho_sed_Sea
m
m
[-]
Kg/m3
Kg/m2
0.25
0.25
0.7
7,0E+02
5,0E+00
0.5
0.5
0.8
1,1E+03
1,0E+01
0.75
0.75
0.9
1,5E+03
1,5E+01
[JE05]
[JE05]
[JE05]
[JE05]
[KB00]
Table 7.3 The radionuclide-specific parameters used in the simulations.
Parameter
Name in model
Unit
Min
Mode
Max
Ref.
Half-time to reach sorption equilibrium
Bioconcentration factor
Tk
Year
0.0005
0.001
0.0015
[JE05]
BCF[Cl-36]
1,0E+02
2,0E+02
3,0E+02
[JE05]
Conc. ratio of nuclide from soil to
tree leaves
Conc. ratio of nuclide from soil to
understorey plants
Conc. ratio of nuclide from soil to
tree wood
Distribution coefficient in sediment
in rivers
Distribution coefficient in stream
water
Distribution coefficient, lakes
CR_L[Cl-36]
(Bq/kg·fw)/(Bq/
l)
[-]
0.8
1,0E+01
2,8E+01
[Avi06]
CR_U[Cl-36]
[-]
3,0E+00
2,8E+01
1,7E+02
[Avi06]
CR_W[Cl-36]
[-]
0.8
3,0E+00
1,1E+01
[Avi06]
KB[Cl-36]
m3/kg
0.003
0.03
0.3
[JE05]
Kd[Cl-36]
m3/kg
0.03
0.3
3,0E+00
[KB00]
Kd_lake[Cl-36]
m3/kg
0.1
1,0E+00
1,0E+01
[KB00]
0.0001
0.001
0.01
[KB00]
Distribution coefficient soil
Kd_soil[Cl-36]
3
m /kg
42
7.4
Results of the sensitivity and uncertainty analysis
7.4.1
Results of screening study
The Morris method was used to screen out the parameters with negligible influence on
the outputs. The total number of realisations was 13750 and the quantiles of the distributions were used to define the p levels among which each input parameter was varied.
The results obtained with the Morris method for the three studied outputs are shown in
figures 7.4-7.6.
In addition to the Morris method, local sensitivity indexes were calculated for screening
out unimportant parameters. These were obtained by varying each parameter by 5% of
its nominal value. The results from the local method for the three studied outputs are
shown in figures 7.7-7-9.
It can be seen that the Local method only shows the parameters that are indicated as the
most significant by the Morris method as significant. The parameters indicated by the
Local method as significant are however coherent with the results from the Morris
method, and no additional parameters are indicated as important. The results of the
Morris and the Local screening methods are given in Table 7.4, listing the parameters
that were indicated as important for the selected outputs, in ranked order (constituting
totally 9 unique parameters of the model).
-9
1.6
x 10
Estimated means and standard deviations of elementary effects, for output ConcForest
Acatch Forest
1.4
Standard Deviations (σ )
1.2
Kd soil
1
0.8
0.6
0.4
0.2
0
0
1
2
3
Estimated means (µ)
4
5
6
-9
x 10
Figure 7.4 The results of the Morris method for the concentration of Cl-36 in forest
soil.
43
-11
8
x 10
Estimated means and standard deviations of elementary effects, for output ConcLake
7
Flowrate Eurajoki
Standard Deviations (σ )
6
5
Kd_soil,
WP,
Acatch_Forest,
CR_W,
Area_Forest
4
3
2
1
0
0
0.5
1
1.5
2
2.5
3
Estimated means (µ)
3.5
4
4.5
5
-10
x 10
Figure 7.5 The results of the Morris method for the concentration of Cl-36 in the water
of Lake 2.
-16
1.2
x 10
Estimated means and standard deviations of elementary effects, for output ConcCoast
1
Kd Soil,
Acatch_Forest,
WP,
CR_W,
area_Forest,
Tlife
Standard Deviations (σ )
0.8
0.6
OC_D
area_OuterCoast
0.4
0.2
0
0
1
2
3
4
5
Estimated means (µ)
6
7
8
-16
x 10
Figure 7.6 The results of the Morris method for the concentration of Cl-36 in the water
of the outer coast area.
44
local on output ConcForest
Acatch Forest
Kd soil
area Forest
WP
CR W
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 7.7. The local sensitivity indices for the concentration of Cl-36 in the forest soil .
local on output ConcLake
Flowrate Eurajoki
area Forest
Kd soil
Acatch Forest
WP
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 7.8 The local sensitivity indices for the concentration of Cl-36 in water of Lake
2.
45
local on output ConcCoast
area OuterCoast
OC D
area Forest
Kd soil
Acatch Forest
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 7.9 The local sensitivity indices for the concentration of Cl-36 in water of the
outer coast area.
Table 7.4 The most influential parameters as indicated by the screening methods.
Output:
Parameters:
Conc. In forest soil
1) Forest catchment area (‘Acatch_Forest’)
2) Distribution coefficient soil (‘Kd_Soil’)
Conc. In lake water
1) Yearly average flowrate of Eurajoki River (‘Flowrate_Eurajoki’)
2) Distribution coefficient soil (‘Kd_Soil’)
3) Yearly production of tree wood (‘WP’)
4) Forest catchment area (‘Acatch_Forest’)
5) Conc. ratio of nuclide from soil to tree wood (‘CR_W’)
6) Forest area (‘area_Forest’)
Conc. In coast water
1) Average depth of outer coast (‘OC_D’)
2) Area of outer coast (‘area_OuterCoast’)
3) Distribution coefficient soil (‘Kd_Soil’)
4) Forest catchment area (‘Acatch_Forest’)
5) Yearly production of tree wood (‘WP’)
6) Conc. ratio of nuclide from soil to tree wood (‘CR_W’)
7) Forest area (‘area_Forest’)
8) Average tree lifetime (‘Tlife’)
46
7.4.2
Results of the sampling based methods
A total of 10000 simulations were carried out with samples from all parameters generated using Latin Hypercube Sampling. LHS evaluations are presented in figures 7.167.24. They are further discussed together with the results from other methods in section
7.4.4 below. In figures 7.20, 7.22 and 7.24, the data plotted are the ranked values.
It is worth noticing that in all applied methods, the whole set of parameters listed in tables 7.1-7.3 were used, to be able to study how the results compare with the screening
methods. In a real assessment, only the nine most influential parameters from table 7.4
might have been used for further analysis.
Figure 7.16 Histogram-plot from EIKOS showing the distribution of the concentration
of Cl-36 in forest soil (at t=1000 years).
47
Figure 7.17 Histogram-plot from EIKOS showing the distribution of the concentration
of Cl-36 in the water of Lake 2 (at t=1000 years).
Figure 7.18 Histogram-plot from EIKOS showing the distribution of the concentration
of Cl-36 in the water of the outer coast area (at t=1000 years).
48
SRRC on output ConcForest
Kd soil
Acatch Forest
WP
theta
CR W
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 7.19 Sensitivity indices (using SRRC) for the concentration of Cl-36 in the forest
soil
.
10000
9000
8000
7000
6000
t
s
er
o
F
c
n
o
C
5000
4000
3000
2000
1000
1000
2000
3000
4000
5000
Kd soil
6000
7000
8000
9000
Figure 7.20 The concentration of Cl-36 in the forest soil vs. Kd_Soil.
10000
49
SRRC on output ConcLake
Flowrate Eurajoki
WP
Kd soil
Acatch Forest
CR W
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 7.21 Sensitivity indices (using SRRC) for the concentration of Cl-36 in the water
of Lake 2.
10000
9000
8000
7000
6000
e
k
a
L
c
n
o
C
5000
4000
3000
2000
1000
1000
2000
3000
4000
5000
6000
Flowrate Eurajoki
7000
8000
9000
10000
Figure 7.22 The concentration of Cl-36 in the water of Lake 2 vs. the Eurajoki flowrate.
50
SRRC on output ConcCoast
OC D
area OuterCoast
WP
Kd soil
CR W
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 7.23 Sensitivity indices (using SRRC) for the concentration of Cl-36 in the water
of the outer coast area.
10000
9000
8000
7000
6000
t
s
a
o
C
c
n
o
C
5000
4000
3000
2000
1000
1000
2000
3000
4000
5000
6000
area OuterCoast
7000
8000
9000
10000
Figure 7.24 The concentration of Cl-36 in the water of the outer coast area vs. the
outer coast mean depth.
51
7.4.3
Results of the Variance based methods
The results of the EFAST evaluations are presented in figures 7.25-7.27. They are further discussed together with the results from other methods in section 7.4.4 below. A
total of 8946 model evaluations were performed.
The results of the Sobol evaluations are presented in figures 7.28-7.30. They are further
discussed together with the results from other methods in section 7.4.4 below. A total of
10000 model evaluations were performed.
The results of the Jansen evaluations are presented in figures 7.31-7.33. They are further
discussed together with the results from other methods in section 7.4.4 below. A total of
9000 model evaluations were performed.
TSI on output ConcForest
Acatch_Forest
Kd_soil
area_OuterCoast
CR_W
WP
Tlife
Flowrate_Eurajoki
area Forest
OC_D
0
0,2
0,4
0,6
0,8
Figure 7.25 Total sensitivity indices (using EFAST) for the concentration of Cl-36 in
forest soil.
52
TSI on output ConcLake
Flowrate_Eurajoki
Kd_soil
WP
Acatch_Forest
CR_W
area Forest
Tlife
OC_D
area_OuterCoast
0
0,2
0,4
0,6
0,8
Figure 7.26 Total sensitivity indices (using EFAST) for the concentration of Cl-36 in
the water of Lake 2.
TSI on output ConcCoast
OC_D
area_OuterCoast
Kd_soil
Acatch_Forest
WP
CR_W
area Forest
Tlife
Flowrate_Eurajoki
0
0,2
0,4
0,6
0,8
Figure 7.27 Total sensitivity indices (using EFAST) for the concentration of Cl-36 in
the water of the outer coast area.
53
TSI on output ConcForest
Acatch_Forest
Kd_soil
CR_W
WP
Tlife
area Forest
Flowrate_Eurajoki
OC_D
area_OuterCoast
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 7.28 Total sensitivity indices (using Sobol) for the concentration of Cl-36 in the
forest soil.
TSI on output ConcLake
Flowrate_Eurajoki
Kd_soil
WP
Acatch_Forest
area Forest
CR_W
Tlife
OC_D
area_OuterCoast
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Figure 7.29 Total sensitivity indices (using Sobol) for the concentration of Cl-36 in the
water of Lake 2.
54
TSI on output ConcCoast
OC_D
area_OuterCoast
Acatch_Forest
Kd_soil
WP
CR_W
areaForest
Tlife
Flowrate_Eurajoki
0
0,2
0,4
0,6
0,8
Figure 7.30 Total sensitivity indices (using Sobol) for the concentration of Cl-36 in the
water of the outer coast area.
TSI on output ConcForest
Acatch_Forest
Kd_soil
WP
CR_W
area Forest
Tlife
Flowrate_Eurajoki
OC_D
area_OuterCoast
0
0,2
0,4
0,6
0,8
Figure 7.31 Total sensitivity indices (using Jansen) for the concentration of Cl-36 in the
forest soil.
55
TSI on output ConcLake
Flowrate_Eurajoki
Kd_soil
WP
Acatch_Forest
CR_W
area Forest
Tlife
OC_D
area_OuterCoast
0
0,2
0,4
0,6
0,8
Figure 7.32 Total sensitivity indices (using Jansen) for the concentration of Cl-36 in the
water of Lake 2.
TSI on output ConcCoast
OC_D
area_OuterCoast
Acatch_Forest
Kd_soil
WP
CR_W
area Forest
Tlife
Flowrate_Eurajoki
0
0,2
0,4
0,6
0,8
Figure 7.33 Total sensitivity indices (using Jansen) for the concentration of Cl-36 in the
water of the outer coast area.
56
7.4.4
Conclusions of the sensitivity analysis
The results obtained with the different sensitivity analysis methods used are summarized in tables 7.5-7.7. The quantitative contribution of the parameters to the output
variance is given in percentages (for the variance based methods). The parameters contributing less than 2 % were not included in the tables.
The distribution coefficient in forest soil and the catchment area of the forest are the
overall dominating parameters for the concentration in the forest soil. It can be seen
from Table 7.5 that the order of importance of the distribution coefficient and catchment
area are reversed for the sampling based method (with Latin Hypercube Sampling), although the difference between them is small, as shown by the variance-based methods.
For the variance-based methods, the agreement is very good, with the minor difference
being that EFAST indicates a somewhat lower overall contribution to the output variance. It was realized during the calculations that when EFAST is applied to models with
many input parameters, the noise level is high compared to the methods of Sobol and
Jansen. This explains why the area of the outer coast contributes (3%) to the uncertainty
of the estimate of the concentration in forest soil. As the area of the outer coast cannot
influence at all on the concentration in soil, this result indicates that the noise level of
EFAST is higher than for the other methods. This is probably caused by the presence of
interference in the sampling scheme of EFAST, and should be further investigated.
From Table 7.6 it can be seen that the overall agreement between methods is also very
good for the concentration in water of Lake 2, but still the noise-level for EFAST is
somewhat high. For the concentration in the water of Lake 2, the flow rate of the Eurajoki River is totally dominating. This is expected since the only release of radionuclides
into the water of Lake 2 is directly from the forest, and the flow rate of the Eurajoki
River, used to calculate the water retention time in the lake, is much higher than the runoff from the surrounding catchment areas of Lake 2.
Table 7.5 Ranking of the parameters with significant influence on the concentration in
the Forest soil by the different SA methods used.
Sampling based (LHS)
EFAST
Sobol
Jansen
Kd soil
Acatch Forest
-
Acatch Forest [44%]
Kd soil [42%]
area_OuterCoast
[3%]
Acatch Forest [52%]
Kd soil [44%]
-
Acatch Forest [51%]
Kd soil [45%]
-
Table 7.6 Ranking of the parameters with significant influence on the concentration in
the water of Lake 2 by the different SA methods used.
Probabilistic (LHS)
EFAST
Sobol
Jansen
Flowrate Eurajoki
Flowrate Eurajoki
[78%]
Kd soil [7%]
WP [5%]
Acatch Forest [3%]
CR_W [3%]
Flowrate Eurajoki
[77%]
Kd soil [11%]
WP [5%]
Acatch Forest [4%]
-
Flowrate Eurajoki
[73%]
Kd soil [7%]
WP [7%]
Acatch Forest [7%]
CR_W [4%]
WP
Kd soil
Acatch Forest
CR_W
57
Table 7.7 Ranking of the parameters with significant influence on the concentration in
the water of the outer coast by the different SA methods used.
Probabilistic (LHS) EFAST
Sobol
Jansen
OC_D
area_OuterCoast
WP
Kd soil
CR_W
-
OC_D [28%]
area_OuterCoast [26%]
Acatch Forest [11%]
Kd soil [11%]
WP [8%]
CR_W [8%]
area Forest [6%]
OC_D [26%]
area_OuterCoast [21%]
Acatch Forest [14%]
Kd soil [14%]
WP [13%]
CR_W [8%]
area Forest [4%]
OC_D [28%]
area_OuterCoast [21%]
Kd soil [14%]
Acatch Forest [13%]
WP [12%]
CR_W [6%]
area Forest [5%]
The concentration in the water of the outer coast is the only output that can be dependant on any parameter of the model, since the radionuclides flow directly with water
from the release point in the forest to the coast, via the network of rivers and lakes. This
explains why a larger number of parameters contribute to this output (Table 7.7). Yet,
almost perfect agreement is obtained between the variance based methods.
Improvements to the results could be made by increasing the number of total model
evaluations for the different methods used, thereby minimizing any statistical error introduced. Still, the obtained correlation between the screening methods and the sampling- and variance-based methods was very good.
The Morris method has shown being a very robust screening method, and therefore the
results could be improved by applying the variance based methods only to those parameters indicated by Morris as important. This is because the parameters that are not
indicated as important by the Morris method will only increase the overall statistical
noise level in the variance based methods.
Further, in this test case no selective mapping was made between input parameters and
outputs, meaning that the analyses included irrelevant parameters for several model outputs (for instance the area of the outer coast was considered in the SA for the concentration in the forest soil). Had such a selective mapping been made, the noise-level might
have been lower and the results, especially for the EFAST method, would have been
better.
58
59
8
CONCLUDING REMARKS
Overall, sensitivity analysis provides a way to identify the model inputs that have the
strongest effect on the uncertainty in the model predictions. However, sensitivity analysis does not provide an explanation for such effects. This explanation must come from
the analysts involved and, of course, be based on the mathematical properties of the
model under consideration.
Sensitivity analysis methods are essential tools in risk assessment and simulation
modelling. Due to difficulties of implementing the methods, most sensitivity analyses
are today performed using local methods, correlation or regression coefficients. The
variance-based methods described here are recommended for studies of environmental
risk assessment models, since these are often non-linear and can be also non-monotonic.
The test application has shown that EIKOS is an easy to use and robust tool for
performing SA studies of models developed with Pandora. For this test, the model
created in Pandora was simply loaded into EIKOS, and all SA and simulation settings,
as well as the parameter management, was handled from within EIKOS. A full
assessment could also be saved from within EIKOS and later opened to reproduce the
results.
Also the landscape modelling procedure was found technically feasible and the workflow combining Pandora and EIKOS seen well controlled, although some streamlining
is recommended in future.
60
61
REFERENCES
[Avi06] R. Avila. Model of the long-term transfer of radionuclides in forest. Svensk
Kärnbränslehantering Ab, SKB Technical report, in print.
[CFS73] R. I. Cukier, C. M. Fortuin, Kurt E. Shuler, A. G. Petschek, and J. H. Schaibly.
Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients.
I theory. The Journal of Chemical Physics, 59(8):3873–3878, October 1973.
[Con99] William Jay Conover. Practical Nonparametric Statistics. John Wiley & Sons,
Inc., 3rd edition, 1999.
[CSS75] R. I. Cukier, J. H. Schaibly, and Kurt E. Shuler. Study of the sensitivity of
coupled reaction systems to uncertainties in rate coefficients. III analysis of the approximations. The Journal of Chemical Physics, 63(3):1140–1149, August 1975.
[CST00] Karen Chan, Andrea Saltelli, and Stefano Tarantola. Winding stairs: A sampling tool to compute sensitivity indices. Statistics and Computing, 10(3):187–196, July
2000.
[CTSS00] Karen Chan, Stefano Tarantola, Andrea Saltelli, and Ilya M. Sobol0. Variance-based methods. In Sensitivity analysis, Wiley Ser. Probab. Stat., pages 167–197.
Wiley, Chichester, 2000.
[HS96] Toshimitsu Homma and Andrea Saltelli. Importance measure in global sensitivity analysis of nonlinear models. Reliability Engineering and System Safety, 52(1):1–7,
1996.
[IC82] Ronald L. Iman and W. J. Conover. A distributionfree approach to inducing rank
correlation among input variables. Communications in Statistics: Simulation and Computations, 11(3):311–334, 1982.
[JE05] Jonsson, K. and Elert, M. Model for radionuclide transport in running waters.
Svensk Kärnbränslehantering Ab, SKB TR 05-03.
[JRD94] M.J.W. Jansen, W.A.H. Rossing, and R.A. Daamen. Monte Carlo estimation
of uncertainty contributions from several independent multivariate sources. Predictability and Nonlinear Modelling in Natural Sciences and Economics, pages 334–343, 1994.
[KB00] Karlsson, S. and Bergström, U. Dose rate estimates for the Olkiluoto site using
the biospheric models of SR 97. Posiva Working report 2000-20.
[Mor91] Max D. Morris. Parameterial sampling plans for preliminary computational
experiments. Technometrics, 33(2):161–174, May 1991.
62
[MTS82] Gregory J. McRae, James W. Tilden, and John H. Seinfeld. Global sensitivity
analysis - a computational implementation of the Fourier amplitude sensitivity test
(FAST). Computers and Chemical Engineering, 6(1):15–25, 1982.
[PG05] Åstrand, P-G., Jones, J., Broed, R. & Avila, R. Pandora technical description
and user guide. Posiva Working report 2005-64.
[RAI05] Rautio, P., Aro, L. & Ikonen, A. T. K. Terrain development at Olkiluoto site
and implications to biosphere assessment. Proceedings of the 2nd International Conference on Radioactivity in the Environment, Nice, October 2-6, 2005. Norwegian Radiation Protection Authority (printed by). pp. 372-375.
[Sal00] Andrea Saltelli. What is sensitivity analysis? In Sensitivity analysis, Wiley Ser.
Probab. Stat., pages 3–13. Wiley, Chichester, 2000.
[SB98] Andrea Saltelli and Ricardo Bolado. An alternative way to compute Fourier amplitude sensitivity test (FAST). Computational Statistics and Data Analysis, 26:445–
460, 1998.
[SH91] Andrea Saltelli and T Homma. Lisa package user’s guide, part III: SPOP (Statistical POst Processor) uncertainty and sensitivity analysis for model output program
description and user guide. Technical report, CEC/JRC Nuclear Science and Technology, 1991.
[SM90] Andrea Saltelli and J Marivoet. Non-parametric statistics in sensitivity analysis
for model output: A comparison of selected techniques. Reliability Engineering and
System Safety, 28:229–253, 1990.
[Sob93] I. M. Sobol0. Sensitivity analysis for nonlinear mathematical models. Mathematical Modeling and Computational Experiment, 1(4):407–414, 1993.
[Sob01] I. M. Sobol0. Global sensitivity indices for nonlinear mathematical models and
their Monte Carlo estimates. Math. Comput. Simulation, 55(1-3):271–280, 2001. The
Second IMACS Seminar on Monte Carlo Methods (Varna, 1999).
[SS73] J. H. Schaibly and Kurt E. Shuler. Study of the sensitivity of coupled reaction
systems to uncertainties in rate coefficients. II applications. The Journal of Chemical
Physics, 59(8):3879–3888, October 1973.
[STC99] Andrea Saltelli, Stefano Tarantola, and Karen Chan. A quantitative modelindependent method for global sensitivity analysis of model output. Technometrics,
41(1):39–56, February 1999.
[STCR04] Andrea Saltelli, Stefano Tarantola, Francesca Campolongo, and Marco
Ratto. Sensitivity analysis in practice. John Wiley & Sons Ltd., Chichester, 2004. A
guide to assessing scientific models.
63
[Wey38] H. Weyl. Mean motion. American Journal of Mathematics, 60(4):889–896,
October 1938.
64
65
APPENDIX 1: COAST MODEL APPLIED
In this appendix, the compartment model, equations and parameters of the coast model
used in the exercise are described as implemented for the landscape model analysed.
The coast model is adopted without changes from [KB00].
Inflow
4
Water
bay
Water
sea
5
1
2
Top
Sediment
bay
6
7
Upper sediment
sea
3
Deep
Sediment
bay
Figure A-1.1. Compartment model for the coast model.
8
Deep
Sediment
sea
Outflow
66
Table A-1.1. Equations used in the coast model.
Transfer Nr.:
1, 6
Transfer Coefficient (Bq/Yr):
Resusp
ρ sed
2, 7
3, 8
Kd ⋅ SR
D ⋅ (1 + Kd ⋅ Susp )
SR − Resusp
ρ sed
4
5
1
Re tTime
V
1
⋅ bay
Re tTime VSea
Table A-1.2. Parameters used in the coast model.
Parameter Name
Resusp
Description
Resuspended fraction
Mass of upper sediment
Kd
SR
D
Susp
RetTime
Distribution factor
Gross sediment rate
Mean depth
Concentration of suspended matter in water
Retention Time for inner coast water (bay)
Water volume of sea (outer coast)
ρ sed
VSea
Vbay
Water volume of bay (inner coast)
67
APPENDIX 2: LAKE MODEL APPLIED
In this appendix, the compartment model, equations and parameters of the lake model
used in the exercise are described as implemented for the landscape model analysed.
The lake model is adopted without changes from [KB00].
Water
Inflow
Outflow
4
1
2
Top
Sediment
3
Deep
Sediment
Figure A-2.1. Compartment model for the lake model.
68
Table A-2.1. Equations used in the lake model.
Transfer Nr.:
1
Transfer Coefficient (Bq/Yr):
Resusp
ρ sed
2
3
Kd ⋅ SR
D ⋅ (1 + Kd ⋅ Susp )
SR − Resusp
ρ sed
4
1
Re tTime
Table A-2.2. Parameters used in the lake model.
Parameter Name
Resusp
Description
Resuspended fraction
Mass of upper sediment
Kd
SR
D
Susp
RetTime
Distribution factor
Gross sedimentation rate
Mean depth
Concentration of suspended matter in water
Retention time in lake
ρ sed
69
APPENDIX 3: RIVER MODEL APPLIED
In this appendix, the compartment model, equations and parameters of the river model
used in the exercise are described as implemented for the landscape model analysed.
The river model is adopted without changes from [JE05].
Biota
1
Inflow
2
Water,
Dissolved
7
Water,
Adsorbed
Flooding
8
3
4
Surface Sediment
5
Outflow
9
10
Sediment Transport
6
Deep Sediment
Inflow
Figure A-3.1. Compartment model for the river model.
70
Table A-3.1. Equations used in the river model.
Transfer Nr.:
1
Transfer Coefficient (Bq/Yr):
2
k biota
3
V z P∆x
2 DηP∆x
+
∆z1 (1 + K B ρ sed )∀ sed ,1 2 (1 + K B ρ sed )∀ sed ,1
4
V
2 Dη
+ z
∆z1 Rh 2 Rh
5
BCF
M biomass , tot
kbiota
∀ water
2 DηAdeep
∆z 2 (1 + K B ρ sed )∀ sed , 2
6
2 DηAdeep
∆z1 (1 + K B ρ sed )∀ sed ,1
V z Adeep
+
2 (1 + K B ρ sed )∀ sed , 2
+
V z Adeep
2 (1 + K B ρ sed )∀ sed ,1
7
ln(2)
Tk c p K d
8
ln(2)
Tk
K d c pV partsed
9
(1 + K c )R
d
p
h
10
Resuspension K B
(1 + K B ρ sed )∀ sed ,1
Outflow
Vadv
∆x
qb w
Sediment
Transport
Flooding
⎛
1
∀ sed ,1 ⎜⎜1 +
⎝ ρ sed K B
q flooded water
∆x Across
⎞
⎟⎟
⎠
71
Table A-3.2. Parameters used in the river model. Some of the parameters in table A-3.2
is used to calculate other parameters, or are calculated from other parameters. For full
model-description see [JE05].
Parameter Name
Kd
KB
Tk
BCF
qs
Aws
α
Sb
∆x
ymax
ρsed
Vz
D50, D90
cp
η
D
∆zi
Mbiomass
n
∀ sed ,1
Description
Distribution coefficient in the stream water
Distribution coefficient in the sediment
Half-time to reach sorption equilibrium
Bioconcentration factor
Specific run-off
Watershed area
Cross-sectional angle
Slope of the channel
Length of the channel
Maximum depth in main channel
Sediment density
Advective transport velocity in bed sediment
Particle size distribution in the sediment
Suspended particulate matter in stream water
Sediment porosity
Diffusion coefficient
Depth of surface and deep sediment
Plant biomass
Manning friction coefficient
Volume of upper sediment
∀ sed , 2
Volume of deep sediment
∀ water
Volume of water
Across
Adeep
Cross-sectional area of water stream
Area of deep sediment equal to area of upper
sediment
Advective velocity
Resuspension factor
Hydraulic radius
Yearly average of flooded water flowing out of
water stream
Sedimentation velocity
Vadv
Resuspension
Rh
qflooded water
Vpartsed
72
73
APPENDIX 4: FOREST MODEL APPLIED
In this appendix, the compartment model, equations and parameters of the forest model
used in the exercise are described as implemented for the landscape model analysed.
The forest model from [Avi06] is modified to distribute the input over the full forest
area instead of calculating the activity per square meter as originally.
2
Leaves
4
1
Wood
Fauna
5
Litter
Understorey
6
7
3
Soil
Input
Figure A-4.1. Compartment model for the forest model.
Outflow
74
Table A-4.1. Equations used in the forest model.
Transfer Nr.:
1
2
3
4
5
6
Outflow
Transfer Coefficient (Bq/Yr):
CRW
ρ ⋅h
CR
LP ⋅ L
ρ ⋅h
CR
UP ⋅ U
ρ ⋅h
TC LToLi
TCWToLi
TCUToLi
P − ET
h ⋅ (θ + Kd ⋅ ρ )
WP ⋅
Table A-4.2. Parameters used in the forest model. Some of the parameters in table A4.2 is used to calculate other parameters. For full model-description see [Avi06].
Parameter
Name
WP
CRW
LP
CRL
UP
CRU
TCLToLi
TCWToLi
TCUToLi
ρ
H
P
ET
θ
Kd
Description
Yearly production of tree wood
Concentration ratio of nuclides from soil to tree wood
Yearly production of tree leaves
Concentration ratio of nuclides from soil to leaves
Yearly production of understorey plants
Concentration ratio of nuclides from soil to understorey plants
Yearly fractional loss of tree leaves biomass
Yearly fractional loss of tree wood biomass
Yearly fractional loss of understorey plants biomass
Soil bulk density
Thickness of the soil rooting layer
Precipitation rate
Area normalised evapotranspiration rate
Volumetric water content in soil
Distribution factor in soil
75
APPENDIX 5: Reproducing the results obtained in section 7.
Version of software used:
Matlab Ver. 7.1.0.246 (R14) Service Pack 3
Simulink Ver. 6.3 (R14SP3)
Pandora Ver 1.0
EIKOS Ver. 2.0
Required files:
Apart from the software packages listed above, the following files are required and must
be on the Matlab path:
TestCase.mdl
ConcEnd.m
Landscape2000AP.m
the model-file
to extract the solution at t=tfinal
to allow for post-processing within EIKOS (temporary solution only)
Step-by-step guide
First the model created in Pandora is opened with EIKOS. At the Matlab prompt, type:
> eikos2(Landscape2000AP)
Note: the argument (‘Landscape2000AP’) is only necessary to specify for this example
case. For other cases, EIKOS is started by simply typing ‘eikos2’ at the Matlab prompt.
In the following dialog-window, locate the model file (‘TestCase.mdl’) and open it. In
EIKOS we are then given an overview of the model settings, and inputs and outputs
(Fig. A-5.1). Depending on what method is to be performed, select the parameters to be
included in the analysis either from Table 7.1. (the parameters included in the screening) or from Table 7.2 (the parameters selected for detailed analysis). The selection of
inputs/outputs is done by selecting the “Parameters/Outputs” button (Fig. A-5.2).
After this, it is necessary to update some of the assigned probability density functions
(and min/max values if a method requiring it is to be applied), and this is done by selecting the “Parameter Settings” option (Fig. A-5.3). The parameter data is then updated
using the data listed in table 7.1.
The next step is the selection of sensitivity analysis method. The method and methodspecific settings are accessed by selecting the “Select SA Method” option (Fig. A-5.4).
Once the required method and settings have been applied, the model can be simulated
by selecting the “Simulate” option. After the simulation is finished, the results can be
viewed graphically by clicking on the “Results” button or choosing any of the plot
methods in the “Results” menu. It is also possible to view the results numerically by
clicking on any of the four buttons below EIKOS main window (Summary, Input Data,
Output Data and Results).
Figure A-5.1. EIKOS main window showing model information.
76
77
Figure A-5.2. The Parameters/Outputs selection window.
Figure A-5.3. The parameter settings window in EIKOS.
Figure A-5.4. Sensitivity Analysis Method window.