Uncertainty
Risk
Typology
Sources
Sensitivity
Risk Assessment
16. Uncertainty and Sensitivity
Marvin Rausand
[email protected]
RAMS Group
Department of Production and Quality Engineering
NTNU
(Version 0.11)
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
1 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
M ARVIN R AUSAND
Risk Assessment
Theory, Methods, and Applications
Slides related to the book
Risk Assessment
Theory, Methods, and Applications
Wiley, 2011
STATISTICS IN PRACTICE
Homepage of the book:
http://www.ntnu.edu/ross/
books/risk
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
2 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Prologue
“There are known knowns. These are things that we know that we know.
There are known unknowns. That is to say, there are things that we know we
don’t know. But there are also unknown unknowns. These are things we
don’t know that we don’t know.”
Rumsfeld (2002)
We may also add a fourth category, unknown knowns, to represent hidden knowledge, for
example, that a hazard is known to one department of a company but not to another.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
3 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Important developments
I
The Reactor Safety Study (NUREG-75/014, 1975) studied the effects of
uncertainty of input parameters by using the Monte Carlo simulation
program SAMPLE
I
Morgan, M.G. and M. Henrion (1990). Uncertainty: A Guide to Dealing
with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge
University Press.
I
NUREG-1855 (2009): Guidance on the Treatment of Uncertainties
Associated with PRAs in Risk-Informed Decision-Making
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
4 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Risk and uncertainty
The concepts of risk and uncertainty are intimately linked:
Risk occurs because the future is uncertain!
If the future were certain (i.e., deterministic), no risk would exist.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
5 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Our uncertain future
The uncertainty about the future can be classified in four categories:
1. A clear enough future: We can forecast a future that is “close enough”
for the situation at hand
2. Alternative futures: We can define a limited set of possible futures, one
of which will occur
3. A range of futures: We can define a range of possible future outcomes
4. True ambiguity: We cannot even define a range of possible future
outcomes (i.e., the Black swan category)
Courtney (2003)
Only categories 1 and 2 can be analyzed adequately by the methods
presented in the book.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
6 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Definition of uncertainty
General
1. Uncertainty is about not knowing for sure.
2. A state of incomplete knowledge
Grote (2009)
Cullen and Frey (1999)
3. Any departure from the unachievable ideal of complete determinism.
Walker et al. (2003)
4. A person’s degree of belief related to the truth of a statement or event.
Lindley (2006)
5. Lack of knowledge about unknown quantities.
6. A degree of ignorance
Marvin Rausand (RAMS Group)
Aven (2011)
Beven (2009)
Risk Assessment
(Version 0.11)
7 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Definition of uncertainty
Related to risk assessment
1. Measure of the confidence we have in the results of the risk assessment.
Rausand (2011)
2. Lack of certainty with respect to. (i) the future state and behavior of
the system and (ii) the soundness and relevance of the results of the
risk assessment in a specific decision context.
Johansen (2014)
“Uncertainty does not reside in the world itself, but in the imperfect quality of
our knowledge about the world” (NENT, 2009). Uncertainty is therefore an
epistemological concept.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
8 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
What is risk? - 1
The concept risk has two dimensions:
1. The likelihood that harmful events will occur
2. The consequences of these events
The likelihood (p) of a harmful event is most often quantified as a
probability or a degree of belief, but p may also be specified by using
possibility theory, fuzzy set theory, evidence theory (Dempster-Schafer),
plausibility theory, etc.
The consequences (C) of a harmful event will usually be undesirable and
may be quantified by the magnitude of the harm to one or more assets. The
event may harm several types of assets (e.g., people, the environment, and
material assets) and C will therefore be multi-dimensional.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
9 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
What is risk? - 2
As several harmful events may occur, each with a likelihood pi and a
consequence Ci , for i = 1, 2, . . ., the risk spectrum 1 may be represented as
R = hpi , Ci i
p1
p2
Activity
C1
C2
pn
Cn
1 Also
called the risk picture.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
10 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
What is risk? - 3
A more operational definition of risk is to focus on intermediate events that
are called hazardous events, HE. An example of such an HE is a gas leak that
may or may not give harm to assets – depending on whether safety barriers
are functioning or not. Risk is then defined by the three questions:
1. Which hazardous events (HEs) can happen?
2. What is the likelihood (p) that each HE will happen?
3. What are the consequences (C) of each HE?
Since the consequence of HEi depends on the correct functioning of one or
more safety barriers, each will be a spectrum of possible consequences Ci,j
that will occur with certain likelihoods pi,j such that
(
)
Ci = hpi,j Ci,j i
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
11 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Bow-tie model
Hazardous
event
Proactive
barriers
Marvin Rausand (RAMS Group)
Consequences
Hazards / Threats
The operational definition of risk is in line with the bow-tie model for risk
analyses.
Reactive
barriers
Risk Assessment
(Version 0.11)
12 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
What is risk? - 4
By considering the bow-ties for all relevant hazardous events, HEi , for
i = 1, 2, . . . , n, we may represent the risk as the set of triplets
R = hHEi , pi , Ci i
This set may be rearranged to obtain the risk spectrum presented above.
“Risk is distinguished from uncertainty by postulating that risk is measurable,
while uncertainty is not.”
Knight (1921)
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
13 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Classification of uncertainty
The uncertainty of a risk assessment can be classified in several ways.
A typology of uncertainty is presented in the following.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
14 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Knowledge uncertainty - 1
Z Knowledge uncertainty: Uncertainty due to lack of knowledge about
specific factors, parameters, or models related to the system or the
environment.
Knowledge uncertainty is also called epistemic uncertainty,2 subjective
uncertainty, and reducible uncertainty.
I
Knowledge uncertainty can, for example, come from inadequate
physical models used to describe a structural response.
I
Knowledge uncertainty can, in principle, be reduced by acquiring
additional information of the study object and its operation.
2 Knowledge
uncertainty is called epistemic uncertainty in the book.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
15 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Knowledge uncertainty - 2
Knowledge uncertainty cannot be adequately described by a
classical/frequentist probability distribution, but some authors (e.g., Lindley,
2007) claim that the Bayesian framework can be used.
Several authors claim that the Bayesian framework is not applicable and
that we need to use other representations, such as possibility theory, fuzzy
set theory, evidence theory, and so on.
e.g., see Dubois (2010)
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
16 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Metaphors
Two types of knowledge uncertainty:
Z Black swan: An event that has never been seen before and is truly
unimaginable
Black swans constitute the most extreme form of knowledge uncertainty.
Z Perfect storm: A very rare conjunction of known events
Perfect storms can be anticipated and their probabilities can be assessed
based on systematic risk analyses based on historical data and fundamental
knowledge.
See Paté-Cornell (2012)
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
17 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Variability - 1
Z Variability: Uncertainty caused by natural variation and randomness,
such as inherent fluctuations or differences in a quantity or a process,
within or between time, location, or category/group.
Variability is also called aleatory uncertainty,3 random uncertainty, inherent
uncertainty, and irreducible uncertainty.
This is a known known uncertainty. The randomness is understood and
characterized, for example, by a probability distribution (mainly within the
Bayesian framework).
Variability can be characterized but not reduced with additional data.
3 Variability
is called aleatory uncertainty in the book
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
18 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Variability - 2
Variability may (as indicated above) be classified as:
1. Inter-individual variability (e.g., an individual’s robustness against
stressors varies due to personal properties and protection equipment)
2. Spacial variability (e.g., damage to an individual varies according to
where the person is located)
3. Temporal variability (e.g., the number of persons exposed to a hazard
may vary over time)
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
19 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Indeterminism
A risk analysis is based on assumptions and prediction about the future
behavior of technical systems and personnel.
but, “what we know influences how we behave.”
“[. . . ] because tomorrow’s discovery is by definition unknown today,
tomorrow’s behavior is not entirely predictable today.”
This is called Shackle-Popper Indeterminism by Ben-Haim (2012).
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
20 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Measurement uncertainty
Z Measurement uncertainty: Uncertainty in field or laboratory data on
which models are based.
Measurement uncertainty may be classified as
1. Lack of precision
2. Inaccuracy
3. Sampling and analysis errors
Measurement uncertainty is usually not a significant contributor to the
uncertainty of risk assessments.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
21 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Parameter uncertainty
Z Parameter uncertainty: Uncertainty related to the numerical values of
the input parameters to the risk analysis models, such as failure rates,
repair times, leak rates, strengths, and so on.
The effect of parameter uncertainty can be assessed, for example, by
uncertainty propagation performed by Monte Carlo simulation (see below).
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
22 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Model uncertainty
Z Model uncertainty: Uncertainty related to the (simplified) structural and
mathematical representations of the conceptual models and the imprecision
in numerical solutions implicit in mathematical models.
Model uncertainty may be assessed by comparisons between alternative
models and between model predictions and empirical observations.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
23 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Completeness uncertainty
Z Completeness uncertainty: Uncertainty due to unidentified holes and
flaws in our understanding. We do not know how many hazards there are,
and may even be uncertain about the types of hazards that we may be
running.
Completeness uncertainty is also called scenario uncertainty and represent
unknown unknowns.
Completeness uncertainty can be considered to be a subcategory of
knowledge uncertainty.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
24 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Linguistic uncertainty
Some authors4 include linguistic uncertainty as a fourth category. Linguistic
uncertainty can be classified in fours categories:
1. Ambiguity – arises when words have more than one meaning and it is
not clear which one is meant
2. Context dependence – Caused by a failure to specify the context in
which a term is to be understood: e.g., “large scale escape”
3. Underspecificity – occurs when there is unwanted generality
4. Vagueness – arises when terms allow borderline cases: “medium risk”
4 e.g.,
see K.R. Hayes (2009)
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
25 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Contextual uncertainty
Z Contextual uncertainty: Uncertainty related to the social, political, and
ethical context in which the system is going to be operating in. This
uncertainty also applies to the stakeholders who are going to evaluate and
accept the results from the risk assessment.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
26 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Uncertainty due to errors
Z Errors: Recognizable deficiencies in modeling or calculations.
Errors of this category are:
I
Not due to lack of knowledge, but are slips, lapses, or mistakes made
by the analysts.
I
Not identified through the validation and verification processes
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
27 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
NUREG-1855
NUREG-1855 defines three categories of knowledge (i.e., epistemic)
uncertainty.
1. Completeness uncertainty relates to risk contributors that are not
incorporated into the risk model (see above)
2. Model uncertainty can be divided into two subcategories:
•
•
Inadequate model comprehensiveness (i.e., does not take into account all
the variables that can significantly affect the results)
Inadequate model characterization (i.e., inadequate modeling of
structures and interrelations)
3. Parameter uncertainty is the uncertainty in the values of the
parameters of a model given that the mathematical form of that model
has been agreed to be appropriate.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
28 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Three dimensions of uncertainty
1. The location of uncertainty – where the uncertainty manifests itself in
the model complex
2. The level of uncertainty – where the uncertainty manifests itself along
the spectrum between deterministic knowledge and total ignorance
3. The nature of uncertainty – whether the uncertainty is due to the
imperfection of our knowledge or is due to the inherent variability of
the phenomena being described
Walker et al. (2003)
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
29 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Some specific sources of uncertainty
I
I
I
I
I
I
Poor understanding and characterization of what operators will do in
an accident scenario
Poor understanding of the hazards that might be introduced due to
computer hardware and software faults
Poor quantification of an organization’s safety culture or maintenance
procedures
Poor understanding and modeling of how systematic faults 5 will
influence the reliability of safety barriers
Inadequate understanding and modeling of common cause failures
(CCFs) will influence the risk – and inadequate data to support
CCF-modeling
Inadequate understanding of system complexity and how complexity
should be treated.
5 e.g.,
see IEC 61508
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
30 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Deep uncertainty - 1
The term deep uncertainty was introduced in environmental risk assessment.
Z Deep uncertainty: Deep uncertainty exists when analysts do not know,
or the parties to a decision cannot agree on:
I
The appropriate models to describe the interactions among a system’s
variables,
I
The probability distributions to represent uncertainty about key
variables and parameters in the models, and/or
I
How to value the desirability of alternative outcomes.
“Shaping the next one hundred years,” Lempert et al. (2003)
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
31 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Deep uncertainty - 2
A similar definition for strategic risk assessment may be:
Z Deep uncertainty: Deep uncertainty exists when the risk analysts do not
know, or the central stakeholders to the risk assessment cannot agree on:
I
The accident scenarios that may potentially occur,
I
The likelihood of these scenarios, and/or
I
The possible consequences of the scenarios and/or how to value them
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
32 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Uncertainty propagation
Z Uncertainty propagation: To study how uncertainty in input parameters
to a model influences the uncertainty of output values.
This can be studied by
1. Analytical methods
2. Random sampling (Monte Carlo or Latin hypercube) of the model
Uncertainty propagation is used to quantify parameter uncertainty.
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
33 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Analytical methods
The analytical approach is usually done by approximating a “response” by
Taylor expansion and by linearizing the response.
See section 16.5.1 of the book
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
34 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Monte Carlo simulation - 1
Assume that we have a model such as Z = h(X1 , X2 , . . . , Xm ) and that we
want to study the uncertainty of Z caused by uncertainties of the “input”
variables X1 , X2 , . . . , Xm . Further, assume that we can describe the
uncertainty of Xi by a probability density function fi (x) for 1 = 1, 2, . . . , m.
The following procedure is used:
1. Decide the shape and parameters of fi (x) for each i = 1, 2, . . . , m
2. Draw a value xi for each Xi from the distribution fi (x)
3. Generate an output value z = h(x1 , x2 , . . . , xm )
4. Repeat steps 2 and 3 n times and collect the results
5. Analyze the distribution of the model output values
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
35 / 36
Uncertainty
Risk
Typology
Sources
Sensitivity
Monte Carlo simulation - 2
The process of Monte Carlo simulation is illustrated by:
f(x1)
x1
f(x2)
Model
f(y)
Y= g(X1, X2, X3)
x2
y
f(x3)
x3
Marvin Rausand (RAMS Group)
Risk Assessment
(Version 0.11)
36 / 36