Bayesian uncertainty
Is it useful in metrology?
François O. Bochud
Jean-Pascal Laedermann
Claude J. Bailat
University institute for radiation physics (IRA)
University Hospital and University of Lausanne
Lausanne, Switzerland
1
Heard yesterday at the UCWG meeting
„
„
„
I cannot estimate an uncertainty
if you just provide me with the
data. I need to know "something
else"!
In many cases mathematics
cannot help the evaluation of
uncertainty!
Uncertainty is a lot of guessing!
What is uncertainty?
„
"Parameter characterizing the
dispersion of the quantity values being
attributed to a measurand, based on
the information used" [VIM:2006:2.27]
"the available information is encoded in
terms of probability density functions for
the input quantities" [GUM-S1:2008:intro]
What is probability?
„
„
In scientific language, probability has both a
mathematical and interpretational aspect
Mathematics
Kolmogorov’s
axioms
„
A quasi-consensus exists about the set of axioms
(e.g.) and theorems that probability theory
should satisfy
P (E ) ≥ 0
∀E ⊂ Ω
Ω is the event space
P (Ω) = 1
P (E ) = ∑ P (Ei )
i
Ei pairwise disjoint E = ∑ Ei
i
What is probability?
„
Interpretation
„
„
Wide divergence of opinions
Can be divided into two classes
„
„
Epistemic (επιστήμη) interpretations
Objective interpretations
What is probability?
„
Epistemic views
„
„
„
Probability is
concerned with
human beliefs &
knowledge
Probability
lies in the mind
Example
„
Bayesian
interpretations
„
Objective views
„
„
„
Probability is an
objective feature of
the material world
Probability
lies in nature
Example
„
Frequency
interpretations
Frequentist measurement
Relative frequency of the occurrence of an observable random event in an
infinite series of trials performed under identical physical conditions
Measurand
A
Observations
Ensemble of
outcomes
nA
P ( A ) = lim
N→∞ N
nA : number of trials in which outcome A is observed
N : total number of repeated trials
Bayesian measurement
Degree of belief of a proposition, conditional on all relevant
information that is available about that proposition
Measurand
Observations
All prior knowledge
(experience, previous
measurements, etc...)
a priori information
Decisions
observer
Bayesian measurement
Degree
of belief
Reality
(Measurand)
observation
Plato's
cave
observer
Bayes' Theorem
P (A B) =
P ( A ∩ B ) = P ( A B ) P (B )
P (A ∩ B)
P (B )
P ( A ∩ B ) = P (B A ) P ( A )
Definition of
conditional probability
P (A B) =
P (B A ) P ( A )
AIB
A
B
Ensemble of
outcomes
P (B )
Bayes' theorem and Bayesian stat
„
Bayes' theorem is not the
sole property of Bayesian
statistics
„
It can be used within
the frequentist framework
„
„
Remember your college lectures
However
„
Bayes' theorem is to the theory of probability
what Pythagoras theorem is to geometry
(Jeffreys)
Bayes' theorem & measurement
Physical model
P(x|θ)
Θ
States a priori
X
θ
x
P(θ)
Physical states
observations
P(x,θ)=P(x|θ) P(θ)
Bayes' theorem & measurement
Physical model
P(x|θ)
Θ
States a priori
X
θ
x
P(x,θ)=P(x|θ) P(θ)
P(θ)
Physical states
observations
M(x)
θ
x
P(x,θ)= P(θ|x) M(x)
States a posteriori
P(θ|x)
Observations marginal
Bayes' theorem & measurement
Physical model
States a priori
Most controversial
aspect of Bayes' theory
P(θ|x) = P(x|θ) P(θ) / M(x)
A posteriori
State after observation of the
value x
It is a complete distribution
M( x ) =
∫ P ( x θ) P ( θ)
dθ
Θ
Distribution of the observations
for this model and a priori
(normalization factor)
Prior knowledge
„
What the frequentists say
„
„
„
Priors are subjective
The whole inference processes liable to the
whims of the person that does the evaluation
What the Bayesians answer
„
„
„
Most of the time we have some prior knowledge
Some degree of subjectivity may be desirable
P(θ) should not be chosen arbitrarily
„
Caution must always be exercised
Prior knowledge
„
If we have no info, use non-informative priors
„
For instance with the principle of maximum entropy
[GUM-S1:2008:Table 1]
Example of activity measurement
P(θ,x)
15
P(x|θ)
10
given by the
Poisson law
State θ
Activity [Bq]
20
Joint probability of
observing x counts
with activity θ
5
…
0
0
10
20
30
Observation x
40
50
Number of counts
Example of activity measurement
Actually observed value
P(θ,x)
15
10
State θ
Activity [Bq]
20
Joint probability of
observing x counts
with activity θ
5
0
0
10
20
30
Observation x
40
50
Number of counts
Example of activity measurement
P(θ|x) = P(x|θ) P(θ) / M(x)
x is a fixed parameter
Having observed x counts, what is our
degree of belief about the activity θ?
20
15
State θ
10
5
0
0
10
20
30
Observation x
A priori of the
activity θ
P(θ,x)
40
50
Example of activity measurement
P(θ|x) = P(x|θ) P(θ) / M(x)
a piori
20
a posteriori
P(θ|x)
P(θ,x)
P(θ)
15
State θ
10
5
0
0
10
20
30
Observation x
40
50
Example of activity measurement
The whole
distribution is
available
Expectation of θ
E ⎡⎣θ x ⎤⎦ =
∫ θ P (θ x )
dθ
Θ
Standard-uncertainty
square root the variance of θ
u ( θ ) = V ⎡⎣θ x ⎤⎦
=
∫ ( θ − E ⎡⎣θ x ⎤⎦ ) P ( θ x )
2
Θ
activity
E[θ x ]
dθ
a posteriori
P(θ|x)
Example of activity measurement
„
Coverage interval
Straightforward to obtain
any coverage interval
activity
θup
For instance a 95% interval
θup
∫
P ( θ x ) dθ = 0.95
θlow
(NB : such an interval is not unique)
θlow
a posteriori
P(θ|x)
Example of activity measurement
„
What if we change the a priori?
a piori
20
a posteriori
P(θ|x)
P(θ,x)
P(θ)
15
State θ
10
5
0
0
10
20
30
Observation x
40
50
Example of activity measurement
„
What if we change the a priori?
a piori
20
a posteriori
P(θ|x)
P(θ,x)
P(θ)
15
State θ
10
5
0
0
10
20
30
Observation x
40
50
Example of activity measurement
„
What if we change the a priori?
a piori
20
a posteriori
P(θ|x)
P(θ,x)
P(θ)
15
Hint of a
bad a priori!
State θ
10
5
0
0
10
20
30
Observation x
40
50
Example of activity measurement
„
What if we change the a priori?
a piori
20
a posteriori
P(θ|x)
P(θ,x)
P(θ)
15
Hint of a
bad a priori!
State θ
10
5
0
0
10
20
30
Observation x
40
50
Example of activity measurement
noise signal
ε = 0.1
t0 = 3600 s
t1 = 180 s
Probability density [1]
0.20
Conventional
Bayes
0.15
0.10
0.05
0.00
55
60
65
70
Activity [Bq]
Laedermann et al, Metrologia 42 (2005)
75
80
85
Example of activity measurement
noise signal
ε = 0.1
t0 = 3600 s
t1 = 180 s
Probability density [1]
3.0
Conventional
Bayes
2.0
1.0
0.0
0.0
0.5
1.0
Activity [Bq]
Laedermann et al, Metrologia 42 (2005)
1.5
Example of activity measurement
noise signal
ε = 0.1
t0 = 3600 s
t1 = 180 s
Conventional
Bayes
Probability density [1]
8
6
4
2
0
-0.4
-0.2
0.0
0.2
Activity [Bq]
Laedermann et al, Metrologia 42 (2005)
0.4
0.6
Type A uncertainties
„
"Type A uncertainties are obtained from a
probability density function […] derived from
an observed frequency distribution" [GUM:1995:3.3.5]
Type A uncertainties
„
"Type A uncertainties are obtained from a
probability density function […] derived from
an observed frequency distribution" [GUM:1995:3.3.5]
„
„
„
Clearly a frequentist approach
Can become tricky with few
measurement samples
Can be incorporated into the Bayesian framework
„
As a posteriori distribution coming from a statistical
analysis of a series of measurements
Type A uncertainties
Gauss(μ,σ)
we pick up N values randomly
from this underlying
distribution
1
x = ∑ xi
E ⎡⎣ x ⎤⎦ = μ
N i
2
1
−
s2 ( x ) =
x
x
E ⎡⎣s ⎤⎦ = σ
(
)
∑
i
N −1 i
⎡s (x)⎤
E⎢
⎥ = σ(x)
⎢⎣ N ⎥⎦
Usually reported std-uncertainty
values obtained by repetition
Type A uncertainties
Rel. std uncertainty [%]
„
s ( x ) / N can be a poor estimate of σ ( x )
"Uncertainty of the uncertainty"
80
60
40
[GUM:1995:Table E1]
20
0
2
3
4
5
6
7 8 9
10
Sample size nN
2
3
4
5
Type A uncertainties
Rel. std uncertainty [%]
If we estimate u(x) from 10 samples,
its uncertainty is about 25%
80
60
40
GUM Table E.1
20
0
2
3
4
5
6
7 8 9
10
Sample size nN
2
3
4
5
Type A uncertainties
Kacker & Jones Metrologia 40:235 (2003)
„
Propose considering Type A uncertainties as
Bayesian a posteriori
„
N −1 s (x)
uBayes ( x ) =
N−3 {
N
123
Bayesian correction
correction
Bayesian
1.8
1.8
1.6
1.6
=s( x )
1.4
1.4
1.2
1.2
1.0
1.0
33
44
55
66
77 88 99
10
10
22
Sample
Sample size
size nnN
33
44
55
66
77 88 99
100
100
Type B uncertainties
„
"Type B uncertainty is obtained from an
assumed probability density function based
on the degree of belief that an event will
occur" [GUM:1995:3.3.5]
Type B uncertainties
„
"Type B uncertainty is obtained from an
assumed probability density function based
on the degree of belief that an event will
occur" [GUM:1995:3.3.5]
„
Examples
„
„
Defined as a complement to Type A
„
„
calibration certificate, manufacturer's specifications,
values published in a data book, expert opinion
Not a statistical analysis of a series of measurements
Intrinsically Bayesian
„
We know a priori what is our degree of belief
(or knowledge)
Type B uncertainties
„
How good are we at estimating our
uncertainty?
„
„
Experiment
Give me a 98% confidence interval for
„
The Black
Swan (2007)
NN Taleb
The temperature of this room
„
„
Lower bound:
Upper bound:
?
?
Type B uncertainties
„
How good are we at estimating our
uncertainty?
„
„
Experiment
Give me a 98% confidence interval for
„
The Black
Swan (2007)
NN Taleb
The temperature of this room
„
„
Lower bound:
Upper bound:
18°C
24°C
22°C
"true" value
If we repeat this test with a large number
of people, 98% of the proposed confidence
intervals should include the value 22°C
Type B uncertainties
„
Give me a 98% confidence interval for
„
The area of Iraq in km2 or sq mi
„
„
„
Lower bound:
Upper bound:
?
?
The number of books in Umberto Eco's
personal home library
„
„
Lower bound:
Upper bound:
?
?
Type B uncertainties
„
Give me a 98% confidence interval for
„
„
The area of Iraq in km2 or sq mi
„ Lower bound:
?
437,072 km2
„ Upper bound:
?
(168,743 sq mi)
The number of books in Umberto Eco's
personal home library
„
„
Lower bound:
Upper bound:
?
?
32,000
Type B uncertainties
We should expect 2%
of incorrect intervals
(we are 34, 1% ~ 1 persons)
„
Study realized on
students from Harvard
Business School
„
45-50% OUT
Not the most modest
people…
„
Study realized on
New-York
taxi drivers
„
20-25% OUT
Maybe more modest
people…
Conclusions
What is it?
Bayesians
Frequentists
Subjective (state of belief)
Objective (part of the world)
Conclusions
Bayesians
Frequentists
What is it?
Subjective (state of belief)
Objective (part of the world)
Prob distribution
Directly available
Often not available
Conclusions
Bayesians
Frequentists
What is it?
Subjective (state of belief)
Objective (part of the world)
Prob distribution
Directly available
Often not available
Type A uncertainty
A posteriori
Frequentist by essence
Conclusions
Bayesians
Frequentists
What is it?
Subjective (state of belief)
Objective (part of the world)
Prob distribution
Directly available
Often not available
Type A uncertainty
A posteriori
Frequentist by essence
Type B uncertainty
Bayesian by essence
(a priori)
Out of the blue
Conclusions
Bayesians
Frequentists
What is it?
Subjective (state of belief)
Objective (part of the world)
Prob distribution
Directly available
Often not available
Type A uncertainty
A posteriori
Frequentist by essence
Type B uncertainty
Bayesian by essence
(a priori)
Out of the blue
Calculation
Usually hard
(new codes available: e.g.
MCMC)
Usually simple
Conclusions
Bayesians
Frequentists
What is it?
Subjective (state of belief)
Objective (part of the world)
Prob distribution
Directly available
Often not available
Type A uncertainty
A posteriori
Frequentist by essence
Type B uncertainty
Bayesian by essence
(a priori)
Out of the blue
Calculation
Usually hard
(new codes available: e.g.
MCMC)
Usually simple
A priori information
Essential
(can be "non informative")
Not needed
Conclusions
Bayesians
Frequentists
What is it?
Subjective (state of belief)
Objective (part of the world)
Prob distribution
Directly available
Often not available
Type A uncertainty
A posteriori
Frequentist by essence
Type B uncertainty
Bayesian by essence
(a priori)
Out of the blue
Calculation
Usually hard
(new codes available: e.g.
MCMC)
Usually simple
A priori information
Essential
(can be "non informative")
Not needed
Few measurements
Already possible
Hasardous
Conclusion
„
Bayesian and frequentist approaches are
fundamentally different but…
„
Results are the same in most practical cases
„
Exceptions
„
„
Both approaches are useful for metrology
„
Type B uncertainties are Bayesian by essence
„
„
Bayesian a priori are often based on frequentist approaches
Type A uncertainties are frequentist by essence
„
„
Extreme values, small samples
They can be put into a Bayesian framework
Same definition of probability for Type A & Type B
uncertainties would lead to more coherence of the
GUM