Outcome from result forms (first round):
measurement uncertainty
Marina Patriarca
Istituto Superiore di Sanità
Dpt of Public Veterinary Health and Food Safety
CRL-ISS Workshop 2008
on Proficiency Tests
Rome (Italy), 17 October 2008
6th WORKSHOP
PROFICIENCY TESTING IN ANALYTICAL CHEMISTRY,
MICROBIOLOGY AND LABORATORY MEDICINE
Current practice and Future Directions
Rome (Italy) 6-7 October 2008
Organised in co-operation with
Chair of the Organising Committee: Dpt PVH FS, ISS
www.iss.it/eurachem
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Issues
• estimating uncertainty: which approach?
– Affordable?
– Reliable?
– Fit for purpose?
• assessing compliance taking uncertainty
into account: which rules?
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Harmonisation?
y
.
g
7
o
0
l
0
o
2
r
:
t
9
e
9
M
e
f
d
d
o
i
n
u
a
y
r
G
s
a
t
l
O
p
u
e
S
b
I
c
a
n
c
o
o
C
V
l
l
a
a
r
s
n
e
o
m
i
n
r
t
e
e
a
G
T
n
r
d
d
Inte sic an ociate
Ba
Ass (VIM3)
freely available from www.bipm.fr
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Measurement Uncertainty, u(x)
Parameter, associated with the result of a
measurement, that characterizes the
dispersion of the values that could reasonably
be attributed to the measurand
ISO, VIM 2: 1993 (3.9)
Non-negative parameter
characterising the dispersion of the quantity
values being attributed to a measurand,
based on the information used
(ISO, VIM 3: 2007, 2.26)
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Measurement result
2.9 (VIM3)
set of quantity values being attributed
to a measurand together with any other
available relevant information
3.1 (VIM2) value attributed to a
measurand obtained by measurement
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Measurement result
NOTES
…
2 — A measurement result is generally
expressed as a single measured quantity
value and a measurement uncertainty.
…
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
12th PT - Lead in Milk - Sample A
0,05
Pb in milk, mg/kg
0,04
0,03
0,02
0,01
0
ab
cd
gh
ed
gg
ff
yy
mm
dd
kk
bb
ww
aa
rr
jj
af
ag
aq
Laboratories
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
pp
Relevant documents
• ISO. 1993. Guide to the expression of uncertainty
in measurement. (GUM)
• EURACHEM/CITAC Guide. Quantifying
Uncertainty in Analytical Measurement (QUAM).
2000. www.eurachem.org
• Eurolab Technical Report No 1/2007.
Measurement uncertainty revisited: Alternative
approaches to uncertainty evaluation.
www.eurolab.org
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
A summary of approaches to MU
Definition of the measurand
List of uncertainty components
Intralaboratory
Yes
Mathematical
model?
Evaluation of
standard
uncertainties
Interlaboratory
No
Organisation of
replicate
measurements
Method
validation
CRL-ISS Workshop on PT 2008
Yes
PT or method
performance
study?
Method accuracy
ISO 5725
ISO TS 21748
No
Proficiency
testing
ISO Guide 43
ISO 13528
Rome (Italy), 17 ottobre 2008
Intralaboratory approaches:
Modelling
The model of the measurement process (independent variables)
A × Cs V f
C=
×
As
Vi
e.g.
or: y = f(x1, x2, …xi)
The uncertainties on each variable (influence quantity)
e.g. u(V), u(A), u(xi) can be combined
uc ( y ) =
CRL-ISS Workshop on PT 2008
N
∑
i =1
 ∂f ( xi )  u 2 ( xi )


 ∂x1 
2
Rome (Italy), 17 ottobre 2008
Modelling approach for analytical
chemistry: drawbacks
• appropriate models?
• measurement standards/methods?
• can all effects be individually identified
and quantified (e.g. matrix variability)?
• how reliable is the MU estimate?
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Intralaboratory approaches:
Method validation studies/IQC (1)
Information from method validation studies:
the best available estimate of overall precision, u(P)
the best available estimate of overall bias (Rm) and
its uncertainty u(Rm): calculated from the analysis of
CRMs or the recovery of spiked amounts of the pure analyte;
the best estimate of other components of uncertainty
associated with effects incompletely accounted for in
the development and validation study, e.g. residual
matrix effects assessed from the analysis of matrix-matched
EQAS samples, which target values are not accompanied by
uncertainty statements, u(xi).
All terms expressed as relative standard deviations
(i.e relative standard uncertainties) and combined
according to the uncertainty propagation law
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Intralaboratory approaches:
Method validation studies/IQC (2)
Preliminary requirements
1. Quality assurance measures concerning:
staff
measurement procedures
measuring equipment
standards and matrix-matched reference
materials
internal quality control / control charts
2. Any significant bias should be eliminated,
corrected or otherwise taken into account
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Uncertainty components from in house
validation studies
integrazione
del segnale
taratura
A
Bias
interferenze
u(Rm)
della matrice
efficienza della
nebulizzazione
efficienza della
nebulizzazione
efficienza della
atomizzazione
Precision
u(P)
ripetibilità
T
T
Vf
T
operatori
taratura
Vi
operatori
riproducibilità
taratura
T
riproducibilità
2
A × Cs V f
C=
×
As
Vi
ripetibilità
T
T
2
riproducibilità
ripetibilità
u(C )
 u( xn ) 
 u ( P)   u( Rm) 
= 
 +
 + ... + 

C ripetibilità C   Rm 
C 
taratura

taratura
diluizione
operatori
2
C
T
riproducibilità
operatori
operatori
T
riproducibilità
incertezza sul titolo MR
Other components: u(xi) ripetibilità
e.g. sampling, matrix
inhomogeneity, robustness
integrazione
Cs
As
del segnale efficienza della
atomizzazione
efficienza della
nebulizzazione
Interlaboratory approaches
Method performance studies
ISO/TS 21748 provides guidance for the use of
information from collaborative interlaboratory studies
on the performance of a defined and detailed
measurement procedure, designed and carried out
according to ISO 5725:1994:
U = 2sR
ISO/TS 21748 Guidance for the use of repeatability, reproducibility
and trueness estimates in measurement uncertainty estimation.
ISO, Geneva, 2004.
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Interlaboratory approaches
Proficiency Testing (1)
Use results of PT
– as a check on the evaluated uncertainty
– to obtain/assess information on bias and
its uncertainty
– as a basis for a good estimate of the
uncertainty arising from those parts of the
measurement procedure within the scope
of the scheme
[under stated conditions & provided that
systematic deviation and any other sources
of uncertainty are also taken into account]
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Interlaboratory approaches
Proficiency Testing (2)
• An approach based solely on PT data is not
recommended, however:
– PT/EQAS requiring laboratories to adopt the
same or very similar methods provide a rough
indication of MU as:
u = SD PT / EQAS
_ one _ method
– preliminary evaluation of MU as the SD of the
differences between the laboratory reported
values and the assigned values in repeated
rounds
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Which coverage factor?
•
•
•
•
•
•
11 labs: 2
3 labs: did not report U
2 labs did not report k
1 lab: 0.8346
1 lab: 2.306
1 lab: 3.18
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Expanded uncertainty
U=ku
Which coverage factor?
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Choosing K (1)
Issues to be considered
– The level of confidence required
– Knowledge of the underlying distributions
of values
– Knowledge of the number of values used
to estimate random effects
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Choosing K (2)
• For most purposes it is recommended
that k is set to 2.
• This value of k may be insufficient
where the combined uncertainty is
based on statistical observations with
relatively few degrees of freedom (less
than about six). The choice of k then
depends on the effective number of
degrees of freedom.
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Choosing K (3)
k = 2, why???
– The level of confidence required
• 95%
– Any knowledge of the underlying
distributions of values.
• normal
– Any knowledge of the number of values
used to estimate random effects
• always >6, unless based on strict
modelling approach
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
D.1.3. Measurement uncertainty
The analytical result shall be reported as
x ± U whereby x is the analytical result
and U is the expanded measurement
uncertainty, using a coverage factor of 2
which gives a level of confidence of
approximately 95 % (U = 2u).
COMMISSION REGULATION (EC) No 333/2007
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Is my MU estimate reliable?
0,05
Sample A
solid line: Avg A, 0.025 mg/kg
dotted lines: Uf A, 0.0054 mg/kg
Pb in milk, mg/kg
0,04
0,03
0,02
0,01
0
ab
cd
gh
ed
gg
ff
yy
mm
dd
kk
bb
ww
aa
rr
jj
af
ag
aq
Laboratories
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
pp
C.3.3.2. ‘Fitness for purpose’ approach
Maximum standard measurement
uncertainty (Uf)
Uf = (LOD/ 2) + (αC)
2
2
LOD = limit of detection of the method (µg/kg);
C = the concentration of interest (µg/kg);
α = numeric factor - for C<50 µg/kg, use 0.2
COMMISSION REGULATION (EC) No 333/2007
Lead in milk
CRL-ISS Workshop on PT 2008
Uf = (4 / 2) + (0.2 × C)
2
2
Rome (Italy), 17 ottobre 2008
Is my estimate of MU fit for purpose?
0,02
Lead in milk - Sample A
Lead in milk - Sample B
U, mg/kg
0,015
Uf A, 0.0054 mg/kg
Uf B, 0.0066 mg/kg
based on avg conc
0,01
0,005
0
ab
cd
gh
ed
gg
ff
yy
mm
dd
kk
bb
ww
aa
rr
jj
af
ag
aq
Laboratories
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
pp
Interpretation of results
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Maximum levels
COMMISSION REGULATION (EC) No 1881/2006 of 19
December 2006 setting maximum levels for certain
contaminants in foodstuffs
Section 3: Metals
3.1 Lead
3.1.1 Raw milk ( 6 ), heat-treated milk and milk for the
manufacture of milk-based products
0,020 (mg/kg wet weight)
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
D.2.1. Acceptance of a lot/sublot
The lot or sublot is accepted if the analytical
result of the laboratory sample does not exceed
the respective maximum level as laid down in
Regulation (EC) No 1881/2006 taking into
account the expanded measurement
uncertainty and correction of the result for
recovery if an extraction step has been applied
in the analytical method used.
COMMISSION REGULATION (EC) No 333/2007
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
D.2.2. Rejection of a lot/sublot
The lot or sublot is rejected if the analytical
result of the laboratory sample exceeds beyond
reasonable doubt the respective maximum level
as laid down in Regulation (EC) No 1881/2006
taking into account the expanded measurement
uncertainty and correction of the result for
recovery if an extraction step has been applied
in the analytical method used.
COMMISSION REGULATION (EC) No 333/2007
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Harmonised decisions?
0,05
N
Y
N
Pb in milk, mg/kg
0,04
N
N
Y
N
N
Y
N
N
Y
0,03
Y
Y
0,02
Y
0,01
Y
0
ab
cd
gh
ed
gg
ff
yy
mm
dd
kk
bb
ww
aa
rr
jj
af
ag
Laboratories
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
aq
pp
Guidance documents
• ILAC-G8:1996 Guidelines on assessment and reporting of
compliance with specifications
• Eurachem/CITAC Guide. Use of uncertainty information in
compliance assessment. (2007) www.eurachem.org
• DG SANCO Report on the relationship between analytical
results, measurement uncertainty, recovery factors and
the provisions of EU Food and Feed legislation,
with particular reference to Community Legislation concerning
- contaminants in food (Council regulation (EEC) no 315/93 of 8 February 1993 laying
down Community procedures for contaminants in food)
- undesirable substances in feed (Directive 2002/32/EC of the European Parliament and
of the Council of 7 may 2002 on undesirable substances in animal feed)
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
ILAC G8 - Criteria for inerpretation of results
reject
120
Accept/reject only if requirements
expressed as < ; >
accept
100
80
1
2
3
4
5
accept/reject with a less than 95% level of confidence
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
EURACHEM/CITAC Guide - Decision rules
• Prescriptions for the acceptance or rejection of a
product based on the measurement result, its
uncertainty and the specification limit or limits,
taking into account the acceptable level of the
probability of making a wrong decision.
• “Acceptance” and “Rejection” Zones
• A decision rule that is currently widely used is that
a result implies non compliance with an upper
limit if the measured value exceeds the limit
by the expanded uncertainty.
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008
Recommendation
Report on the Relationship Between Analytical Results, Measurement Uncertainty, Recovery
Factors and the Provisions of EU Food and Feed Legislation,
In practice, when considering a maximum value in
legislation, the analyst will determine the analytical
level and estimate the measurement uncertainty at
that level. The value obtained by subtracting the
uncertainty from the reported concentration, is
used to assess compliance.
Only if that value is greater than the maximum
level in the legislation is it certain “beyond
reasonable doubt” that the sample concentration
of the analyte is greater than that required by the
legislation.
CRL-ISS Workshop on PT 2008
Rome (Italy), 17 ottobre 2008