Discrimination metrics for sensory
panel data
Chris Crocker
Sample F-ratio
Most common measure in sensometrics
For panel means
F=
MS (Product)
MS (Assessor × Product)
Expected mean squares (balanced design)
E (MS Product ) = Q + n R Var( Assessor x Product) + Var (rep)
E ( MSAssessor × Product ) = n RVar (Assessor x Product) + Var (rep )
Interpretation not very intuitive
Metric on non-linear scale
2
making more of research
Intraclass Correlation Coefficient
Used in biometrics, psychometrics, etc.
Assume independence of product and error variances:
Var (measurement ) = Var ( product ) + Var (error )
Intraclass Correlation Coefficient
Var ( product )
r=
Var (measurement )
Proportion of observed variance attributable to sample variation
- proportion of “signal” in data
More interpretable than F-ratio but highly non-linear
Extendible to PCA and PLS
3
making more of research
Discrimination Ratio
Transform ICC to a linear metric
Wheeler, D.J. and Lyday, R.W. (1989). Evaluating the Measurement Process, 2nd edition, SPC Press Inc.
DR =
1+ r
1− r
number of distinct categories of product that can be established
DR ~ 2
threshold of usefulness
DR ~ 3
moderate discrimination
DR > 4
high discrimination
Suggestion: ignore attributes with DR < 1.75 (ICC ~ 0.50)
Sample F-ratio typically just significant for dataset of 8-12 products and 10 assessors
4
making more of research
Example – salad dressing data
Discrimination of assessors
Var(error) estimated from replicates
Prefer value based on a single rep
Discrimination of panel means
Var(error) estimated from Assessor x Sample interaction
Discrimination of PCA dimensions
Propagate errors into principal components
Requires error covariances
Can also do for PLS factors
5
making more of research
Assessor discrimination ratios
All panellists have p<0.01
Discrimination of panellists on Sour Flavour
3.0
2.5
2.0
1.5
1.0
0.5
0.0
As
se
ss
or
2
As
se
ss
or
3
As
se
ss
or
4
As
se
ss
or
5
As
se
ss
or
6
As
se
ss
or
7
As
se
ss
or
8
As
se
ss
or
9
As
se
ss
or
10
As
se
ss
or
11
As
se
ss
or
12
Discrimination Ratio for single rep
3.5
6
making more of research
Discrimination of panel means
Intraclass
Correlation
Coefficient
Discrimination Ratio
Acidic Odour
0.95
6.4
Sour Odour
0.93
5.1
Fresh Odour
0.97
8.1
Rancid Odour
0.97
8.1
Off-Odour
0.93
5.3
Whiteness
0.90
4.4
Colour, Hue
0.87
3.8
Colour Intensity
0.93
5.1
Mustard Flavour
0.87
3.8
Acidic Flavour
0.96
7.0
Sour Flavour
0.84
3.3
Sweetness
0.62
2.1
Fresh Flavour
0.97
7.9
Rancid Flavour
0.97
8.4
Off-Flavour
0.94
5.7
Attribute
7
making more of research
Assessor x Sample interaction plot for Whiteness (DR=4.4)
9
8
7
Assessor 3
Assessor 4
6
Mean of reps
Assessor 5
Assessor 6
5
Assessor 7
Assessor 8
4
Assessor 9
Assessor 10
3
Assessor 11
Assessor 12
2
1
0
1
4
7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
Sam ples
8
making more of research
Assessor x Sample interaction plot for Sour Flavour (DR=3.3)
10
9
8
Assessor 3
7
Assessor 4
Mean of reps
Assessor 5
6
Assessor 6
Assessor 7
5
Assessor 8
Assessor 9
4
Assessor 10
3
Assessor 11
Assessor 12
2
1
0
1
4
7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
Sam ples
9
making more of research
Assessor x Sample interaction plot for Sweetness (DR=2.05)
8
7
6
Assessor 3
Assessor 4
Assessor 5
Mean of reps
5
Assessor 6
Assessor 7
4
Assessor 8
Assessor 9
3
Assessor 10
Assessor 11
2
Assessor 12
1
0
1
4
7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
Sam ples
10
making more of research
PCA of salad dressing data - correlation matrix
Dimension
Variance
ICC
Discrimination
Ratio
1
66.9%
0.97
8.6
2
15.5%
0.91
4.5
3
11.2%
0.89
4.2
4
2.7%
0.49
1.7
5
1.5%
0.34
1.4
6
0.7%
0.04
1.0
7
0.6%
0.31
1.4
8
0.3%
0.00
1.0
9
0.2%
0.00
1.0
10
0.1%
0.00
1.0
11
0.1%
0.00
1.0
12
0.1%
0.00
1.0
13
0.0%
0.00
1.0
14
0.0%
0.00
1.0
11
making more of research
PCA map of first 2 dimensions
Sour Flavour
Dimension 2 (15.5%, DR=4.5)
1521
5211 3211
5312
1211 1222
1212
1221
1421
1121
1122
1321
5212
3311
4211
2311
Off-Odour
Off-Flavour
Rancid Odour
Rancid Flavour
4212
6312
3312
1311
2212
2211
3321
4312
5321
5421
4321
6321
6421
6221
1322
1312
1621
2521
6212
2312
2321
3212
5121
4221
5221
3421
6121Colour, Hue
3221
4121
2222
2221 2121
2122
2421
2322
2621
5521
3122
3121
4521
4122
6521
5621
3521 4322
5222
4222
5322 6222 3222
5122
3621
6322
6122
4621
6621
4421
Sour Odour
Whiteness
Fresh Odour
Fresh Flavour
Acidic Flavour
Acidic Odour
Mustard Flavour
Sweetness
3322
Colour Intensity
Colour coded by Date
Dimension 1 (66.9%, DR=8.6)
12
making more of research
Summary
Discrimination Ratio provides a linear, intuitive metric
communicable to non-statisticians
Interpretation robust to distributional assumptions
Applicable to functions of the attributes e.g. principal components
Knowledge of Intraclass Correlation Coefficient useful in modelling
Details in poster
Chris Crocker
[email protected]
13
making more of research