KEY TO THE QCMD EXTERNAL QUALITY ASSESSMENT
INDIVIDUAL REPORT
The QCMD EQA Individual Reports are divided into sections to assist participants’
navigation of the information presented. One individual report is provided for each dataset
submitted by participants.
The Sections are:
•
•
•
Expected results
Analysis of your laboratory’s Qualitative EQA data
Analysis of your laboratory’s Quantitative EQA data, where applicable
Expected results
This section provides participants with the panel composition and expected results. An
example is shown in Table 1.
Table 1: Example - Expected results of the programme in order of sample content and
concentration
Sample
Sample Content
Sample conc.
Sample Status
Sample Type
CMV09-07
CMV (Strain AD 169)
23,174
Frequently detected
Core
CMV09-12
CMV (Strain AD 169)
4,613
Frequently detected
Core
CMV09-06
CMV (Strain AD 169)
3,266
Frequently detected
Core
CMV09-05
CMV (Strain AD 169)
1,028
Frequently detected
Core
CMV09-02
CMV (Strain AD 169)
1,009
Frequently detected
Core
CMV09-11
CMV (Strain AD 169)
245
Detected
CMV09-10
CMV (Strain AD 169)
238
Detected
CMV09-03
CMV (Strain AD 169)
211
Detected
CMV09-09
CMV (Strain AD 169)
2,228,435
Frequently detected
Core
CMV09-04
CMV (Strain AD 169)
252,348
Frequently detected
Core
CMV09-01
CMV (Strain AD 169)
24,099
Frequently detected
Core
CMV09-08
Negative Plasma
Negative
Core
Table key:
Sample: QCMD panel sample codes for the samples distributed to participants
Sample Content: viral or microbial content of the panel samples
Sample conc.: Values should not be used by participants for method comparison or as a target for individual
laboratory performance assessment
Sample Status: the sample status assigned to each panel sample (see below)
Sample Type: Panel samples classified as core proficiency samples in the EQA programme. An acceptable level
of proficiency is demonstrated by obtaining a correct result on all samples classified as ‘core’ within this EQA
programme.
Participants are provided with the assigned status for each panel sample. The statuses for
panel samples containing the target are ‘Frequently detected’, ‘Detected’ and ‘Infrequently
detected’. Panel samples negative for the target are assigned a 'Negative' status. The sample
status is defined based on performance in the EQA programme and the expertise of the
QCMD Scientific Council.
IRK001-04
1/8
QCMD Individual Report Key Version 04
1. Analysis of your laboratory’s Qualitative EQA data
Analysis of your laboratory's performance on the core proficiency samples
The QCMD EQA panels contain a range of samples, designed to look at different aspects of
assay performance. Panel members are designated core proficiency samples on the basis of
scientific information, clinical relevance and clinical experience (published literature and
professional clinical guidelines) and, where available and appropriate, established target
performance limits taken from previous QCMD EQA distributions. Laboratories are expected
to correctly analyse and report the core proficiency samples in order to show acceptable
proficiency. An example is provided below.
Analysis of your laboratory's performance on the core proficiency samples:
The core proficiency samples in this EQA programme were: CMV09-07, CMV09-12, CMV09-06,
CMV09-05, CMV09-02, CMV09-09, CMV09-04, CMV09-01, CMV09-08.
You reported 8/9 (88.9%) of the core proficiency samples correctly.
Of the total datasets reported by all participants in this EQA programme, 95% reported correct results
for all core proficiency samples.
IRK001-04
2/8
QCMD Individual Report Key Version 04
Analysis of your laboratory’s performance on all proficiency samples
This section provides participants with their qualitative results and scores for each individual
panel sample and an overall performance indicator called the Sum Qualitative Panel Score.
An example of the data is given in Table 2 and Figure 1.
Table 2: Example – Your laboratory’s qualitative results and performance scores
Qualitative
Sample
Sample Content
CMV09-07
CMV (Strain AD 169)
CMV09-12
CMV (Strain AD 169)
CMV09-06
CMV09-05
CMV09-02
CMV09-11
CMV09-10
CMV (Strain AD 169)
CMV (Strain AD 169)
CMV (Strain AD 169)
CMV (Strain AD 169)
CMV (Strain AD 169)
Sample
Type
Your
qualitative
result
Your
qualitative
score
Frequently detected
Core
Positive
0
Frequently detected
Core
Negative
3
Core
Positive
0
Core
Positive
0
Core
Positive
0
Detected
Positive
0
Detected
Positive
0
Negative
2
Core
Positive
0
Core
Positive
0
Core
Positive
0
Core
Negative
0
Sample
Status
Frequently detected
Frequently detected
Frequently detected
CMV09-03
CMV (Strain AD 169)
Detected
CMV09-09
CMV (Strain AD 169)
Frequently detected
CMV09-04
CMV09-01
CMV (Strain AD 169)
CMV (Strain AD 169)
CMV09-08
Negative Plasma
Sum Qualitative Panel Score
Frequently detected
Frequently detected
Negative
5
Table key:
Sample: QCMD panel sample codes for the samples distributed to participants
Sample Content: viral or microbial content of the panel sample
Sample Status: the sample status assigned to each panel sample
Sample Type: Panel samples classified as core proficiency samples in the EQA programme. An acceptable level
of proficiency is demonstrated by obtaining a correct result on all samples classified as ‘core’ within this EQA
programme.
Your qualitative result: The qualitative results returned by your laboratory for each panel sample
Your qualitative score: The qualitative score awarded to your laboratory for each panel sample
Sum Qualitative Panel Score: The Sum Qualitative Panel Score awarded to your laboratory
IRK001-04
3/8
QCMD Individual Report Key Version 04
Figure 1: Example - Sum Qualitative Panel scores for all participants
Graph key:
Sum Qualitative Panel Score: the range of Sum Qualitative Panel Scores
Number of Datasets: the number of datasets achieving each Sum Qualitative Panel Score
Cumulative %: The percentage of datasets achieving each Sum Qualitative Panel Score or better
Calculation of your laboratory’s qualitative score for each individual panel sample
The following scoring system applies to your laboratory’s qualitative score for each
individual panel sample. The scores awarded for qualitative data are based on the sample
status where 0 is 'highly satisfactory' and 3 is 'highly unsatisfactory'. Colour has been included
as an extra visual aid.
Scoring system based on the assigned sample status
Sam ple status
Participant's result
Negative
Not determ ined
Frequently detected
3
3
Positive
0
Detected
2
2
0
Infrequently detected
1
1
0
Negative
0
3
3
For example when a participant returns a negative qualitative result for an ‘Infrequently
detected’ panel sample they score 1. However a participant would score 3 if they returned a
Negative qualitative result for a ‘Frequently detected’ panel sample.
IRK001-04
4/8
QCMD Individual Report Key Version 04
Calculation of your laboratory’s Sum Qualitative Panel Score
The Sum Qualitative Panel Score is the sum of your qualitative performance scores for all
panel samples. The perfect Sum Qualitative Panel Score is zero points. The Sum Qualitative
Panel Scores for all participants are presented in graphical form (see Figure 1 for an
example). This graph provides participants with the number of datasets achieving each Sum
Qualitative Panel Score and the percentage of datasets achieving each Sum Qualitative Panel
Score or better.
2. Analysis of your laboratory’s Quantitative EQA data
Your laboratory’s quantitative performance on the paired samples
Assays may differ relatively in the quantitative values they report. In EQA programmes
where sufficient quantitative data are reported, QCMD assesses quantitative proficiency based
on the difference reported between paired samples, which provide participants with a measure
of proficiency that is independent of assay bias. The median difference between the paired
samples is calculated from the log concentration of participants’ results.
The choice of paired samples is made on the basis of scientific information, clinical relevance
and clinical experience (published literature and professional clinical guidelines) and, where
available and appropriate, established target performance limits taken from previous QCMD
EQA distributions. An example is provided below:
Table 3: Example - Your laboratory’s quantitative performance on the paired samples
Paired samples
Sample Content
CMV09-07 and -12
CMV AD 169
Median
difference (log)
0.701
Your
difference (log)
0.687
Within 0.5 log units
of the median
Yes
Table key:
Paired samples: the samples used for the paired analysis
Sample content: viral or microbial content of the paired samples
Median difference: The median difference between the two samples in log10 units
Your difference: The difference between the sample pair (in log10 units) reported by your laboratory
IRK001-04
5/8
QCMD Individual Report Key Version 04
Your laboratory’s quantitative results and performance scores
This section provides participants with their quantitative results and scores for each panel
sample and an overall performance indicator called the Sum Quantitative Panel Score. An
example of the data is given in Table 4.
Table 4: Example - Your laboratory’s quantitative results and performance scores
CMV09-07
CMV (Strain AD 169)
4.337
Quantitative
Your
Consensus
score
0
CMV09-12
CMV (Strain AD 169)
3.650
0
0
CMV09-06
CMV (Strain AD 169)
3.520
0
0
0
Sample
Sample Content
Your
Result (log)
Your
Technology
score
0
CMV09-05
CMV (Strain AD 169)
2.995
0
CMV09-02
CMV (Strain AD 169)
3.000
0
0
0
CMV09-11
CMV (Strain AD 169)
2.348
0
CMV09-10
CMV (Strain AD 169)
2.381
0
0
CMV09-03
CMV (Strain AD 169)
2.350
0
0
0
CMV09-09
CMV (Strain AD 169)
5.847
1
CMV09-04
CMV (Strain AD 169)
5.404
0
0
4.385
0
0
1
0
CMV09-01
CMV (Strain AD 169)
Sum Quantitative Panel Score
Table key:
Sample: QCMD panel sample codes for the samples distributed to participants
Sample content: viral or microbial content of the panel samples
Your Result (log): The quantitative results (in log10 units) returned by your laboratory for each panel sample
Your Consensus score: The quantitative score awarded to your laboratory for each panel sample
Your Technology score: The quantitative score awarded to your laboratory for each panel sample based on the
technology concentration
Sum Quantitative Panel Score: The Sum Quantitative Panel Score awarded to your laboratory
IRK001-04
6/8
QCMD Individual Report Key Version 04
Figure 2: Example - Sum Quantitative Consensus Panel scores for all participants
Graph key:
Sum Quantitative Consensus Panel Score: the range of Sum Quantitative Consensus Panel Scores
Number of datasets: the number of datasets achieving each Sum Quantitative Consensus Panel Score
Cumulative %: The percentage of datasets achieving each Sum Quantitative Consensus Panel Score or better
Sum Quantitative Panel Scores are only provided to participants that return quantitative data
for all panel samples that contain the target agent. Those that return LOD/NR (limit of
detection or not reported) for one or more panel samples are represented by the ‘-’ in Figure
2.
Calculation of your laboratory’s quantitative score for each individual panel sample
The following scoring system applies to your laboratory’s quantitative score for each
individual panel sample. Participants are provided with a summary of their performance
compared to consensus values for each of the positive panel samples.
In order to compare participants’ results within specific technologies or kit methods, where
sufficient datasets are reported (5 or more) methods or kits are assigned to a technology group
e.g. Real time PCR, bDNA or NASBA etc. Where datasets are reported as 'other' for a
technology or kit method this is reviewed by the QCMD Neutral Office and assigned to an
appropriate group where possible. The negative panel samples are not included in these
analyses.
The parameters of the normal distribution are estimated from the log of the positive datasets
submitted by participants for each panel sample. Scores are assigned based on the distance
from the mean value for each panel sample. The scoring system used for quantitative EQA
data ranges from 0 (highly satisfactory) to 3 (highly unsatisfactory).
IRK001-04
7/8
QCMD Individual Report Key Version 04
An assigned value is calculated for each panel sample using two methods. These are:
1.
Consensus concentration - the mean of the participants' results once outliers have
been removed.
2.
Technology consensus concentration - the mean of the participants' results per
technology group once outliers have been removed.
The standard deviation is calculated as the square root of the mean square for error from the
ANOVA table where the response is the log concentration of participants’ results with
outliers removed. The factor is technology group.
Outliers are defined as values with a standardised residual with modulus greater than three.
Although outliers are removed for the calculation of the assigned values they are included in
the data analysis and scored accordingly.
Scores are awarded based on the distance from the calculated mean value for each panel
sample. Zero points are awarded if the quantitative value returned is within one standard
deviation from the mean. One point is awarded if the quantitative value is between one and
two standard deviations, two points if the value is within two and three standard deviations
and three points for quantitative values more than three standard deviations from the mean.
Where no value (or a limit of detection only) is reported this is assigned as LOD/NR (limit of
detection or not reported).
Scoring system based on distance from the consensus value
0
1
1
2
2
3
3
Each coloured division represents one standard deviation (SD) from the mean, so that zero points is awarded for quantitative
values that are within one standard deviation and three points for quantitative values that are more than three standard deviations
from the mean.
Calculation of your laboratory’s Sum Quantitative Panel Score
The Sum Quantitative Panel Score is the sum of your quantitative performance scores for all
panel samples. The perfect Sum Quantitative Panel Score is zero points. The Sum
Quantitative Panel Scores for all participants are presented in graphical form (see Figure 2 for
an example). This graph provides participants with the number of datasets achieving each
Sum Quantitative Panel Score and the percentage of datasets achieving each Sum
Quantitative Panel Score or better. A Sum Quantitative Panel Score is provided for the
performance of laboratories in relation to the Consensus Concentration only.
IRK001-04
8/8
QCMD Individual Report Key Version 04