EP Evaluator Sample Reports
composite report example
6/23/2015
Prepared for
Prepared by
EE11.2.23
My Community Hospital
Tucson, AZ
Inez Kruse
Medical Director approved
Accepted by
Signature
Name / Title
Date
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:59:23
Copyright 1991-2015 Data Innovations LLC
Page 1
1
Table of Contents
Module
SP (Simple Precision)
Lin (Linearity)
SA (Simple Accuracy)
AMC (Alternate MC)
MIC (Multiple Instrument Compare)
QMC (Qualitative MC)
ERI Full (ERI Full Analysis Report)
TRU (Trueness)
CO (Carryover)
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:59:23
Page
3
8
13
15
21
26
30
43
48
Copyright 1991-2015 Data Innovations LLC
Page 2
2
BUN
Instrument ANALYZER
Sample Name level 2
EE11.2.23 -- My Community Hospital
Simple Precision
Precision Statistics
Obs Standard Dev (SD)
Pass/Fail/Uncertain
95% Confidence for Obs SD
Obs Coef of Variation (CV)
Obs Mean
Number of Specimens (N)
95% CI for Obs Mean
Obs 2 SD Range
Precision Plot
Supporting Data
mkf
04 Jun 2012
mg/dL
19.9
10.3
3.0
Analyst
Expt Date
Units
Target Mean
Target CV
Target Range
2.0
Yes
1.6 to 2.8
10.4%
19.4 mg/dL
25 of 25
18.6 to 20.3
15.4 to 23.5
Histogram
User's Specifications
Precision Verification Goal
Within Run SD
Conc at SD Goal
Vendor SD
2
Accepted by:
Signature
EP Evaluator 11.2.0.23
website sample reports Printed: 27 Apr 2015 10:50:16
Date
Copyright 1991-2015 Data Innovations LLC
Page 1
3
BUN
Instrument ANALYZER
Sample Name level 2
EE11.2.23 -- My Community Hospital
Simple Precision
Precision Data
Index
Results
1
2
3
4
5
6
7
21
19
22
20
18
20
15
Index
8
9
10
11
12
13
14
Results
20
21
19
22
20
18
20
Index
15
16
17
18
19
20
21
Results
15
20
21
19
22
20
18
Index
Results
22
23
24
25
20
15
20
21
X: excluded from calculations
EP Evaluator 11.2.0.23
website sample reports Printed: 27 Apr 2015 10:50:16
Copyright 1991-2015 Data Innovations LLC
Page 2
4
EE11.2.23 -- My Community Hospital
Simple Precision
Report Interpretation Guide
CLIA and CAP require periodic verification of Precision. The
Simple Precision experiment is a quick way to fulfill the
requirement for repeatability using a “within-run” type of
experiment. The procedure is to enter repeated
measurements of a single sample on a single method and
compute simple statistics like mean SD and %CV. In
addition EP Evaluator Release 11.2 offers the following
options:
1: Entry of one of three Precision Verification Goal
Modes:
 TEA ased Allowable Random Error: entered as a
percentage of the specified TEA.
 Vendor SD based: Entered as one within run SD at a
specified concentration level.
 Vendor REa based: Entered as one within run SD or
within run Percent or both.
When using RRE data entry techniques the Precision
Verification Goal Mode is defined in the Modules and
Options form in Policies. When any type of goal is entered
the program will report that the test passes if the calculated
SD does not exceed the goal. An option in File Preferences
further enables the Pass/Fail criteria to include a designation
for uncertain if the precision goal is within the 95%
confidence limits around the Observed (O S) SD.
2: Entry of a Target Mean value . Investigators often wish
to compare the recovered mean with a target.
3: Show Histogram Result Distribution. If the Show
Histogram option is enabled then a Histogram displays a
frequency distribution of recovered results centered around
the observed mean. If Show Target Range is also enabled
then entry of a target mean is required and a user selectable
SD target limit of 2 SD 2.5 SD 3 SD or 3.5 SD will be
drawn around the target mean.
,
,
,
,
B
,
,
"
"
B
"
"
,
"
,
"
,
,
,
mean. In other words a measure of Precision. For many
analytes SD varies with sample concentration. Example: For
Glucose an SD of 10 for a 400 mg/dL sample represents
very good precision. For a 40 mg/dL sample an SD of 10
represents very poor precision. The Simple Precision
calculation for one SD is the standard calculation for sample
SD - same as used in a pocket calculator.
Coefficient of Variation (CV): SD expressed as a percent
of the mean. %CV 100 times SD/mean. For analytes
where error varies with concentration CV is more likely to
remain constant across the analytical range. Example: 2.5%
CV for Glucose is good precision at any concentration.
Target CV is calculated as 100 times the target SD /
Observed mean.
Within-Run and Total SD: Manufacturers often publish
these statistics in package inserts. Typically they are
computed from a more complex precision experiment that
requires replicate measurements over several days. Total
SD and within run SD from the complex precision experiment
use Analysis of Variance (ANOVA) calculation techniques
and are NOT calculated in the same way as EP Evaluator s
Simple Precision SD.
Number of Specimens (N): A good precision study should
include 20-50 replicates.
95% Confidence Intervals: These are calculated for both
the observed mean and the observed SD. In a sense this
measures the precision” of the mean or SD. It shows how
much the observed mean or SD might vary if the precision
experiment was repeated. The two-tailed 95% confidence
interval (CI) is calculated from standard Chi-square tables.
Its width depends on N and the observed mean or SD. The
reliability of the precision estimate of the mean or SD
improves as N increases.
Mathematically it is possible to calculate SD and CV from
only 3 replicates but it is not a very reliable estimate. For
example suppose the true SD is 1.00. An estimate based on
3 replicates might easily be as low as 0.52 or as high as 6.29
(95% confidence). At N 20 the 95% confidence interval (CI)
is much narrower: 0.76 to1.46. At N 50 it narrows further:
0.84 to 1.24.
,
,
,
,
=
,
,
'
,
"
,
,
,
Definitions
Precision: Ability to obtain the same result upon repeated
measurement of a specimen - not necessarily the correct
result ust the same result. Ability to get the correct result is
Accuracy. Within-run precision is also known as
Repeatability, which is the repeated measurement of a
specimen under the same set of conditions.
Mean: Average value computed by taking the sum of the
results and dividing by the number of results.
Bias: The Target mean minus the observed mean.
%Bias: 100 times the ias divided by the target mean.
Standard Deviation (SD): The primary measure of
dispersion or variation of the individual results about the
=
,
=
, j
,
B
EP Evaluator 11.2.0.23
Printed 27 Apr 2015 10:50:16
Precision Verification Goal Modes:
Allowable Total Error (TEa), and the Error Budget
TEa states the laboratory s policy for how much error is
medically (or administratively) acceptable. Regulatory
requirements represent an upper limit of that specification.
Example: the CLIA limit for Sodium is 4 mmol/L. The
experimental Total Error has two ma or components:
Systematic Error (synonym: ias) and Random Error
(synonym: Imprecision). The Error udget for Total allowable
'
j
B
B
Copyright 1991-2015 Data Innovations LLC
Page 3
5
EE11.2.23 -- My Community Hospital
Simple Precision
Report Interpretation Guide
error allocates a fraction of the Allowable Total Error for
Systematic Error (SEa) and the remaining fraction for
Random Error (REa). Establishing an appropriate Error
udget allows the lab to evaluate and control accuracy and
precision separately. Recommended ranges of values are
25-50% for the Systematic Error udget and 16-25% for the
Random Error udget at one SD. The SP module uses the
user entered TEa multiplied by the Random Error budget %
to calculate the allowable random error (REa) specification at
the observed mean.
,
B
B
B
Vendor Based Precision Goal
A vendor based precision goal allows the user to compare
observed statistics with vendor recommendations for
acceptable precision performance. Vendor performance
goals can typically be found in the manufacturer s insert
sheets for one or more levels. When using RRE data entry
techniques the vendor goals are extracted from user defined
entries in the SP tab of the Analyte Settings form in Policies.
Two types of vendor based precision goals are available for
selection.
1. The SD based format requires entry of a vendor within-run
precision SD. The concentration level at the vendor SD goal
is optional and is not used in calculations. This is the original
Vendor format used in EE and many manufacturers have
discontinued this format for their claims in favor of the REa
based format.
2. The REa based format is analogous to the REa portion of
the TEa. It requires entry of either a concentration
component or a Percent component representing 1 SD or
both. When using RRE data entry techniques the REa
based format is defined for each class in Modules and
Options and the vendor goals are extracted from user
defined entries in the SP tab of the Analyte Settings form.
'
,
,
,
,
,
Components of the Printed report
The components of the Simple Precision Report are the
Summary page (when more than one experiment is selected
for the report) Precision Statistics Table Precision Plot
Histogram Supporting Data Table User s Specifications
and Precision Data Table.
Summary page:
The summary page displays the following experiment
summary statistics:
 “N of N”: non-excluded data points / total N
Observations
 Mean: Obs mean / target mean. If optional Target mean
is not entered Obs mean / - is displayed unless is suppressed in the File Preferences Reports.
,
,
,
,
, "
EP Evaluator 11.2.0.23
Printed 27 Apr 2015 10:50:16
, "
"
"
\
"
\
,
"
"
"
\
"
\
=
,
,
=
,
z
,
’
,
"
\
SD: Obs SD / target SD goal at the level of the
Observed mean. If optional SD goal is not entered Obs
SD / - is displayed unless - is suppressed in
File Preferences Reports.
 %CV: Obs %CV / target CV% at the level of the
Observed mean. If optional SD goal is not entered
%CV / - is displayed unless - is suppressed in
File Preferences Reports.
Charts and Tables Interpretation:
Precision Plot: A Levey-Jennings type chart of Standard
Deviation Index (SDI) on the Y-axis vs. specimen index on
the X-axis. Specimen index reflects the order in which the
results were typed into the program. SDI (Result - Obs
mean) / SD unless target mean is defined. If target mean is
defined then SDI (result - target mean)/ target SD and
ero SDI would represent the target mean.
Precision Statistics Table: This table may include the
following:
Calculated Statistics – always present:
 Obs Standard Dev (SD)
 95% confidence for Obs SD
 Obs. Coefficient of Variation (CV)
 Observed Mean
 N
 95% CI for Obs Mean
 Obs 2 SD range
Precision Goal: This chart appears in the Precision
Statistics Table only when a Precision Verification Goal is
provided. It shows the Observed SD and its 95% confidence
interval in relation to the allowable random error goal. The
wide colored bar represents the SD. The fence marks above
and below the top of the SD bar represent the 95%
confidence limits.

"
\
"
Pass/Fail/Uncertain?
A line for Pass/Fail/Uncertain appears when a Precision
Verification Goal has been entered.
Yes means the experiment passes.
No means the experiment fails.
Uncertain means that more evaluation may be needed.
The ob ective is to determine whether precision is
acceptable. The default determination that the test passes is
if the computed SD does not exceed the goal. When the
designation Pass/Fail/Uncertain is enabled in Preferences
the test will pass only if the computed upper 95% CI around
the computed SD does not exceed the goal.
j
,
Copyright 1991-2015 Data Innovations LLC
Page 4
6
EE11.2.23 -- My Community Hospital
Simple Precision
Report Interpretation Guide
Example: Suppose Allowable Total Error for Sodium is 4
mmol/L with a Random Error udget of 25%. Allowable
Random Error is 25% x 4 or 1 mmol/L. The precision test
passes when the computed SD is at or below 1 mmol/L.
 The report calls the test a pass if the SD (top of the
wide colored bar) is less than or equal to the goal.
 The bar is green if the test passes or red if it fails.
 If you consider all points within the 95% confidence
interval to be statistically identical you might consider
the test at least marginally acceptable if the lower
confidence limit does not exceed the goal. When
enabled in Preferences the designation Uncertain
appears if the precision goal is within the 95%
confidence limits around the observed SD. In this case
the colored bar is yellow.
 The ideal situation is when the upper 95% confidence
limit does not exceed the goal.
Histogram: The Histogram chart appears when enabled in
the parameters screen. Observed data point results are
clustered around the Observed mean. If a target mean has
been entered optional SD target limits selectable from 1 SD
to 3 SD are drawn around the target mean. ias and % bias
are provided in the experiment detail screen but not shown
on the report.
Supporting Data Table: This table always includes the
analyst name experiment date and the analyte units.
Optional data entries include Target mean target range for
the histogram a Comment and Reagent Control or
calibrator lot numbers.
User’s Specifications: These only appear when a Precision
Verification Goal Mode (PVGM) is selected and will include
the type of PVGM selected and the corresponding entries as
described in the above section on PVGM.
Precision Data Table: This table includes the experiment s
observed results and their index (order of entry). Preliminary
Report
The word PRELIMINARY printed diagonally across the
report indicates that the data is incomplete and the report is
not acceptable as a final report. Some or all of the statistics
may be missing. The Simple Precision report is Preliminary if
there are fewer than 3 unexcluded results.
,
B
"
"
,
,
,
,
,
B
,
,
,
,
,
,
,
,
’
,
EP Evaluator 11.2.0.23
Printed 27 Apr 2015 10:50:16
Copyright 1991-2015 Data Innovations LLC
Page 5
7
GLUCOSE
Instrument ASSAYER
EE11.2.23 -- My Community Hospital
Accuracy and Linearity
CalKit-1
CalKit-2
CalKit-3
CalKit-4
CalKit-5
CalKit-6
Assigned
N
Accuracy & Recovery
Mean % Rec Status
25
100
250
400
700
4
4
4
4
4
25.5
101.5
249.5
407.8
690.3
1000
102.0
101.5
99.8
101.9
98.6
Linearity
Estimate Residual Status
Pass
Pass
Pass
Pass
Pass
26.4
101.4
251.4
401.3
701.2
4
938.0
93.8
Fail
See User's Specifications for Pass/Fail criteria.
Linearity Summary
-0.9
0.1
-1.9
6.5
-10.9
1001.1
-63.1
Pass
Pass
Pass
Pass
Pass
Fail
Experimental Results
Overall
w/o Outliers
0.971
1.000
3.6
1.5
2.3 mg/dl (conc) or
0.9 mg/dl (conc) or
3.9%
1.6%
6
5
N
NON-LINEAR within SEa of 1.5 mg/dl (conc) or 2.5%
Slope
Intercept
Obs Err
CalKit-1
CalKit-2
CalKit-3
CalKit-4
CalKit-5
CalKit-6
24
101
243
400
690
930
26
102
252
410
696
940
25
100
248
409
680
945
27
103
255
412
695
937
X: Excluded from calculations
User's Specifications
Allowable Total Error
Systematic Error Budget
Allowable Systematic Error
6 mg/dl (conc) or 10.0%
25%
1.5 mg/dl (conc) or 2.5%
Supporting Data
Analyst
Date
Value Mode
Units
Controls
Reagent
Calibrators
Comment
Kate Doe
01 Jun 2015
Pre-Assigned
mg/dl
ABC BR6745211 exp 31 Dec 2020
ABC G567843 exp 31 Dec 2019
ABC cal5768432 exp 31 Dec 2019
Evaluation of Results
The Accuracy and Linearity of GLUCOSE were analyzed on ASSAYER over a measured range of 25.5 to 938.0 mg/dl. This analysis
assumes accurate assigned values. Allowable systematic error (SEa) was 1.5 mg/dl (conc) or 2.5%. The accuracy test FAILED. The
maximum deviation for a mean recovery from 100% was 6.2%. 5 of 6 mean recoveries were accurate within the SEa. 24 of 24 results
were accurate within the allowable total error (TEa) of 6 mg/dl (conc) or 10.0%. The results are NON-LINEAR.
Accepted by:
Signature
EP Evaluator 11.2.0.23 (.A.L)
working document website sample reports Printed: 23 Jun 2015 14:51:32
Date
Copyright 1991-2015 Data Innovations LLC
Page 1
8
GLUCOSE
EE11.2.23 -- My Community Hospital
Instrument ASSAYER
Accuracy and Linearity
Scatter Plot
Percent Recovery
Residual Plot
History Plot
EP Evaluator 11.2.0.23 (.A.L)
working document website sample reports Printed: 23 Jun 2015 14:51:32
Copyright 1991-2015 Data Innovations LLC
Page 2
9
GLUCOSE
Instrument ASSAYER
EE11.2.23 -- My Community Hospital
Probability of Failure in Proficiency Testing
User's QC Statistics
Mean
SD
70
150
300
3
5
6
Control 1
Control 2
Control 3
PT Limits
Pct
Units
Reference:
10
6
Reference:
PT Analysis
Concentration Units
Assigned
PT
Conc'ns
Limits
CalKit-1
CalKit-2
CalKit-3
CalKit-4
CalKit-5
CalKit-6
25
100
250
400
700
1000
6.0
10.0
25.0
40.0
70.0
100.0
Percent of PT Limits
Linear
Recovery
Est. SD
Bias
Bias
31%
38%
23%
17%
12%
11%
16%
1%
7%
16%
16%
63%
8%
15%
2%
19%
14%
62%
Probability Result is Outside Limits
for
for
Linearity
Recovery
0.2%
0.5%
<0.01%
<0.01%
<0.01%
0.02%
0.1%
1%
<0.01%
<0.01%
<0.01%
0.02%
Linear bias based on Clinical Linearity
Failure Relationships
Probability
Failure
Outside Limits
Probability
Failure
Next Event
Probability
Failure
1 of Next 2
Probability
Failure
2 of Next 3
0.1%
0.2
0.5
1
2
5
10
20
50
0.001%
0.005
0.02
0.1
0.5
2
10
30
80
0.002%
0.01
0.05
0.2
1
5
15
45
95
0.0000%
0.0000
0.0000
0.0005
0.005
0.2
2
20
90
EP Evaluator 11.2.0.23 (.A.L)
working document website sample reports Printed: 23 Jun 2015 14:51:32
Copyright 1991-2015 Data Innovations LLC
Page 3
10
EE11.2.23 -- My Community Hospital
Linearity
Report Interpretation Guide
In EP Evaluator, Linearity experiments are experiments that
use specimens with defined concentrations. The Linearity
module can also evaluate Calibration Verification, Accuracy,
Reportable Range and Precision. This means that you can
verify three of the four major CLIA '88 requirements with one
properly designed experiment.
User-selectable options determine which of these analytical
properties the report verifies. Also, the user may request
Pass/Fail flags against a specific allowable error criterion, or
he/she may simply report selected statistical measures.
Experiment Procedure: Replicate measurements are made
on 3-11 specimens, with (known) concentrations spread
across the reportable range. Ideally, the lowest and highest
specimens should challenge the limits of the range.
Accuracy (or Recovery)
Definition: The ability to recover the correct amount of
analyte present in the specimen.
Verification process: Accuracy can be verified only when
the "correct" amount of analyte (the Assigned Value) is
known. W hile it is possible to evaluate recovery using a
single replicate, assaying 2 to 4 replicates provides a more
reliable estimate.
Key statistic: Recovery = 100 x Measured Mean / Assigned
Value
Reportable Range
Definition: As used in CLIA, Reportable Range refers to the
Analytical Range or Assay Range -- the maximum range of
values that can be assayed accurately without dilution. The
CAP term "Analytical Measurement Range" (AMR) is a
synonym for Reportable Range.
Verification Procedure: Reportable Range is verified if two
conditions are met: 1) the assigned values of the lowest and
highest specimens are within proximity limits of the
Reportable Range limits, and 2) these two specimens are
acceptably accurate.
Proximity Limits: Proximity Limits define how close the
lowest and highest specimens must be to the Reportable
Range limits.
Calibration Verification
Calibration Verification verifies whether a method is properly
calibrated by verifying both Accuracy and Reportable Range.
The report is titled to match the CAP and CLIA regulatory
requirements.
CLIA requires a minimum of three specimens, each assayed
in duplicate. Two specimens challenge the lower and upper
limits of the reportable range. The third specimen is
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 14:51:32
somewhere in between.
Linearity
Several definitions are in common use. Among them:
 Traditional Linearity (CAP Visual Inspection): Draw a
scatter plot with assigned values on the X-axis and
measured mean on the Y-axis. If it looks like a straight
line, the method is linear.
 Statistical Linearity (CLSI EP6P and EP6-A): These
procedures determine acceptability based on statistical
significance (i.e., p-values) rather than medical
significance. EP Evaluator does not compute Statistical
Linearity.
 Clinical Linearity: The method is linear if it is possible
to draw a straight line that passes within a user-defined
allowable error of each specimen point.
Related concepts:
 Best Fit Line: If the user opts to verify Linearity, this line
is obtained using the Clinical Linearity algorithm.
Otherwise it is a regular linear regression line.
 Outliers: W hen verifying Linearity, the program first
tries to determine an acceptable line using all
specimens. If it fails, it then tries to find some subset of
at least three specimens that are linear within allowable
error. Specimens not in this acceptable subset are
classified as outliers.
 Slope and Intercept: Coefficients of the Best Fit Line.
The ideal slope is 1.00; the ideal intercept is zero.
 Observed Error: For Clinical Linearity, this is the
minimum allowable error that could be defined for a data
set and still have it be linear.
 Standard Error of Estimate: For regular regression,
this measures the dispersion of the data points around
the Best Fit Line.
 Residual: The difference between the best fit line and
either an individual result or a mean measured value,
depending on context.
Precision
Definition: Ability to obtain the same result upon repeated
measurement of a specimen.
Verification Process: Measure the specimen many times.
Compute the SD and CV, and verify that they are acceptably
small. W hile 2-4 replicates are adequate for assessing
accuracy, a minimum of 10 (and preferably 20 or more) is
required to verify Precision.
Copyright 1991-2015 Data Innovations LLC
Page 4
11
EE11.2.23 -- My Community Hospital
Linearity
Report Interpretation Guide
The Precision Index is the ratio of SD to Allowable Random
Error (defined below). The ideal -- and probably unattainable
-- Precision Index is zero. A value of 1.00 indicates
borderline acceptability. Any further increase in SD would
exceed allowable error.
The 95% Confidence Interval (CI) for the Precision Index
indicates how much sampling variation might be expected.
The CI narrows as the number of replicates increases.
may be missing.
The Linearity report is preliminary if there are less than three
specimens.
Allowable Total Error (TEa), and the Error
Budget
TEa states the laboratory's policy for how much error is
medically (or administratively) acceptable. Regulatory
requirements represent an upper limit. Example: the CLIA
limit for Sodium is 4 mmol/L.
Total Error has two major components: Systematic Error
(synonym Bias) and Random Error (synonym Imprecision).
The Error Budget allocates a fraction of the Allowable Total
Error for Systematic Error, and the remaining fraction for
Random Error. Establishing an appropriate Error Budget
allows the lab to control accuracy and precision separately,
with reasonable confidence that Total Error will also remain
in control. The recommended upper value for the Systematic
Error Budget is 50%; for the Random Error Budget it is 25%.
Pass or Fail?
The program reports Pass/Fail for Accuracy and Linearity
based on Allowable Systematic Error (SEa). Pass/Fail for
Precision is based on Allowable Random Error (REa).
 A specimen passes Accuracy if its mean measured
value is within SEa of the Assigned Value.
 The experiment passes Linearity if it is possible to draw
a straight line (on the scatter plot of mean measured
value vs. assigned value) that passes within +/- SEa of
each specimen point.
 A specimen passes Precision if SD does not exceed
REa.
 The experiment passes Reportable Range if 1) the
assigned values of the lowest and highest specimens
are within proximity limits of the Reportable Range
Limits, and 2) these two specimens also pass accuracy.
Preliminary Report
The word PRELIMINARY printed diagonally across the
report indicates that the data is incomplete, and the report is
not acceptable as a final report. Some or all of the statistics
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 14:51:32
Copyright 1991-2015 Data Innovations LLC
Page 5
12
Glucose
Instrument Eximer 250
EE11.2.23 -- My Community Hospital
Simple Accuracy
Scatter Plot
Percent Recovery
Statistical Analysis and Experimental Results
Target
Range
Midpoint
L1
L2
28.2 to 35.6
578.6 to 612.2
31.9
595.4
Measured Values
32.7
603.8
32.6
602.5
(1)
(2)
Accuracy Rpt. Range
Pass
Pass
Pass
Pass
(1) Accuracy passes if all measured values lie within the Target Range. 'x' indicates an excluded result.
Supporting Data
Reportable Range Criteria
mkf
16 Jul 2009
mg/dL
25 to 625 mg/dL
ABC b6849321 exp 20 Dec 2020
ABC B1501251234 exp 31 Dec 2020
ABC cal6940333 exp 31 Dec 2020
Analyst
Date
Units
Reportable Range
Controls
Reagent
Calibrators
Comment
Reportable Range
Proximity Limits
Lower Limit
Upper Limit
25 to 625 mg/dL
12.5 to 37.5 mg/dL
562.5 to 687.5 mg/dL
(2) Reportable range passes if the highest and lowest
specimens meet accuracy criteria, and the midpoints of
their Target Ranges are within Proximity Limits
Evaluation of Results
The Accuracy and Reportable Range of Glucose were analyzed on Eximer 250 over a range of 31.9 to 595.4 mg/dL. The
accuracy test PASSED. All replicate measurements were within the target range for 2 of 2 specimens. Overall, 4 of 4 replicates
were within their target ranges. The system PASSED reportable range tests.
Accepted by:
Signature
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 14:57:16
Date
Copyright 1991-2015 Data Innovations LLC
Page 1
13
EE11.2.23 -- My Community Hospital
Simple Accuracy
Report Interpretation Guide
The Simple Accuracy experiment tests whether
measurements fall within a defined Target Range. The
experiment requires at least two specimens, assayed in
duplicate. Ideally, the lowest and highest specimens should
challenge the limits of the Reportable Range. Verifying
Reportable Range in conjunction with the Simple Accuracy
experiment is optional, but recommended.
Preliminary Report
The word PRELIMINARY printed diagonally across the
report indicates that the data is incomplete, and the report is
not acceptable as a final report. Some or all of the statistics
may be missing.
 This report is preliminary if there are fewer than 2
specimens, or fewer than 2 replicates per specimen.
Accuracy (or Recovery)
Definition: The ability to recover the correct amount of
analyte present in the specimen.
Verification process: Accuracy is verified when all replicate
measurements at each level lie within the Target Range at
that level.
Key statistic: Recovery = 100 x Measured Value / Midpoint
of Target Range
Reportable Range
Definition: As used in CLIA, this term refers to the Analytical
Range or Assay Range -- the maximum range of values that
can be assayed accurately without dilution. The CAP term
"Analytical Measurement Range" (AMR) is a synonym for
Reportable Range.
Verification Procedure: Reportable Range is verified if two
conditions are met: 1) the Target Range midpoints for the
lowest and highest specimens respectively are within
proximity limits of the Reportable Range limits, and 2) these
two specimens are acceptably accurate.
Proximity Limits define how close the lowest and highest
specimens must be to the Reportable Range limits.
Data Requirements
The experiment accepts up to 11 specimens, and up to 5
replicates per specimen. The minimum is 2 specimens,
assayed in duplicate. A Target Range is required for each
specimen. On the Scatter and Percent Recovery plots, the
plotted X value is the midpoint of the target range.
Pass or Fail?


A specimen passes Accuracy if all measured results fall
within the target range.
The experiment passes Reportable Range if 1) the
midpoints of the target ranges for the lowest and highest
specimens respectively are within proximity limits of the
Reportable Range Limits, and 2) these two specimens
also pass accuracy.
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 14:57:16
Copyright 1991-2015 Data Innovations LLC
Page 2
14
BUN
EE11.2.21 -- My Community Hospital
Alternate (Quantitative) Method Comparison
X Method Dina 1
Y Method Dina 2
Scatter Plot
Bias
Percent Bias
Regression Analysis
Slope
Intercept
Std Err Est
SMAD
Deming
1.013 (1.000 to 1.025)
1.051 (0.577 to 1.525)
1.575
1.037
Passing-Bablok
1.011 (1.000 to 1.028)
0.938 (0.666 to 1.045)
-0.991
Regular
1.010 (0.998 to 1.023)
1.113 (0.639 to 1.587)
1.574
1.037
95% Confidence Intervals are shown in parentheses
Medical Decision Point Analysis
Calculated by Deming Regression (R>=0.9)
X Method
MDP
Y Method
Pred. MDP
6
20
7.1
21.3
95% Conf. Limits
Low
High
6.7
21.0
7.5
21.6
Supporting Statistics
Corr Coef (R)
Bias
X Mean ± SD
Y Mean ± SD
0.9980
1.425 (4.670 %)
29.798 ± 24.338
31.223 ± 24.642
Std Dev Diffs
SubRange Bounds
Points (Plotted/Total)
Outliers
1.588
None
109/110
1
Scatter Plot Bounds Allowable Total Error
2.000 mg/dl (conc) or 9.0%
Experiment Description
Expt Date
Rep SD
Result Ranges
Units
Reagent
Calibrators
Analyst
Comment
X Method
02 Apr 2015
1
3.00 to 118.00
mg/dl
ABC B1501251234 exp 31 Dec 2020
-Inez Doe
Dina 1 comment
Y Method
02 Apr 2015
1
4.00 to 120.00
mg/dl
ABC B1501251234 exp 31 Dec 2020
-Inez Doe
Dina 2 Comment
Accepted by:
Signature
EP Evaluator 11.2.0.23
website sample reports Printed: 02 Apr 2015 17:11:46
Date
Copyright 1991-2015 Data Innovations LLC
Page 1
15
BUN
EE11.2.21 -- My Community Hospital
Alternate (Quantitative) Method Comparison
X Method Dina 1
Y Method Dina 2
Experimental Results
Y
Bias
Calc'd SEE
Y
Factor
23.82
21.88
53
83
57
1.82
0.88
4
5
2
23.327
22.315
50.666
80.029
56.741
8.5
90
81.5
9.658
58
15
63
61
62
110
71
56
9
18
13
8
11
10
13
10
20
24
17
16
14
14
9
12
11
15
12
14
8
16
13
19
22
11
13
12
22
26
38
22
66
41
43
73
63
16.09
65
61
63
109
73
58
9.71
19.91
13.56
8.68
11.96
10.86
15.08
11.55
20.57
24.74
19.11
16.3
15.18
16.11
10.48
13.42
11.87
16.54
14.02
15.68
9.58
17.43
14.28
19.99
22.8
11.49
14.34
13.45
22.27
25.25
40.71
22.8
70
39
49
75
Spec ID
X
1
2
3
4
5
1
10
100
1009
1022
22
21
49
78
55
6
1025
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
1057
11
1102
1104
1132
1133
1164
1165
12
13
14
15
16
17
18
19
2
20
21
22
23
24
25
26
27
28
29
3
30
31
32
33
34
35
36
37
38
39
4
40
41
42
43
44
O
Results
5
59.779
1.09 16.239
2
64.841
0
62.816
1
63.829
-1 112.431
2
72.942
2
57.754
0.71 10.164
1.91 19.277
0.56 14.214
0.68 9.152
0.96 12.189
0.86 11.177
2.08 14.214
1.55 11.177
0.57 21.302
0.74 25.352
2.11 18.264
0.3 17.252
1.18 15.227
2.11 15.227
1.48 10.164
1.42 13.202
0.87 12.189
1.54 16.239
2.02 13.202
1.68 15.227
1.58 9.152
1.43 17.252
1.28 14.214
0.99 20.290
0.8 23.327
0.49 12.189
1.34 14.214
1.45 13.202
0.27 23.327
-0.75 27.377
2.71 39.528
0.8 23.327
4
67.879
-2
42.565
6
44.590
2
74.967
0.3
-0.3
1.5
1.9
0.2
51.0
2.0
-0.1
0.1
-1.2
-0.5
-2.2
0.0
0.2
-0.3
0.4
-0.4
-0.3
-0.1
-0.2
0.5
0.2
-0.5
-0.4
0.5
-0.6
0.0
0.6
0.2
0.1
-0.2
0.2
0.5
0.3
0.3
0.1
0.0
-0.2
-0.3
-0.4
0.1
0.2
-0.7
-1.4
0.8
-0.3
1.3
-2.3
2.8
0.0
Spec ID
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
45
46
47
48
49
5
50
51
52
53
54
55
56
57
58
59
6
60
61
62
63
64
65
66
67
68
69
7
70
71
72
73
74
75
76
77
78
79
8
80
81
82
83
84
85
86
87
88
89
9
X
Results
50
39
61
51
19
37
53
15
7
31
51
57
12
54
100
71
14
43
26
3
14
8
5
11
46
118
39
16
6
6
24
20
20
8
19
5
9
9
9
31
9
9
91
6
25
32
12
45
7
16
Y
Bias
Calc'd SEE
Y
Factor
53
40
67
51
19
39.73
56
16
8
31
49
56
13
53
100
70
14.56
40
27
4
15
8
6
12
52
120
40
17.55
8
7
28
20
23
8
20
6
9
11
10.86
31
11
9
94
8
26
34
14
49
8
17.11
3
1
6
0
0
2.73
3
1
1
0
-2
-1
1
-1
0
-1
0.56
-3
1
1
1
0
1
1
6
2
1
1.55
2
1
4
0
3
0
1
1
0
2
1.86
0
2
0
3
2
1
2
2
4
1
1.11
51.678
40.540
62.816
52.691
20.290
38.515
54.716
16.239
8.139
32.440
52.691
58.766
13.202
55.728
102.305
72.942
15.227
44.590
27.377
4.089
15.227
9.152
6.114
12.189
47.628
120.531
40.540
17.252
7.127
7.127
25.352
21.302
21.302
9.152
20.290
6.114
10.164
10.164
10.164
32.440
10.164
10.164
93.192
7.127
26.365
33.453
13.202
46.616
8.139
17.252
0.8
-0.3
2.7
-1.1
-0.8
0.8
0.8
-0.2
-0.1
-0.9
-2.3
-1.8
-0.1
-1.7
-1.5
-1.9
-0.4
-2.9
-0.2
-0.1
-0.1
-0.7
-0.1
-0.1
2.8
-0.3
-0.3
0.2
0.6
-0.1
1.7
-0.8
1.1
-0.7
-0.2
-0.1
-0.7
0.5
0.4
-0.9
0.5
-0.7
0.5
0.6
-0.2
0.3
0.5
1.5
-0.1
-0.1
Values marked with an "X" were excluded from the calculations. Outliers "O" were also excluded.
EP Evaluator 11.2.0.23
website sample reports Printed: 02 Apr 2015 17:11:46
Copyright 1991-2015 Data Innovations LLC
Page 2
16
BUN
EE11.2.21 -- My Community Hospital
Alternate (Quantitative) Method Comparison
X Method Dina 1
Y Method Dina 2
Experimental Results
101
102
103
104
105
Spec ID
X
90
91
92
93
94
27
28
62
52
15
Results
Y
Bias
28
29
66
53
18
1
1
4
1
3
Calc'd SEE
Y
Factor
28.390
29.402
63.829
53.703
16.239
-0.2
-0.3
1.4
-0.4
1.1
106
107
108
109
110
Spec ID
X
95
96
97
98
99
40
41
26
43
9
Results
Y
Bias
43
43
25
48
11
3
2
-1
5
2
Calc'd SEE
Y
Factor
41.553
42.565
27.377
44.590
10.164
0.9
0.3
-1.5
2.2
0.5
Values marked with an "X" were excluded from the calculations. Outliers "O" were also excluded.
EP Evaluator 11.2.0.23
website sample reports Printed: 02 Apr 2015 17:11:46
Copyright 1991-2015 Data Innovations LLC
Page 3
17
EE11.2.21 -- My Community Hospital
Alternate (Quantitative) Method Comparison
Report Interpretation Guide
There are many reasons for doing method comparison
studies. Perhaps the most common:
 To determine the relationship of the Medical Decision
Points (MDPs) of an old method with those of a new
method. In other words, "Can I continue to use the same
MDPs with the new method?"
 To validate a new method being brought into the lab, by
demonstrating the statistical relationship to the method
currently in use.
The statistical tool used is linear regression. The methods
can be considered statistically identical if:
 The slope is 1.00 (within 95% confidence)
 The intercept is 0.00 (within 95% confidence)
 The predicted Y MDPs are equal to the X MDPs (within
95% confidence)
Not all regressions are method comparisons. This Report
Interpretation assumes that X and Y are alternative methods
for measuring the same quantity, and that the purpose of the
experiment is to determine whether X is statistically identical
to Y. If the purpose is to predict weight as a function of
height, or to predict APTT levels from Heparin levels, some
of the interpretive comments may not apply.
Regression Approaches
The report shows at least two, and (optionally) three sets of
regression coefficients.
Regular Regression: This is the ordinary least squares
regression line commonly provided in spreadsheets and
general statistical software. It is shown only to provide a
familiar frame of reference; it is not used to estimate Medical
Decision Points. The problem with using regular regression
to compare methods is that it assumes the X method is
measured with no random error -- not very likely for clinical
laboratory results. Regular regression often underestimates
the true slope, sometimes by a significant amount.
Deming Regression: This approach assumes that both the
X and Y methods are subject to measurement error. In
theory, a Representative SD (precision estimate) is input for
each method. In practice, only the ratio of the two precisions
affects the calculation. If exact precisions are unknown,
entering 1.0 for both Representative SDs says "these
methods have about the same precision", and gives
reasonable results in most cases.
Several studies have shown that Deming Regression is the
best approach to use when the two methods are expected to
be identical, and the data is well-distributed and free of
outliers. It can, however, be seriously affected by outliers. EP
Evaluator provides the option to automatically exclude
extreme outliers, or the user can exclude them manually.
EP Evaluator 11.2.0.23
Printed 02 Apr 2015 17:01:47
All Regression Lines on the AMC graphs are Deming
Regression Lines. W hen MDPs are estimated by linear
regression, Deming linear regression is used.
Passing-Bablok Regression: Passing-Bablok regression is
a non-parametric regression technique developed
specifically to be resistant to outliers. Non-parametric
statistical calculations make no assumptions about the
shape of the population distribution, while parametric
approaches assume that the population has a Gaussian
distribution.
Main strengths: There is no need to exclude perceived
outliers, either manually or automatically. Like Deming, it
does not assume that X is free from error. Comparative
studies show that it performs about as well as Deming
Regression in most cases, and better than Deming when
outliers are present.
Main weaknesses: Passing-Bablok is computationally
intensive, particularly for large N, and it may be unreliable for
very small N. EP Evaluator does not show Passing-Bablok
statistics when N<10 or N>500.
Removing Outliers
An outlier is a point so far from the others as to arouse
suspicion that it was generated by a different mechanism.
Some common causes: typing a number with the decimal
point in the wrong place, analyzing the wrong sample, or
entering incorrect specimen identification. The best way to
deal with an outlier is to (manually) determine its cause and
correct it. Another option is to use a statistical procedure to
remove outliers automatically.
EP Evaluator uses a somewhat complex iterative algorithm
to identify outliers. The goal is to eliminate points whose
distance from the regression line exceeds 10 times the
Standard Error of Estimate (SEE), where SEE is computed
not from the full data set, but from the data set with outliers
excluded. W hen outliers are included, the SEE is
over-stated, and the regression coefficients are suspect.
An outlier is, by definition, a rare occurrence. If more than
5% of the points are excluded, either by the mathematical
algorithm or the user, the report is stamped PRELIMINARY.
It is then incumbent on the user to re-evaluate both methods
to determine their suitability.
Interpreting your Results
hen interpreting a method comparison report, there are
two areas which must be addressed:
 First, is the QUALITY OF THE DATA adequate to
W
Copyright 1991-2015 Data Innovations LLC
Page 4
18
EE11.2.21 -- My Community Hospital
Alternate (Quantitative) Method Comparison
Report Interpretation Guide
accurately draw conclusions?
 Second, what conclusions can be drawn from those
data?
These issues MUST be addressed in this order. If the data
quality is not adequate, then any additional conclusions
drawn from those data may well be wrong.
Data Quality Statistics
The most important elements of a good method comparison
study are a reasonable N (number of x-y pairs) and a good
distribution of results. Generally a good experiment will
include 30 to 50 specimens with their results distributed
more or less evenly across the method's reportable range.
Results Range: The minimum and maximum values of X
and Y. It is inappropriate to draw conclusions outside the
range of data studied. W hen evaluating MDPs, it is important
to include data points that cover the full range of MDPs.
Result Range Analysis: This (optional) table shows how the
X values are distributed within the range. A relatively even
distribution is desirable. If 99% of the values are at the low
end and 1% are at the high end, with none in the middle, the
regression slope is almost totally determined by the handful
of high points.
Points (Plotted/Total): More commonly called N, the
number of x-y pairs in the regression. "Plotted" is the number
on which calculations are based. The difference between
Plotted and Total is points that were excluded, either
manually or by the automatic outlier removal procedure.
CLSI considers N=40 to be the minimum for a good method
comparison study. Increasing N improves the quality up to a
point, but a good distribution of data is much more important
than a large N.
Correlation Coefficient (R): R generally corresponds to the
width of an ellipse drawn around the data. The narrower the
ellipse relative to its length, the higher R will be. If there is
large amount of error, the width of the ellipse will increase
which will in turn cause a lower R.
R ranges from -1 to 1. Zero means there is absolutely no
relationship. +1 or -1 means there is a perfect relationship,
and a very high-quality regression. An R of 1.000 could be
achieved just as easily with a slope and intercept of 1.000
and 0.0 as with a slope and intercept of 0.5 and 400
respectively. In other words, it specifies the degree of
correlation, not the degree to which the two methods match.
.
In a method comparison setting, R has special significance:
 A small R may be a sign that the Results Range is
inadequate. Adding samples to increase the range of X
will improve both the R value, and the quality of the
EP Evaluator 11.2.0.23
Printed 02 Apr 2015 17:01:47

study.
If R is less than a user-selectable cutoff value (0.90,
0.95, or 0.975), regression is not used to evaluate
Medical Decision Points. Instead, they are evaluated by
the method of Partitioned Biases, as explained in more
detail, below.
Interpreting the Regression Statistics
Assuming that the quality of the data is adequate, you may
proceed to interpreting the results.
Slope, Intercept, and their Confidence Intervals: W hen
two methods are statistically identical, the 95% confidence
interval for the slope includes 1.00, and the 95% confidence
interval for the intercept includes 0.0.
Example: If the 95% CI for the slope is 0.92 to 1.02, 1.00 is
included in the interval. However, if the 95% CI is 0.82 to
0.92, 1.00 is not included in the interval.
If the experiment were repeated with different data, the slope
and intercept would be a bit different. But 95% of such
experiments are expected to fall within the confidence
interval.
Medical Decision Point Analysis: A Medical Decision Point
is an analyte concentration at which medical decisions
change. If the concentration is to one side of the MDP, one
decision is made; if on the other side of the MDP, a different
decision is made. For example, Fasting Plasma Glucose
above 126 mg/dL (7 mmol/L) indicates hyperglycemia which,
if confirmed, establishes a diagnosis of diabetes. For
obvious reasons, it is particularly important that the two
methods agree at the MDPs.
W hen the two methods are statistically identical, the 95%
Confidence Interval for each Y MDP includes the
corresponding X MDP.
Standard Error of Estimate (SEE): measures the spread of
the x-y data around the linear regression line. If both
methods have the same constant precision SD across the
full analytical range, SEE should be about 1.4 times the
precision SD.
Bias, and its Relationship with Regression
Bias is the difference Y-X. The Bias Plot is a scatter plot
with X on the x-axis, and Y-X on the y-axis. The ideal bias
plot would have all points falling exactly on the zero line.
That is unlikely to occur in practice, because both X and Y
are measured with some random error. A good bias plot is
centered on the zero-line, and forms an envelope of
approximately constant width about it.
Copyright 1991-2015 Data Innovations LLC
Page 5
19
EE11.2.21 -- My Community Hospital
Alternate (Quantitative) Method Comparison
Report Interpretation Guide
Constant Bias is present when Y is consistently greater
than (or less than) X by a constant amount. The bias plot
forms a constant-width envelope around the average bias
line instead of the zero line. The regression intercept
measures constant bias. In fact, if the slope is exactly 1.000,
the regression intercept is equal to the average bias.
Proportional Bias is present when Y differs from X in a way
that is proportional to X. For example, Y may be consistently
5% higher than X instead of 5 units higher. On the Bias plot,
the points center around an upward or downward-sloping line
instead of a horizontal line. The regression slope is a
measure of proportional bias.
The Method of Partitioned Biases comes into play when R
is below the cutoff value (0.90, 0.95, or 0.975). In this
situation, the Bias Plot is divided into three segments, with
the same number of points in each segment. It is assumed
that bias is approximately constant within each segment.
This segmented structure provides an estimate of bias and
its 95% confidence interval at the Medical Decision Points.
The method of Partitioned Bias is an alternative approach
used to define medical decision points. If it is used then
slope, etc. are not used.
Preliminary Report
The word PRELIMINARY printed diagonally across the
report indicates that the data is incomplete, and the report is
not acceptable as a final report. Some or all of the statistics
may be missing. Causes:
 Less than 3 unexcluded x-y pairs.
 More than 5% of points are outliers.
 Excluding outliers reduced the range of X by more than
50%. The range of X is a significant aspect of data
quality, and it should be confirmed by the analyst in this
case, rather than by a mathematical algorithm.
EP Evaluator 11.2.0.23
Printed 02 Apr 2015 17:01:47
Copyright 1991-2015 Data Innovations LLC
Page 6
20
Glucose
Experiment MIC-Q4-2014
EE11.2.21 -- My Community Hospital
Multiple Instrument Comparison
Error Index by Instrument
All within Allowable Error? No
Range of Target Values 51.0 to 193.0
User's Specifications
Supporting Data
Allowable Total Error
6 mg/dl (conc) or 10%
Reportable Range
0 to 550 mg/dl
Target value is median for 4 comparative instrument(s).
Analyst
Units
Number of Specimens
Expt Date
Comment
Coverage Ratio 26%
dgr
mg/dl
5 of 5
11 Dec 2000
Accepted by:
Signature
EP Evaluator 11.2.0.23
website sample reports Printed: 07 Apr 2015 17:09:11
Date
Copyright 1991-2015 Data Innovations LLC
Page 1
21
Glucose
Experiment MIC-Q4-2014
EE11.2.21 -- My Community Hospital
Multiple Instrument Comparison
Scatter Plot (all instruments)
Error Index by Concentration (all instruments)
EP Evaluator 11.2.0.23
website sample reports Printed: 07 Apr 2015 17:09:11
Copyright 1991-2015 Data Innovations LLC
Page 2
22
Glucose
Experiment MIC-Q4-2014
EE11.2.21 -- My Community Hospital
Multiple Instrument Comparison
Results Listing
Obs.
Allow Error
Obs.
Spec.
Result
Target
Error
Error Index
Spec.
Result
Target
Error
POC-20
POC-21
POC-Excel, 175 S/N A-1234
POC-Excel, 175 S/N A-2468
A01
48
51.0
-3.0
±6.0 -0.50
A01
52
51.0
1.0
A02
82
83.0
-1.0
±8.3 -0.12
A02
84
83.0
1.0
A03
104
114.0
-10.0
±11.4 -0.88
A03
120
114.0
6.0
A04
150
155.0
-5.0
±15.5 -0.32
A04
156
155.0
1.0
A05
188
193.0
-5.0
±19.3 -0.26
A05
191
193.0
-2.0
Lab-Inst A
POC-22
SMA-750, 750 S/N C-423
POC-Excel, 175 S/N A-2555
A01
50
51.0
-1.0
±6.0 -0.17
A01
54
51.0
3.0
A02
95
83.0
12.0
A02
80
83.0
-3.0
±8.3 -0.36
A03
115
114.0
1.0
±11.4 0.09
A03
113
114.0
-1.0
A04
154
155.0
-1.0
±15.5 -0.06
A04
160
155.0
5.0
A05
195
193.0
2.0
±19.3 0.10
A05
201
193.0
8.0
F: value exceeds allowable error; X: specimen excluded from the analysis; T: target instrument.
EP Evaluator 11.2.0.23
website sample reports Printed: 07 Apr 2015 17:09:11
Allow Error
Error Index
±6.0
±8.3
±11.4
±15.5
±19.3
0.17
0.12
0.53
0.06
-0.10
±6.0
±8.3
±11.4
±15.5
±19.3
0.50
1.45 F
-0.09
0.32
0.41
Copyright 1991-2015 Data Innovations LLC
Page 3
23
EE11.2.21 -- My Community Hospital
Multiple Instrument Comparison
Report Interpretation uide
G
Multiple Instrument Comparison (MIC) compares two or
more instruments to determine whether they are within Total
Allowable Error (TEa) of the target.


"
Select 2-30 instruments to compare.
Some of these instruments may be target instruments.
Target instruments are not evaluated. They are, instead,
used to establish assigned values (target values) for the
specimens. At least two of the instruments must be
non-target instruments.
Assay a minimum of 3 specimens on each instrument (5
to 10 are recommended). Large numbers of specimens
(>10) are not required.
Ideally, specimens should be distributed uniformly
across the Reportable Range. You may use either
patient specimens or manufactured specimens
(calibrators, QC specimens, linearity standards).
By default, EE requires that for a specimen to be
included, a result must be present for every instrument;
otherwise that specimen is excluded from the analysis. If
you are going to be evaluating many instruments, it may
be useful to tell EE to relax this requirement. See the
Help or the User s Manual for details on how to do this.
'
Key Statistics and Evaluation Criteria
Total Allowable Error (TEa). TEa states the laboratory s
'
policy for how much error is medically (or administratively)
acceptable. Regulatory requirements represent an upper
limit.
Examples: the CLIA limit for Sodium is 4 mmol/L; the CLIA
limit for lucose is 6 mg/dL or 10%, whichever is greater.
Error Index. The ratio of the difference (Result - Target) to
Total Allowable Error. The Error Index is measured for each
specimen on each instrument. An index greater than 1.00 or
less than -1.00 is unacceptable -- it means the difference
between the instrument and the target exceeds TEa. If any
data point has an unacceptable Error Index, the experiment
fails.
Reportable Range. The maximum range of values that can
be measured accurately without diluting the specimen.
Synonyms: Analytical Measurement Range, Assay Range,
Analytical Range.
Target Value. If some of the instruments are target
instruments, the target value is the median result over the
target instruments. therwise it is the median result over all
instruments. A Target Value is computed for each specimen.
G
O
EP Evaluator 11.2.0.23
Printed 07 Apr 2015 17:09:11
error indices of the data points. The calculation does not
include results that have been excluded, or failed data points
(with an error index > 1.0). The overall error index mean is
shown as a vertical dotted line labeled verall mean on the
Error index By Instrument chart.
Coverage Ratio. The percent of the Reportable Range
covered by the analysis. The ideal is 100%.
Coverage is computed as 100 x (Xhi - Xlo) / (Rhi - Rlo). Rlo
and Rhi are the lower and upper limits of the Reportable
Range. Xhi is the smaller of the maximum Target Value and
the upper limit of the Reportable Range. Xlo is the larger of
the minimum Target Value and the lower limit of the
Reportable Range.
Example: Suppose the Reportable Range is 100 to 300, and
the range of Target Values is 200 to 350. There is a 50%
overlap between the target value range and the Reportable
Range. The coverage ratio is 100 x (300 - 200) / (300 - 100)
50%.
"O
Experiment Design

Overall Mean. The program computes the average of the
"
"
=
Plots
Error Index by Instrument. This plot shows the Error Index
for each non-target instrument on a bar chart, with a
separate hori ontal bar for each instrument. If either end of a
bar extends into the yellow (unacceptable) area, that
instrument differs from the target by more than TEa.
After completing a MIC experiment, you may choose to save
the results to history before entering new results for the next
time period. If you have saved history, the Error Index by
Instrument plot includes bars for both the current period and
the previous two history periods. The current period bar
shows a circle mar ing each separate specimen.
Scatter Plot. This chart plots result data from all the
instruments (Y-axis) against Target Value (X-axis). You can t
tell from loo ing at this plot which point comes from which
instrument. However, you can see which concentrations
have the most error.
Example: Suppose you have three instruments and five
concentrations. The Error Index by Instrument plot shows
that error for all three instruments exceeds TEa. The Scatter
Plot might show that the problem is confined to the
low-concentration specimen.
Error Index by Concentration. Li e the Scatter Plot, this
alternate form of the Error Index Plot combines all the
instruments, and plots their Error Index against target value.
z
k
'
k
k
Pass or Fail?
The experiment passes if every non-target instrument is
Copyright 1991-2015 Data Innovations LLC
Page 4
24
EE11.2.21 -- My Community Hospital
Multiple Instrument Comparison
Report Interpretation uide
G
within TEa of the target for every specimen.
Preliminary Report
The word PRELIMINARY printed diagonally across the
report indicates that the data is incomplete, and the report is
not acceptable as a final report. Some or all of the statistics
may be missing.
The MIC report is preliminary if there are less than three
unexcluded specimens.
EP Evaluator 11.2.0.23
Printed 07 Apr 2015 17:09:11
Copyright 1991-2015 Data Innovations LLC
Page 5
25
TDM PHN
EE11.2.23 -- My Community Hospital
Semi-Quantitative Method Comparison
Ref Method Chem Assay
Test Method Analyzer
Statistical Analysis
(Comparison of two Laboratory Methods)
Agreement
Agreement within two
72.3% (61.8 to 80.8%)
98.8% (93.5 to 99.8%)
95% confidence interval calculated by the "Score" method.
McNemar Test for Symmetry:
Test < Reference
22 (26.5%)
Test > Reference
1 (1.2%)
Symmetry test FAILS p < 0.001 (ChiSq=19.174, 1 df)
A value of p<0.05 suggests that one method is consistently "larger".
Cohen's Kappa
61.6% (48.2 to 74.9%)
Kappa is the portion of agreement above what is expected by chance.
The rule of thumb is that Kappa > 75% indicates "high" agreement.
We would like to see VERY high agreement (close to 100%).
Statistical Summary
Test
1
2
3
4
5
Total
Legend
Reference
1
2
3
4
5
Total
10
5
--
--
--
15
1
--
20
4
--
--
24
2
--
--
30
3
--
33
1
--
--
--
10
11
--
--
--
--
--
0
4
11
25
34
3
10
83
5
3
Reference
very Neg
(<=101)
Negative
(102 to 200)
Equivocal
(201 to 298)
positive
(299 to 350)
Very positive
(>350)
Test
very Neg
(<=101)
Negative
(102 to 200)
Equivocal
(201 to 298)
positive
(299 to 350)
Very positive
(>350)
Number excluded or missing: 0
Experiment Description
Analyst:
Date:
Comment
Reference Method
Test Method
mkf
03 Feb 2015
mkf
03 Feb 2015
Accepted by:
Signature
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:05:21
Date
Copyright 1991-2015 Data Innovations LLC
Page 1
26
TDM PHN
EE11.2.23 -- My Community Hospital
Semi-Quantitative Method Comparison
Ref Method Chem Assay
Test Method Analyzer
Experimental Results
Spec ID
Ref
Test
S00007
S00006
S00008
S00010
S00009
S00002
S00001
S00003
S00005
S00004
30
30
30
30
30
30
30
30
30
30
42
42
42
42
42
42
42
42
42
42
98
300
178
178
178
178
178
194
194
194
194
194
194
194
194
194
194
194
194
23
23
23
23
23
159
159
159
159
159
159
159
159
159
159
159
159
S00073
S00014
S00015
S00011
S00012
S00013
S00029
S00030
S00027
S00028
S00031
S00034
S00035
S00032
S00033
S00026
S00019
S00020
F
Spec ID
Ref
S00018
194
S00016
194
S00017
194
S00024
194
S00025
194
S00023
194
S00021
194
S00022
194
S00059
277
S00058
277
S00061
277
S00060
277
S00055
277
S00054
277
S00057
277
S00056
277
S00067
277
S00066
277
S00069
277
S00068
277
S00063
277
S00062
277
S00065
277
S00064
277
S00053
277
S00043
277
S00044
277
S00045
277
X:Excluded F:No Agreement
Test
Spec ID
Ref
Test
159
159
159
159
159
159
159
159
263
263
263
263
263
263
263
263
263
263
263
263
263
263
263
263
263
263
263
263
S00040
S00041
S00042
S00046
S00050
S00051
S00052
S00047
S00048
S00049
S00037
S00036
S00039
S00038
S00072
S00070
S00071
S00080
S00079
S00081
S00083
S00082
S00075
S00074
S00076
S00078
S00077
277
277
277
277
277
277
277
277
277
277
286
286
286
286
331
331
331
369
369
369
369
369
369
369
369
369
369
263
263
263
263
263
263
263
263
263
263
104
104
104
104
292
292
292
300
300
300
300
300
300
300
300
300
300
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:05:21
Copyright 1991-2015 Data Innovations LLC
Page 2
27
EE11.2.23 -- My Community Hospital
Qualitative Method Comparison
Report Interpretation Guide
Qualitative Method Comparison
Sensitivity: The probability that the test will be positive in a
Gold Standard: A method that is absolutely correct.
True Positive (TP): A specimen that tests positive by both
population in which everyone should test positive. The ideal
sensitivity is 100%. Applicable only when the reference
method is a Gold Standard.
Specificity: The probability that the test will be negative in a
population in which everyone should test negative. The ideal
specificity is 100%. Applicable only when the reference
method is a Gold Standard.
Prevalence: The frequency of positives in a given population
-- possibly some population other than the one evaluated in
the method comparison experiment.
Predictive Value: These statistics vary with prevalence, and
are applicable only when the reference method is a Gold
Standard.
 Predictive Value Positive: Probability that a positive
result accurately shows the presence of the underlying
condition.
 Predictive Value Negative: Probability that a negative
result accurately shows the absence of the underlying
condition.
Agreement: The percent of total cases in which the two
methods give the same result. Related statistics for
qualitative tests:
 Positive Agreement is the percent of cases that match
when the reference method is positive: TP/(TP+FP).
 Negative Agreement is the percent of cases that match
when the reference method is negative: TN/(TN+FN).
Cohen's Kappa: Similar to Agreement, but adjusted for the
probability that the two methods agree by chance. Kappa
ranges from -100% to 100%. A value of zero indicates
random agreement. A value of 100% indicates perfect
agreement. It is desirable for Kappa to be well above 75%.
McNemar Test for Symmetry: A test for bias -- whether one
method is consistently larger than the other. If the number of
cases where X>Y is equal (within random error) to the
number of cases X<Y, the method is unbiased, and the
symmetry test passes. If most of the differences between X
and Y occur when X>Y (or when X<Y), the symmetry test
fails.
True Negative (TN): A specimen that tests negative by both
Preliminary Report
False Negative (FN): A specimen that is negative by the test
The word PRELIMINARY printed diagonally across the
report indicates that the data is incomplete, and the report is
not acceptable as a final report. Some or all of the statistics
may be missing.
This report is preliminary if there are less than 20
unexcluded results.
The purpose of this experiment is to compare two methods
that report results as Positive/Negative. QMC is useful in the
following circumstances:
 To determine the ability of an instrument to detect a
pathological condition.
In this case, the reference method is a definitive
diagnosis, or Gold Standard. The key statistics are
Sensitivity and Specificity, and the proportion of False
Positives and False Negatives.
 To compare the performance of two laboratory
instruments, perhaps when a new instrument is
introduced into the lab.
Here both the reference method and the test method
are subject to experimental error. There is no reason to
assume the reference method is correct, or even that it
is better than the test method. In fact, when the
reference method is an old method and the test method
is a new method, it is quite possible that the new
method is the better method.
When comparing two laboratory methods, the key
statistic is Agreement -- do the two methods give the
same results?
Semi-Quantitative Method Comparison
Purpose: to compare methods that return non-quantitative
results. Example: a study comparing a dipstick method for
urinary protein to a quantitative method. The results for the
quantitative method are divided into groups corresponding to
the dipstick categories. This approach can be used to
compare semi-quantitative tests with quantitative tests. The
key statistic used in semi-quantitative comparison is
Agreement.
Definitions
methods.
methods.
method and positive by the reference method, with the
implicit assumption that the reference method is correct.
False Positive (FP): A specimen that is positive by the test
method and negative by the reference method, again with
the implicit assumption that the reference method is correct.
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:05:21
Copyright 1991-2015 Data Innovations LLC
Page 3
28
EE11.2.23 -- My Community Hospital
Qualitative Method Comparison
Report Interpretation Guide
Chart Interpretation
The Bubble Chart is the equivalent of a scatter plot for
non-quantitative data. Green circles on the central diagonal
represent points of agreement. The size (area) of the circle
is proportional to number of specimens. The ideal chart has
only green circles.
Yellow and red circles represent points of disagreement.
 For a qualitative comparison, red indicates false
positives, and yellow indicates false negatives.
 For a semiquantitative comparison, yellow circles
represent cases where the reference and test are in
adjacent levels. Red circles are cases where the test
and reference are two or more levels apart.
References
1. CLSI Document EP12-A. User protocol for evaluation of
qualitative test performance; Approved guideline. CLSI, 940
West Valley Road, Suite 1400, Wayne, PA 19087-1898
USA, 2002. (References to this document will be to
CLSI:EP12)
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:05:21
Copyright 1991-2015 Data Innovations LLC
Page 4
29
ERI Example
EE11.2.23 -- My Community Hospital
Reference Interval Summary
Central 95% Interval
Meth
N
ALT
NP
240
8
Gender=Male
Calcium
Gender=Female
Gender=Male
NP
NP
NP
NP
120
240
120
120
10
9.1
8.9
9.2
Gender=Female
NP
120
Value
6
Lower Limit
95% CI
Value
Upper Limit
95% CI
6 to 9
54
48 to 65
<9 to 12
8.9 to 9.2
<8.8 to 9.2
<9.1 to 9.3
55
10.3
10.2
10.3
51 to >69
10.2 to 10.4
10.0 to >10.3
10.3 to >10.6
<5 to 8
46
30 to >65
Confidence
Ratio
0.22
>0.48 x
>0.23
0.21
>0.27
>0.23
NP: Nonparametric P: Parametric TP: Transformed Parametric X: Confidence Ratio > 0.30
Confidence Limits for Nonparametric CLSI C-28A method computed by exact formula.
Filter: None
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Copyright 1991-2015 Data Innovations LLC
Page 1
30
ALT
ERI Example
EE11.2.23 -- My Community Hospital
Reference Interval Estimation: Combined
Central 95% Interval
(N = 240)
Value
Nonparametric (CLSI C28-A)
Alternatives
Transformed Parametric
Parametric
Lower
95% CI
Value
Upper
95% CI
Confidence
Ratio
8
6 to 9
54
48 to 65
0.22
8
-1
7 to 8
-3 to 2
52
46
47 to 58
43 to 48
0.14
0.11
Confidence Limits for Nonparametric CLSI C-28A method computed by exact formula.
Histogram
Probability Plot
(Original Data)
Selection Criteria
Bounds
Filter
None
None
Statistics
Mean
SD
Median
Range
N
Distinct Values
Zeroes
Central 95% Index
22.5 U/L
11.9
19.5
5 to 69
240 of 240
50
0
6.0 to 235.0
Analyst
Expt Date
mkf
13 Apr 2015
Probability Plot
(Transformed Data)
Normalizing Transformation
Exponent
Constant
0.00 (Log)
0.00
Accepted by:
Signature
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:29
Date
Copyright 1991-2015 Data Innovations LLC
Page 2
31
ALT
ERI Example
EE11.2.23 -- My Community Hospital
Reference Interval Estimation: Female
Central 95% Interval
(N = 120)
Value
Lower
95% CI
Value
Upper
95% CI
Confidence
Ratio
Nonparametric (CLSI C28-A)
6
<5 to 8
46
30 to >65
>0.48
Alternatives
Transformed Parametric
Parametric
7
1
6 to 8
-2 to 3
38
35
33 to 43
32 to 38
0.19
0.16
Confidence Limits for Nonparametric CLSI C-28A method computed by exact formula.
Histogram
Probability Plot
(Original Data)
Selection Criteria
Bounds
Filter
None
None
Statistics
Mean
SD
Median
Range
N
Distinct Values
Zeroes
Central 95% Index
17.7 U/L
8.8
16.0
5 to 65
120 of 120
30
0
3.0 to 118.0
Analyst
Expt Date
mkf
13 Apr 2015
Probability Plot
(Transformed Data)
Normalizing Transformation
Exponent
Constant
0.00 (Log)
0.00
Accepted by:
Signature
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Date
Copyright 1991-2015 Data Innovations LLC
Page 3
32
ALT
ERI Example
EE11.2.23 -- My Community Hospital
Reference Interval Estimation: Male
Central 95% Interval
(N = 120)
Value
Lower
95% CI
Value
Upper
95% CI
Confidence
Ratio
Nonparametric (CLSI C28-A)
10
<9 to 12
55
51 to >69
>0.23
Alternatives
Transformed Parametric
Parametric
10
2
9 to 12
-1 to 6
60
52
52 to 69
48 to 56
0.20
0.16
Confidence Limits for Nonparametric CLSI C-28A method computed by exact formula.
Histogram
Probability Plot
(Original Data)
Selection Criteria
Bounds
Filter
None
None
Statistics
Mean
SD
Median
Range
N
Distinct Values
Zeroes
Central 95% Index
27.3 U/L
12.6
25.0
9 to 69
120 of 120
44
0
3.0 to 118.0
Analyst
Expt Date
mkf
13 Apr 2015
Probability Plot
(Transformed Data)
Normalizing Transformation
Exponent
Constant
0.00 (Log)
0.00
Accepted by:
Signature
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Date
Copyright 1991-2015 Data Innovations LLC
Page 4
33
ALT
ERI Example
EE11.2.23 -- My Community Hospital
Results Listing
Spec ID
S00001
S00002
S00003
S00004
S00005
S00006
S00007
S00008
S00009
S00010
S00011
S00012
S00121
S00013
S00014
S00122
S00123
S00015
S00016
S00017
S00018
S00019
S00020
S00021
S00124
S00125
S00126
S00127
S00022
S00023
S00024
S00025
S00026
S00027
S00028
S00029
S00030
S00031
S00032
S00128
S00129
S00033
S00034
S00035
S00036
S00037
S00038
S00039
S00040
S00041
S00042
S00130
S00131
S00132
S00043
S00044
S00045
S00046
S00047
S00048
Gender
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Male
Female
Female
Male
Male
Female
Female
Female
Female
Female
Female
Female
Male
Male
Male
Male
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Male
Male
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Male
Male
Male
Female
Female
Female
Female
Female
Female
ALT
5
6
6
6
7
8
8
8
8
8
9
9
9
10
10
10
10
11
11
11
11
11
11
11
11
11
11
11
12
12
12
12
12
12
12
12
12
12
12
12
12
13
13
13
13
13
13
13
13
13
13
13
13
13
14
14
14
14
14
14
Spec ID
Gender
ALT
Spec ID
S00049
Female
14
S00085
S00133
Male
14
S00086
S00134
Male
14
S00087
S00135
Male
14
S00088
S00136
Male
14
S00089
S00137
Male
14
S00157
S00138
Male
14
S00158
S00050
Female
15
S00159
S00051
Female
15
S00160
S00052
Female
15
S00161
S00053
Female
15
S00162
S00054
Female
15
S00163
S00055
Female
15
S00164
S00056
Female
15
S00165
S00139
Male
15
S00166
S00140
Male
15
S00090
S00141
Male
15
S00091
S00057
Female
16
S00092
S00058
Female
16
S00093
S00059
Female
16
S00094
S00060
Female
16
S00095
S00061
Female
16
S00167
S00062
Female
16
S00168
S00063
Female
16
S00169
S00142
Male
16
S00170
S00143
Male
16
S00171
S00144
Male
16
S00096
S00145
Male
16
S00097
S00064
Female
17
S00098
S00065
Female
17
S00099
S00066
Female
17
S00172
S00067
Female
17
S00173
S00068
Female
17
S00174
S00069
Female
17
S00175
S00070
Female
17
S00100
S00071
Female
17
S00101
S00146
Male
17
S00102
S00072
Female
18
S00103
S00073
Female
18
S00176
S00074
Female
18
S00177
S00075
Female
18
S00178
S00076
Female
18
S00179
S00077
Female
18
S00104
S00147
Male
18
S00105
S00148
Male
18
S00106
S00149
Male
18
S00180
S00150
Male
18
S00181
S00078
Female
19
S00182
S00079
Female
19
S00183
S00080
Female
19
S00184
S00081
Female
19
S00185
S00082
Female
19
S00186
S00083
Female
19
S00187
S00084
Female
19
S00107
S00151
Male
19
S00108
S00152
Male
19
S00188
S00153
Male
19
S00189
S00154
Male
19
S00190
S00155
Male
19
S00191
S00156
Male
19
S00109
Values marked with an "X" were excluded from the calculations.
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Gender
Female
Female
Female
Female
Female
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Female
Female
Female
Female
Female
Female
Male
Male
Male
Male
Male
Female
Female
Female
Female
Male
Male
Male
Male
Female
Female
Female
Female
Male
Male
Male
Male
Female
Female
Female
Male
Male
Male
Male
Male
Male
Male
Male
Female
Female
Male
Male
Male
Male
Female
ALT
20
20
20
20
20
20
20
20
20
20
20
20
20
20
20
21
21
21
21
21
21
21
21
21
21
21
22
22
22
22
22
22
22
22
23
23
23
23
23
24
24
24
25
25
25
25
25
25
25
25
25
25
25
26
26
26
26
26
27
28
Copyright 1991-2015 Data Innovations LLC
Page 5
34
ALT
ERI Example
EE11.2.23 -- My Community Hospital
Results Listing
Spec ID
S00110
S00192
S00193
S00194
S00195
S00111
S00196
S00112
S00113
S00197
S00198
S00199
S00200
S00201
S00202
S00203
S00204
S00205
S00206
S00207
Gender
Female
Male
Male
Male
Male
Female
Male
Female
Female
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
ALT
28
28
28
28
28
29
29
30
30
30
30
30
31
31
31
31
31
32
33
34
Spec ID
Gender
ALT
Spec ID
S00208
Male
34
S00224
S00209
Male
35
S00225
S00210
Male
35
S00226
S00114
Female
36
S00227
S00211
Male
36
S00118
S00212
Male
36
S00119
S00213
Male
36
S00228
S00214
Male
36
S00229
S00215
Male
36
S00230
S00115
Female
37
S00231
S00116
Female
37
S00232
S00216
Male
37
S00233
S00217
Male
38
S00234
S00218
Male
38
S00235
S00117
Female
39
S00236
S00219
Male
39
S00237
S00220
Male
39
S00238
S00221
Male
40
S00239
S00222
Male
40
S00120
S00223
Male
40
S00240
Values marked with an "X" were excluded from the calculations.
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Gender
Male
Male
Male
Male
Female
Female
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Female
Male
ALT
41
42
45
45
46
47
47
48
48
49
51
51
51
53
54
55
55
62
65
69
Copyright 1991-2015 Data Innovations LLC
Page 6
35
Calcium
ERI Example
EE11.2.23 -- My Community Hospital
Reference Interval Estimation: Combined
Central 95% Interval
(N = 240)
Value
Lower
95% CI
Value
Upper
95% CI
Confidence
Ratio
Nonparametric (CLSI C28-A)
9.1
8.9 to 9.2
10.3
10.2 to 10.4
0.21
Alternatives
Parametric
Transformed Parametric
9.1
--
9.0 to 9.1
--
10.3
--
10.2 to 10.4
--
0.11
--
Confidence Limits for Nonparametric CLSI C-28A method computed by exact formula.
Histogram
Probability Plot
(Original Data)
Selection Criteria
Bounds
Filter
None
None
Statistics
Mean
SD
Median
Range
N
Distinct Values
Zeroes
Central 95% Index
9.68 mg/dL
0.32
9.70
8.8 to 10.6
240 of 240
18
0
6.0 to 235.0
Analyst
Expt Date
mkf
13 Apr 2015
Probability Plot
(Transformed Data)
Normalizing Transformation
Exponent
Constant
---
Accepted by:
Signature
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Date
Copyright 1991-2015 Data Innovations LLC
Page 7
36
Calcium
ERI Example
EE11.2.23 -- My Community Hospital
Reference Interval Estimation: Female
Central 95% Interval
(N = 120)
Value
Lower
95% CI
Value
Upper
95% CI
Confidence
Ratio
Nonparametric (CLSI C28-A)
8.9
<8.8 to 9.2
10.2
10.0 to >10.3
>0.27
Alternatives
Parametric
Transformed Parametric
9.0
--
8.9 to 9.1
--
10.1
--
10.1 to 10.2
--
0.16
--
Confidence Limits for Nonparametric CLSI C-28A method computed by exact formula.
Histogram
Probability Plot
(Original Data)
Selection Criteria
Bounds
Filter
None
None
Statistics
Mean
SD
Median
Range
N
Distinct Values
Zeroes
Central 95% Index
9.57 mg/dL
0.29
9.60
8.8 to 10.3
120 of 120
16
0
3.0 to 118.0
Analyst
Expt Date
mkf
13 Apr 2015
Probability Plot
(Transformed Data)
Normalizing Transformation
Exponent
Constant
---
Accepted by:
Signature
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Date
Copyright 1991-2015 Data Innovations LLC
Page 8
37
Calcium
ERI Example
EE11.2.23 -- My Community Hospital
Reference Interval Estimation: Male
Central 95% Interval
(N = 120)
Value
Lower
95% CI
Value
Upper
95% CI
Confidence
Ratio
Nonparametric (CLSI C28-A)
9.2
<9.1 to 9.3
10.3
10.3 to >10.6
>0.23
Alternatives
Parametric
Transformed Parametric
9.2
--
9.1 to 9.3
--
10.4
--
10.3 to 10.5
--
0.16
--
Confidence Limits for Nonparametric CLSI C-28A method computed by exact formula.
Histogram
Probability Plot
(Original Data)
Selection Criteria
Bounds
Filter
None
None
Statistics
Mean
SD
Median
Range
N
Distinct Values
Zeroes
Central 95% Index
9.80 mg/dL
0.31
9.80
9.1 to 10.6
120 of 120
15
0
3.0 to 118.0
Analyst
Expt Date
mkf
13 Apr 2015
Probability Plot
(Transformed Data)
Normalizing Transformation
Exponent
Constant
---
Accepted by:
Signature
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Date
Copyright 1991-2015 Data Innovations LLC
Page 9
38
Calcium
ERI Example
EE11.2.23 -- My Community Hospital
Results Listing
Spec ID
S00001
S00002
S00003
S00004
S00005
S00006
S00007
S00121
S00122
S00008
S00009
S00010
S00011
S00012
S00013
S00014
S00015
S00016
S00017
S00018
S00123
S00019
S00020
S00021
S00022
S00023
S00024
S00025
S00026
S00027
S00028
S00029
S00124
S00125
S00126
S00127
S00128
S00129
S00130
S00131
S00030
S00031
S00032
S00033
S00034
S00035
S00036
S00037
S00132
S00133
S00134
S00135
S00136
S00137
S00038
S00039
S00040
S00041
S00042
S00043
Gender
Female
Female
Female
Female
Female
Female
Female
Male
Male
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Male
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Male
Male
Male
Male
Male
Male
Male
Male
Female
Female
Female
Female
Female
Female
Female
Female
Male
Male
Male
Male
Male
Male
Female
Female
Female
Female
Female
Female
Calcium
Spec ID
Gender Calcium
Spec ID
8.8
S00044
Female
9.5
S00081
8.9
S00045
Female
9.5
S00082
8.9
S00046
Female
9.5
S00083
9
S00047
Female
9.5
S00084
9.1
S00048
Female
9.5
S00085
9.1
S00049
Female
9.5
S00086
9.1
S00050
Female
9.5
S00087
9.1
S00051
Female
9.5
S00088
9.1
S00052
Female
9.5
S00089
9.2
S00053
Female
9.5
S00090
9.2
S00138
Male
9.5
S00091
9.2
S00139
Male
9.5
S00092
9.2
S00140
Male
9.5
S00093
9.2
S00141
Male
9.5
S00094
9.2
S00142
Male
9.5
S00095
9.2
S00143
Male
9.5
S00161
9.2
S00144
Male
9.5
S00162
9.2
S00145
Male
9.5
S00163
9.2
S00146
Male
9.5
S00164
9.2
S00147
Male
9.5
S00165
9.2
S00148
Male
9.5
S00166
9.3
S00054
Female
9.6
S00167
9.3
S00055
Female
9.6
S00168
9.3
S00056
Female
9.6
S00169
9.3
S00057
Female
9.6
S00170
9.3
S00058
Female
9.6
S00171
9.3
S00059
Female
9.6
S00172
9.3
S00060
Female
9.6
S00173
9.3
S00061
Female
9.6
S00096
9.3
S00062
Female
9.6
S00097
9.3
S00063
Female
9.6
S00098
9.3
S00064
Female
9.6
S00099
9.3
S00065
Female
9.6
S00100
9.3
S00066
Female
9.6
S00101
9.3
S00067
Female
9.6
S00102
9.3
S00068
Female
9.6
S00103
9.3
S00069
Female
9.6
S00174
9.3
S00149
Male
9.6
S00175
9.3
S00150
Male
9.6
S00176
9.3
S00151
Male
9.6
S00177
9.4
S00152
Male
9.6
S00178
9.4
S00153
Male
9.6
S00179
9.4
S00154
Male
9.6
S00180
9.4
S00155
Male
9.6
S00181
9.4
S00156
Male
9.6
S00182
9.4
S00157
Male
9.6
S00183
9.4
S00158
Male
9.6
S00184
9.4
S00159
Male
9.6
S00185
9.4
S00160
Male
9.6
S00186
9.4
S00070
Female
9.7
S00187
9.4
S00071
Female
9.7
S00188
9.4
S00072
Female
9.7
S00189
9.4
S00073
Female
9.7
S00104
9.4
S00074
Female
9.7
S00105
9.5
S00075
Female
9.7
S00106
9.5
S00076
Female
9.7
S00107
9.5
S00077
Female
9.7
S00108
9.5
S00078
Female
9.7
S00109
9.5
S00079
Female
9.7
S00110
9.5
S00080
Female
9.7
S00190
Values marked with an "X" were excluded from the calculations.
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Gender
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Female
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Female
Female
Female
Female
Female
Female
Female
Female
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Female
Female
Female
Female
Female
Female
Female
Male
Calcium
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.7
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.8
9.9
9.9
9.9
9.9
9.9
9.9
9.9
9.9
Copyright 1991-2015 Data Innovations LLC
Page 10
39
Calcium
ERI Example
EE11.2.23 -- My Community Hospital
Results Listing
Spec ID
S00191
S00192
S00193
S00194
S00195
S00196
S00197
S00198
S00199
S00200
S00201
S00202
S00203
S00111
S00112
S00113
S00204
S00205
S00206
S00207
Gender
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Male
Female
Female
Female
Male
Male
Male
Male
Calcium
9.9
9.9
9.9
9.9
9.9
9.9
9.9
9.9
9.9
9.9
9.9
9.9
9.9
10
10
10
10
10
10
10
Spec ID
Gender Calcium
Spec ID
S00208
Male
10
S00223
S00209
Male
10
S00224
S00210
Male
10
S00225
S00114
Female
10.1
S00226
S00115
Female
10.1
S00227
S00211
Male
10.1
S00228
S00212
Male
10.1
S00229
S00213
Male
10.1
S00230
S00214
Male
10.1
S00231
S00215
Male
10.1
S00119
S00216
Male
10.1
S00120
S00217
Male
10.1
S00232
S00218
Male
10.1
S00233
S00219
Male
10.1
S00234
S00220
Male
10.1
S00235
S00116
Female
10.2
S00236
S00117
Female
10.2
S00237
S00118
Female
10.2
S00238
S00221
Male
10.2
S00239
S00222
Male
10.2
S00240
Values marked with an "X" were excluded from the calculations.
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:08:30
Gender
Male
Male
Male
Male
Male
Male
Male
Male
Male
Female
Female
Male
Male
Male
Male
Male
Male
Male
Male
Male
Calcium
10.2
10.2
10.2
10.2
10.2
10.2
10.2
10.2
10.2
10.3
10.3
10.3
10.3
10.3
10.3
10.3
10.3
10.3
10.4
10.6
Copyright 1991-2015 Data Innovations LLC
Page 11
40
EE11.2.23 -- My Community Hospital
Establish Reference Interval
Report Interpretation Guide
Establishing a Reference Interval
A Reference Interval (RI) in this case refers to the central
95% range -- or "normal" range -- for endogenous analytes
of a healthy patient. RIs are established by assaying a large
number of specimens from healthy people and using the
resulting sampleto estimate the central 95% range of the
population. "Normal Range" is a pun. The statistical meaning
is that it represents (often inaccurately) a Gaussian
distribution of results. The clinical meaning is that it is
derived from a healthy population.
While estimating the central 95% may seem like a simple
task, there are some complicating factors:





It is inappropriate to establish a reference interval for an
exogenous analyte (i.e., a drug) using this program. This
warning also applies to cutoff values for endogenous
analytes such as critical levels of CKMB. In these cases,
use the ROC Curve approach.
The population used to establish the RI must be the
same population to which the RI applies. Determine a
PSA on a male population, not female lab employees.
Similarly, determine first trimester bHGC's on women in
that stage, not on non-pregnant women.
RIs can differ dramatically with age, gender, and
lifestyle. Hemoglobin concentrations are quite different
for a pediatric population than for an adult population.
Cholesterol concentrations are much lower for a
Japanese population than for an American population.
The number of specimens is important. CLSI:C28
recommends using at least 120 specimens. This is the
smallest sample size to obtain both an RI estimate and
a 90% confidence interval for it without making an
assumption about the population distribution. All other
things being equal, a larger sample size gives a more
accurate RI and a tighter confidence interval.
The quality of the RI cannot be judged solely by the
width of the confidence interval. A parametric RI
estimate will almost always have a narrower confidence
interval than a non-parametric RI estimate. However, if
the population distribution is materially non-Gaussian,
the narrow parametric confidence interval is computed
from a false assumption. See the discussion of
parametric vs. non-parametric, below.
Definition of Statistical Terms
Reference Interval. The range of values that includes the
central 95% of the population. The two endpoints of the
Reference Interval are called the reference limits.
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:08:30
Occasionally a reference interval is also calculated for some
other percent, like 90% or 80%. This may be appropriate if
itis not possible to obtain enough reference subjects to
calculate a 95% interval.
Confidence Interval (CI). A measure of the precision of the
Reference Interval estimate. For example, suppose you
estimate the upper reference limit for Haptoglobin at 193
mg/dL, with a 90% confidence interval of 175-210. If you
repeat the calculation on many different samples, you would
expect 90% of the estimates to be between 175 and 210.
Confidence Ratio The ratio of the average confidence
interval width to the reference interval width:
0.5*(URLU-URLL+LRLU-LRLL)/(URL-LRL).
A value of 0.10 or less is desirable. Values over 0.30 are
flagged in the report. The primary determinant of the
confidence ratio is sample size -- confidence ratio improves
(gets smaller) as sample size increases.
Distribution. A statistical concept describing what percent of
the population values fall into various ranges. A familiar
example is the bell-shaped Gaussian Distribution
curve, where 95% of the values lie within +/- 2 SD of the
mean.
Nonparametric Method. A method of calculating a
Reference Interval that makes no assumption about the
shape of the population distribution. The simplest
nonparametric method is that recommended by CLSI:C28 -obtain 120 samples and list them in ascending order. The
lower reference limit is the 3rd sample, and the upper
reference limit is the 118th sample. W ith as few as 40 results
it is still possible to estimate a Reference Interval using
selected sample ranks, but we refer to it as the
Nonparametric Index method instead of the CLSI method.
(CLSI:C28 recommends no fewer than 120 results.)
Parametric Method. A method of calculating a Reference
Interval which assumes the population either follows a
Gaussian distribution or can be converted to a Gaussian
distribution -- perhaps by taking a log or square root. The
simplest parametric method is to calculate the reference
limits as Mean +/- 2 SD. The Transformed Parametric
Method first attempts to change the scale of the data so it
has a Gaussian distribution. Then it computes mean +/- 2
SD, and converts the answer back to the original units.
A parametric method based on a false assumption may be
unreliable. For example, when the distribution is not
Gaussian the parametric method may give a negative value
for the lower reference limit. Not only is the estimate
unreliable, the 90% confidence interval is also overly
Copyright 1991-2015 Data Innovations LLC
Page 12
41
EE11.2.23 -- My Community Hospital
Establish Reference Interval
Report Interpretation Guide
optimistic. It is based on the same false assumption as the
reference interval itself. Don't use the Reference Interval
from mean +/- 2 SD unless it is very clear that the curve
really is Gaussian. If Mean - 2SD is negative, then the curve
is not Gaussian.
Histogram. A bar chart that illustrates the distribution of
results. Each rectangle represents an equal-width range of
values, with height proportional to the number of results in
the range. A Gaussian distribution curve is plotted in the
background.
Probability Plot. A chart that helps you judge whether your
data is Gaussian, as well as assessing how reasonable the
Reference Interval estimate is. Plotted points represent
specimens from your sample. The Y axis is result value, and
the X axis is SD's of a hypothetical Gaussian distribution.
 If your data is Gaussian, the points lie in a straight line.
 The nonparametric Reference Interval estimate is the Y
value where the scatter of dots crosses the +/- 2 SD X
values.
 If you draw a straight line that passes through the
sample mean, and has the slope of the sample SD, it
intersects +/- 2 SD to give the parametric Reference
Interval estimate.
 If the points fall very close to this true Gaussian line, it is
likely that the parametric and nonparametric Reference
Interval calculations will agree.
60 samples from each subclass, and estimate a
separate Reference Interval for each group.
References
1. Harris EK, Boyd JC. Statistical bases of reference values
in laboratory medicine. New York: Marcel Dekker, 1995:1-61.
2. CLSI Document C28-A2. How to define and determine
reference intervals in the clinical laboratory; Approved
guideline-second edition. CLSI, 940 W est Valley Road, Suite
1400, W ayne, PA 19087-1898 USA, 2000. (References to
this document will be to CLSI:C28)
Normalizing Transformation (Box-Cox Transformation).
With the aid of a moderately complex algorithm, the software
attempts to transform the data into a Gaussian model so
simple Gaussian statistics can be calculated. This
normalizing transformation produces an exponent and a
constant. It really does not matter to you what these two
numbers are. The only thing that really matters is the degree
to which the data is linearly distributed in the transformed
probability plot.
Partitioning Analysis. An analysis to determine whether
separate reference intervals are statistically justified for
population subclasses. Example: Should separate reference
intervals be estimated for men and women, or should there
be a single combined reference interval for both? The
CLSI:C28 guideline recommends the following procedure:
 Collect a preliminary sample consisting of at least 60
reference subjects from each subclass.
 Perform a statistical test to determine whether these
samples are significantly different from each other.
 If there is evidence of a difference, collect an additional
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:08:30
Copyright 1991-2015 Data Innovations LLC
Page 13
42
Sodium
Method Eximir 100
Level Low-EQC
EE11.2.23 -- My Community Hospital
Trueness - EQC
%Bias vs Time
At a value of 122.4 mmol/L, uncertainty is
estimated to be 3 (absolute) or 1.7% (relative).
Calculated using maximum of bias.
Sigma: 3.9
The experiment Passes.
Supporting Data
CRL
08 Jan 2014
mmol/L
ABC B677744663 exp 12/31/2017
ABC cal6940333 exp 12/31/2020
ABC MQC7764583 exp 12/31/2018
8 (Calculated using maximum of bias)
Analyst
Expt Date
Units
Reagent
Calibrator
Control
SD Rel Thresh
Comment
Sigma
3.9
User Specifications
Statistical Summary
Mean
Bias
122.7
122.6
--
-0.2
--
Lab
Peer Group
All Methods
%Bias Max %Bias
-0.1
--
-0.3
--
N: 3
Group Mode
AG Mode
Budget Conc
Budget Pct
Sys Error Budget
Peer Group
Budget
4
IQC Stats
Mean
SD
CV
122.4
0.995
0.8
50
Trueness Data
Lab
Result
SD
Result
Jan
Feb
Mar
122.2
122.8
123.2
0.962
1.03
0.996
122.4
122.5
122.8
----
-0.2
0.2
0.3
Means
122.7
1.0
122.6
--
0.1
Period
Peer Group
SD %Bias
Pass
Yes
Yes
Yes
%Bias
Cutoff Uncert
1.6
1.6
1.6
----
X: excluded from calculations
Accepted by:
Signature
Date
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:51:14
Page 1
43
EE11.2.23 -- My Community Hospital
Trueness
Report Interpretation Guide
The purpose of the EE Trueness Module is to provide a
mechanism for clinical laboratories to assess the
characteristics of Trueness, Accuracy, and Uncertainty for
the methods used in their settings. These concepts are
described in laboratory regulations or documents from
standardization or certification organizations as noted in the
Reference list below. This module is intended to satisfy the
Trueness, Accuracy and Uncertainty requirements for
COFRAC certification in France. In EE11.1, it is also
possible to determine the Sigma Metric as an expression of
laboratory performance.
The source of the data for this module is monthly IQC
summary data, along with EQC group data or EQA group
data, as defined below.
Definitions and Terms:
Measurand : In the Trueness module, the term "measurand"
is used instead of "analyte". The technical difference
between the two terms is described in ISO 17511. The term
analyte refers to a measurement of a given analytical
component regardless of its environment. Measurand refers
to an analytical component in a specific environment. For
example, the analyte glucose in serum glucose, whole blood
glucose and urine glucose corresponds to three different
measurands.
Method : In this module the term "method" is used rather
than Instrument. The method describes the specific assay
formulation or test for the measurand as applied to a specific
platform. It includes the elements of sample type(s),
reagents, software, and hardware. It could also denote a
manually performed method.
Trueness : Trueness is defined in ISO 3534-1 as "the
closeness of agreement between the average value obtained
from a large series of test results and an accepted reference
value." The input data comes from an External Quality
Control (EQC) Program, and the participating laboratory's
values are compared to the selected group summary mean
for the same time period.
Accuracy : ISO/IEC Guide 99 defines Accuracy as the
closeness of agreement between a measured quantity value
and a true quantity value of a measurand. The key phrase
here is "a measured quantity value". This refers to a single
measurement, not the mean of many measurements. The
input data may be drawn from External Quality Assurance
(EQA) Programs like CAP or EQAS Proficiency surveys, and
the laboratory's single result is compared to the selected
group mean for the same specimen.
Analytical Goal (AG) and %Bias cutoff : Analytical Goals
for Trueness and Accuracy may be referred to by several
Synonyms, such as AG, TEa, allowable total error, allowable
error, systematic analytical goal, desirable analytical goal, or
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:51:14
"%Bias cutoff". Analytical goals can be shown in many forms
and can be derived from several sources as defined below.
IQC : Internal Quality control result(s) for the laboratory daily
QC. The IQC mean, SD, and CV% are input into the module
parameter screen. Comparisons are improved if the lab IQC
mean is near the level chosen for the experiment.
EQC : External Quality Control. Laboratories which
participate in EQC programs use the specified lot numbers
of control materials for their daily QC, and submit the
summary results for the defined period (often monthly) to the
provider for analysis. The group mean from multiple labs is
the reference quantity, assumed to be the "true" result. The
bias assessed in this manner is a measure of Trueness
EQA : External Quality Assurance. Also known as
proficiency testing, EQA Programs like CAP or EQAS
provide a set of specimens to be analyzed by a laboratory as
though they were patient specimens. The laboratory's single
result for a measurand is submitted to the provider for
accuracy assessment versus the defined target evaluation
criteria. Proficiency testing for many measurands is required
by CLIA and other regulatory bodies. In the Trueness
module, the laboratory's single result is compared to the
selected group mean for the same specimen. The bias
assessed in this manner is a measure of Accuracy.
Sigma: The sigma metric assesses the laboratory’s
analytical bias and imprecision compared to the total
allowable error Analytical Goal. A Sigma metric of “6” or
greater means that less than 3.4 results per million is outside
the TEa limit.
Trueness Module Calculation modes:
The Trueness module uses two calculation modes,
depending on the source of the data being used. The EQA
mode uses EQA data for an Accuracy experiment. The EQC
mode uses EQC data for a Trueness experiment. The choice
to use EQC data or EQA data is made when each
experiment is created.
Selected Group : The Trueness module parameters screen
allows the user to choose whether to use the Peer Group or
"All Method" Group of the source data in the ensuing
calculations. The Peer group or "All Method" Group will be
referred to as the "Selected" Group in the rest of this
discussion.
EQA Accuracy Report : The calculated bias for each EQA
specimen is the difference between the lab value and the
selected group mean. The calculated %bias is compared to
the selected Analytical Goal (AG) %bias cutoff to determine
Pass / Fail for each specimen, as well as for the overall
experiment. In the EQA mode the user is free to choose the
lab IQC level. The estimate of bias or uncertainty may be
Copyright 1991-2015 Data Innovations LLC
Page 2
44
EE11.2.23 -- My Community Hospital
Trueness
Report Interpretation Guide
improved if the lab IQC value selected is near a key medical
decision level contained in the EQA data. W hile this mode
allows specimen data from multiple levels to be used in a
single experiment, estimates may also be improved if
separate experiments are created for different levels. The
experiment evaluation is expressed as Pass / Fail versus the
selected Analytical Goal (AG).
EQC Trueness Report : In this calculation mode, the lab
value is the lab's Monthly Summary Mean for the level of
EQC material being evaluated. The input IQC values are the
lab's Overall Mean and SD. The difference between the lab
value and the selected group mean for the corresponding
time period is the calculated bias for each EQC specimen.
The calculated %bias is compared to the AG %bias cutoff to
determine Pass / Fail for each specimen as well as for the
overall experiment. The experiment evaluation is expressed
as Pass / Fail versus the selected Analytical Goal. The EQC
calculation mode also permits evaluation of the Sigma Metric
using EQC data if the lab SD for each time period is
included.






SD Reliability Threshold (SDRT)
The allowed number of specimens (N) for EQC experiments
can range from 3 to 12 active specimens, while EQA
experiments can have up to 50 specimens. W hen N is small,
the uncertainty calculation using the mean of the biases may
be unreliable. The Standard Deviation Reliability Threshold
(SDRT) determines the statistical approach that is used for N
above, at, and below the SDRT. The SDRT is user-defined
in the "File\Preferences" menu option as an integer between
4 and 12 (inclusive), but it can be edited in the Parameters
screen for each experiment if desired.
When (N) >= SDRT, the absolute value of the mean of the
biases is used in the uncertainty calculation.
When N < SDRT, the maximum of the absolute value of the
biases is used. SDRT is displayed in the Supporting Data
section of the report.
Analytical Goals (AG) and Specifications
The Trueness module's parameters screen allows the user
to choose from four models to define AG to assess Pass/Fail
for Trueness or Accuracy:
 Budget model : This is the traditional EE TEa model. A
concentration and/or a percent are specified for TEa,
and a budget % of the total is specified for systematic
error. The remainder is assumed to be the budget for
random error. W hen both a concentration component
and a percent are specified, there is a crossover point
where the percent of that level is equivalent to the
concentration component.
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:51:14




Component specification model : Allowable bias is
specified as either "bias conc" or "bias pct" or both, and
often is available from the IVD manufacturer. W hen both
a concentration component and a percent are specified,
there is a crossover point where the percent of that level
is equivalent to the concentration component
%Bias cutoff model : The %bias cutoff value is entered
as a single value by the user.
Biological variation model (BV) : EP Evaluator uses
the Ricos (ref 8) model to determine the AG goal as
follows. This model lets the user select a Factor to
evaluate their method compared to a specified
performance level as: minimal (3), desirable (2), and
optimal (1). Values for many measurands can be
obtained from the Tools Menu\ Biological Variation
Table and entered in the parameter screen as follows:
CVi : intra-individual biological variation
CVg : inter-individual biological variation
Factor (F) for level of performance desired : (minimal
(3), desirable (2), or optimal (1))
Z = 1.65 for a 95% Confidence level and 2.33 for a 99%
Confidence level. From these inputs, the AG goal is
calculated as follows:
CVa (allowable analytical precision) = F * 0.25 * CVi
Ba (allowable bias) = F * 0.125 * sqrt(CVi^2 + CVg^2)
TEa (Total Allowable Error) = (Z * CVa) + Ba )
Mode Specific Calculations









EQA mode : AG is calculated assuming the presence of
both systematic and random error components in the
reported results. Therefore 100% of the TEa is used as
the AG.
Budget model: AG = TEa
Specimen component model: AG = Bias component
specification + (1.65 * (IQC SD))
Biological Variation model: AG = TEa.
%Bias cutoff: AG = %Bias cutoff as entered by the user
EQC mode : AG is calculated assuming the presence of
systematic error only.
Budget model: AG = TEa * Systematic error budget
Specimen component model: AG = Bias component
specification
Biological Variation model: AG = Ba
Copyright 1991-2015 Data Innovations LLC
Page 3
45
EE11.2.23 -- My Community Hospital
Trueness
Report Interpretation Guide
%Bias cutoff: AG = %Bias cutoff as entered by the user
For Both modes : If a specimen mean for the selected
comparator group is zero, it will be impossible to compute
the Percent Bias for that specimen, which will make it
impossible to evaluate Pass/Fail for that specimen; in this
case, the specimen (and the experiment) will always Fail,
unless the specimen is excluded.
Uncertainty : ISO/IEC Guide 99 defines Uncertainty as a
"non-negative parameter characterizing the dispersion of the
quantity values being attributed to a measurand, based on
the information used".
The magnitude of Uncertainty depends on the contributions
of the variables involved in the scope of measurement that
impact either bias or imprecision, such as different days,
different runs, different operators, method interferences,
calibration error, etc. In the Trueness module, only the
precision and bias components are used to calculate the
Uncertainty statistic. Uncertainty is evaluated for both EQA
and EQC types of experiments.

Uncertainty Calculations
Uncertainty will be calculated for each specimen only if the
SD from the selected group is entered. The overall
Uncertainty at the IQC level is always calculated.
Relative uncertainty is also estimated at the crossover
point, if the AG mode is either the budget model or
component model, and if both a concentration and percent
are entered for Allowable Error. The following example is for
glucose with an IQC mean of 115, and a TEa of 6 mg/dL or
10%. The crossover value is 60:
"At a value of 115 mg/dL, uncertainty is estimated to be 10
(absolute) or 8.3% (relative). At values less than or equal to
60 mg/dL, uncertainty is estimated to be 5 mg/dL. Calculated
using mean of bias." The experiment Passes ."
Uncertainty Calculations follow the standard calculation as
described in SH-GTA-14. Uncertainty is computed for 1 SD,
and then multiplied by 2 to closely approximate the 95%
confidence limit.
Uncertainty (U2) for individual specimens:
 U2 = 2 * (sqrt(SD of selected group specimen)^2) +
(ABS(Bias of lab result to selected group)^2)
Overall Uncertainty :
 U2 = 2 * sqrt (A^2 + B^2 + C^2) where
 A = IQC SD
 B = mean SD of the group Bias across all
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:51:14



specimens/periods. W hen N <SDRT, B = 0.
C = D / sqrt(3).
If N>= SDRT, D = mean Bias of the group
If N < SDRT, D = max(abs value of the Bias across all
specimens/periods)
Sigma Calculation
The Sigma calculation is enabled by a checkbox in the
parameters screen option box, or if enabled in Trueness
policies\Modules and Options. Also, Sigma will only be
calculated when the calculation mode is EQC, and the TEa
budget mode is the selected Analytical Goal. The Lab SD is
required for every period of EQC data. A single Sigma metric
is calculated for the entire experiment, and uses mean
values for all periods of EQC data.
The Sigma metric = (AG goal TEa Percent minus the
absolute value of the observed mean bias percent) / the
laboratory mean CV.
The AG TEa percent is derived from the budget TEa in the
parameters screen. If the applicable TEa for the selected
group result is expressed as a concentration, the TEa
percent used in the sigma calculation is TEa conc *100 /
selected group mean. Sigma is calculated using 100% of the
TEa, and is not affected by the SEa Percent. The Sigma
metric does not have pass / fail criteria, however a Sigma
metric greater than 6 is usually considered excellent
performance. The larger the Sigma, the better the process:
 A Sigma metric of 6.0 or greater indicates a "Six Sigma"
process.
 Sigma metric of 4.0 to 6.0 - Good performance
 Sigma metric of 3.0 to 4.0 - Fair performance
 Sigma metric of 2.0 to 3.0 - Marginal performance
 Sigma metric less than 2.0 - Poor performance
Report Components:



Lab Mean is displayed on the report summary page
along with the number of points (n) used.
Uncertainty is calculated for each specimen only if the
SD for the selected group is entered. The overall
Uncertainty is always calculated. The report Summary
page column "Uncert ABS / %" shows the absolute
Uncertainty and % uncertainty at the IQC level.
%Bias cutoff displays the analytical goal (AG) as the
%Bias cutoff at the selected group value, using the AG
formulas defined above for the selected AG mode. For
Copyright 1991-2015 Data Innovations LLC
Page 4
46
EE11.2.23 -- My Community Hospital
Trueness
Report Interpretation Guide






the AG modes that define both a conc and PCT, the
%Bias cutoff will be variable below the crossover point.
Sigma Metric : W hen enabled, the Sigma metric is
shown on the Overview screen and report summary
page in the Pass / Sigma column instead of Pass or
Fail. The experiment detail screen statistics tab and the
report experiment page includes the Sigma metric as
well as the Trueness pass/fail assessment.
Graphs : Four graphs are possible, depending on the
choices made in parameters and whether EQC data or
EQA data is used.
Scatter plot : Shown for the EQA mode only, it shows
each lab result vs. selected group mean. A slope and
intercept are calculated using ordinary least squares
linear regression.
Uncertainty plot : Shown for the EQA mode only, and
then only if at least one selected group SD is present;
the uncertainty calculated for each lab value is displayed
vs. the corresponding group value. This is especially
useful if multiple EQA levels have been entered.
%Bias vs. Selected Group : Shown for the EQA mode
only, the observed %bias compared to the selected
group is plotted vs. the selected group mean for each
specimen. This is useful to visually compare the
different levels
%Bias vs. Time : Shown for both EQA and EQC
modes, the observed %Bias compared to the selected
group mean is shown in the order of the date of
observations. If both Peer Group and All Method means
are available, then both sets of comparisons are shown
for the EQC mode. The EQA mode graph shows only
the selected group. Because this is chronological, it is
useful for identifying bias trends over time.
Pass / Fail Criteria:
The experiment report will be marked PRELIMINARY if there
are fewer than 3 unexcluded specimens.
In the Trueness Data table, passing specimens are
presented in a black, unbolded font; failing specimens are in
red bold; and excluded specimens are in red italics.
The experiment passes if there are at least 3 unexcluded
specimens and all unexcluded specimens pass the bias test,
as defined below:
 The bias test for Trueness (EQC mode) : Each
specimen passes if the bias from the selected group
mean does not exceed the specified analytical goal (AG)
(computed %bias cutoff), Trueness is evaluated as
Pass/Fail and is not revealed as a numerical value. If
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:51:14

any specimen fails, then the experiment fails.
The bias test for Accuracy (EQA mode) : Each
specimen passes if the bias from the selected group
mean does not exceed the analytical goal (AG)
(computed %bias cutoff) Accuracy is only evaluated in a
Pass/Fail manner: it is not revealed as a numerical
value. If any specimen fails, then the experiment fails.
References:
1. Fraser, Biological Variation: from Principles to Practice,
AACC Press, 2001.
2. COFRAC, Cofrac, Comité Français d'Accréditation,
www.cofrac.fr
3. CLSI Document EP15-A2. User Verification of
Performance for Precision and Trueness; Approved
Guideline, second edition. CLSI, W est Valley Road, Suite
1400, W ayne, PA 19087-1898 USA, 2005.
4. CLSI C51A. Expression of Measurement Uncertainty in
Laboratory Medicine: Approved Guideline. CLSI, W est Valley
Road, Suite 1400, W ayne, PA 19087-1898 USA, 2012
5. Guide Technique D'accreditation Pour L'evaluation Des
Incertitudes De Mesure En Biologie Medicale SH- GTA-14
6. JCGM_100_2008_E.pdf Guide to the Expression of
Uncertainty in Measurement (www.bipm.org). Copyright of
this document is shared jointly by the JCGM member
organizations (BIPM, IEC, IFCC, ILAC, ISO, IUPAC, IUPAP
and OIML).
7. Document Cofrac SH GTA 06, "Les contrôles de qualité
en Biologie Médicale" (dans l'attente de sa parution, se
référer au document Cofrac LAB GTA 06).
8. ISO Guide 15189:2007. Medical Laboratories - Particular
Requirements for Quality and Competence.
9. Ricos C, Alvarez V, Cava F, Garcia-Lario JV, Hernandez
A, Jimenez CV, Minchinela J, Perich C, Simon M. "Current
databases on biologic variation: pros, cons and progress."
Scand J Clin Lab Invest 1999;59:491-500.
10. ISO/IEC Guide 99:2007. International vocabulary of
metrology - Basic and general concepts and associated
terms. International Organization for Standardization. 2007.
11. ISO Guide 3534-1:2006. Statistics - Vocabulary and
symbols.
12. ISO Guide 17511:2003. In vitro diagnostic medical
devices – Measurement of quantities in biological samples
- Metrological traceability of values assigned to calibrators
and control materials.
Copyright 1991-2015 Data Innovations LLC
Page 5
47
HCG-a
Instrument Analyzer
EE11.2.23 -- My Community Hospital
Carryover
Carryover Analysis
High-Low Mean
Low-Low Mean
Carryover
0.8
0.4
0.4
Error Limit
Passes?
5.0
Yes
Experimental Results
Sample
Result
L1
L2
L3
H1
H2
L4
H3
H4
L5
L6
L7
L8
H5
H6
L9
H7
H8
L10
H9
H10
L11
0
0
1
1205
1210
1
1201
1218
2
0
0
1
1224
1204
1
1221
1218
0
1209
1217
0
Mean
SD
Low-Low
Results
High-Low
Results
0
1
1
Supporting Data
Units
Analyst
Expt Date
Low Conc
High Conc
Error Limit Type
User Error Limit
Comment
MIU/ml
CRL
27 Apr 2015
1
1200
Conc
5
2
0
0
1
1
0
0
0.4
0.5
0.8
0.8
Accepted by:
Signature
EP Evaluator 11.2.0.23
working document website sample reports Printed: 23 Jun 2015 15:55:05
Date
Copyright 1991-2015 Data Innovations LLC
Page 1
48
EE11.2.23 -- My Community Hospital
Carryover
Report Interpretation Guide
The Carryover module determines specimen to
specimen carryover. Specimens with very high results
are followed by specimens with very low results. If the
mean of the High-Low results minus the mean of the
Low-Low results is less than or equal to the
experiment’s pass/fail error limit, then the experiment
will pass the carryover Test.
Experiment Design
Two specimens are prepared, one with a very high
concentration and one with a very low concentration.
These specimens are aliquoted into a total of 21
samples: 11 with low concentration and 10 with high
concentration.
While assaying these 21 samples, no other specimens
or tests should be assayed in the instrument, and the
samples must be assayed in the following order;
otherwise the experiment is invalid.
3 Low specimens
2 High specimens
1 Low specimen
2 High specimens
4 Low specimens
2 High specimens
1 Low specimen
2 High specimens
1 Low specimen
2 High specimens
1 Low specimen
Definitions
Low-Low Result: A Low result that immediately
follows another Low result.
Error Limit: the user can define the error limit in one
of three ways:

Automatically calculated (the default): Three times
the SD of the Low-Low results. In other words, the
SD that would be expected if no High results were
measured.

User entry of percent of the Low-Low mean.

User entry of analyte Concentration
Traditional carryover experiments (ref 1,2) have
suggested using a percentage of the low value as the
carryover cutoff. However, using a concentration limit
as the cutoff may be more appropriate when the low
value is close to zero.
The default criterion (ref 3.) calculates the error limit
as 3 times the low-low SD. It is worth noting that if the
Low-Low SD is zero, the experiment might fail. This is
because the automatically calculated limit is set to the
next higher significant digit based on the maximum
number of decimal points entered in the data. For
example, with data having two decimal places, the
error limit would be set to 0.01. Such a limit might be
appropriate for some analytes, but for other analytes,
might be unrealistically small.
The report lists the Low-Low and High-Low results in
separate columns to emphasize which specimens
enter into the key mean and SD calculations.
Pass or Fail?
The carryover test passes if Carryover is less than the
Error Limit.
High-Low Result: A Low result that immediately
follows a High result.
Preliminary Report
Carryover: The mean of the High-Low results minus
the mean of the Low-Low results.
The word PRELIMINARY printed diagonally across the
report indicates that the data is incomplete, and the
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:55:05
Copyright 1991-2015 Data Innovations LLC
Page 2
49
EE11.2.23 -- My Community Hospital
Carryover
Report Interpretation Guide
report is not acceptable as a final report. Some or all
of the statistics may be missing.
The Carryover report is preliminary if less than 21
results are entered.
References:
1. Teitz (ed). Textbook of Clinical Chemistry, page
421. WB Saunders Co, Philadelpha, 1986.
2. Broughton, PMG. Journal of Automatic Chemistry.
Vol 6. No 2. (April – June 1984) pp 94-95.
3. Stanley, L. Private Communication. 2001
EP Evaluator 11.2.0.23
Printed 23 Jun 2015 15:55:05
Copyright 1991-2015 Data Innovations LLC
Page 3
50