SYMPOSIUM
Performance Characteristics of
Quality Control Systems
James R. Hackney, MD
The purpose ofa quality control program
is to monitor analytical systems, detecting errors when they become significant.
Quality control rules and procedures can
be characterized by their performance
characteristics. These characteristics
quantitate thepowerofa particular procedure in terms ofits probability oferror
detection (PJ) and probability offalse rejection (PfJ. Quality control systems
should be designed so as to maximize P^
and minimize Py The performance characteristics ofmany ofthe more common
quality control rules have been determined and are available in the literature.
Computer simulation experiments are
valuable tools for delineating the characteristics ofdifferent quality control rules
and procedures, and provide a basis for
predicting how effective a given system
will be in detecting error.
very laboratory result contains a certain amount of analytical error. This
^ ^ error consists of a random (unpredictable) component, which can be quantified as the standard deviation of replicate control analyses, and a systematic
component (predictable), which can be
expressed as bias, or the difference between the mean of multiple control analyses and the expected or "true" value.
Random error is often referred to as imprecision, while systematic error (bias) is
referred to as inaccuracy. Even the best
methods exhibit systematic and random
error. When a new method or instrument
is introduced into the laboratory, it is the
purpose of the comparison of methods
Some Common Control Rules
Rule
From the Hematology Laboratory, Ochsner Foundation Hospital, New Orleans, LA 70121.
388
Laboratory Medicine June 1989
experiment to establish the inherent random error and bias. Thereafter, it is the
purpose of the quality control program to
detect added error, whether it is increased
random error or systematic error.
The quality control program must be
tolerant of the method's inherent analytical error but should detect errors that exceed the inherent error. Therefore, the
optimal quality control program would
be insensitive to small errors, but very sensitive to moderate and large errors. Ideally, the descriptors "small," "moderate,"
and "large" should be defined in terms
that are clinically relevant rather than statistically relevant.
Definition
Sensitivity*
One control value outside the mean ± 2s
RE and SE (Warning)
One control value outside the mean ± 3s
RE and SE
2 consecutive controls exceeding mean+2s or mean—2s
SE
4 consecutive controls exceeding mean + 1s or mean—1s
SE
"4S
Range between 2 consecutive controls exceeds 4s
RE
10,
10 consecutive control values on one side of the mean
SE (Warning)
*RE indicates random error; SE, systematic error.
The purpose of any quality control
program is to monitor the analytical process and to intervene whenever analytical
errors exceed certain limits. This implies
at least two things. First, monitoring must
take place. Running a stabilized control
material once a day on an instrument that
is used throughout the day is usually not
adequate. On the other hand, running
multiple controls in every analytical run is
generally excessive. A balance must be
achieved which provides for adequate
and efficient monitoring. Second, since
the intervention decision is based on the
violation of control limits, those limits
must be relevant to the analysis. Limits
that are too narrow are frustrating and
consume resources, while limits that are
too broad are overly tolerant of poor
quality. The control limits should be clinically relevant rather than statistically relevant, but should also take into account
the inherent capabilities of the analytical
system. The table contains a list of some
common control rules and the types of error to which they are sensitive. Some of
these rules, such as the 2^ rule, are more
sensitive to systematic error, while others
such as the RM rule are especially useful
for random error. Control rules can be
used together in a control procedure, also
referred to as a "multirule."1 When a control procedure is designed, care should be
taken to include at least one rule sensitive
to systematic error and one rule sensitive
to random error. In general, combinations of control rules enhance the probability of detecting an error without dramatically increasing the probability of
falsely rejecting a run. However, increasing the number of rules also increases the
complexity of interpretation by the analyst, a factor which is particularly important if the control procedure is applied
manually.
Designers of quality control procedures must specify the frequency of monitoring, the form of monitoring (use of patient data averages, retained patient specimens, and/or stabilized control material)
and the clinical or statistical limits upon
which the intervention decision is to be
based. To do this objectively, one must
determine the characteristics of a given
quality control procedure, including the
probability of detecting an error of a given magnitude, and the chance of signaling
an error when no added error is present.
These characteristics determine the power of a quality control procedure. How
1.0
0.8.
0.6.
0.4.
0.2.
i
1
2
3
4
5
Error (SD)
Fig 1. Ideal power function curve. Probability of falserejection(y-intercept) is zero. Probability of error detection of more moderate to large errors is high.
1.0
0. ..
0.6..
I
0.4..
0.2..
Error (SD)
Fig 2. Morerealisticpower function curve. Probability of false rejection is not zero. Probability of error detection is not 1.0 even for large errors.
Laboratory Medicine June 1989
389
1-2S
PROBABILITY
N = 1
o.a
0.6
p
ea
=03a
0.4
0.2
0
1
8
3
4
5
.*,,.-
0.05
2
3
1
SYSTEMATIC ERROR (s)
4
5
RANDOM ERROR (s)
Fig 3.1^ control rule for random error and systematic error. Number of controls (N) = 1.
PROBABILITY
0.8
0.6
0.4
1
if/ /
/
/
W / /
••• / /
/
/
/ / /
/
/
'Mi/
r! / /
/
/
/
0.8
g
0
1
2
3
4
5
SYSTEMATIC ERROR fSJ
Fig 4.1% control rule for systematic error using multiple controls.
does one know whether a given quality
control regimen has adequate power to
detect error? In other words, how do you
quality control the quality control procedure?
Clinicians often speak of the performance characteristics of laboratory tests,
usually expressed as sensitivity (preval-
390
Laboratory Medicine June 1989
ence of positive tests in diagnosed patients) and specificity (prevalence of negative tests in patients without disease). The
performance characteristics of quality
control procedures can be expressed in
similar terms. The probability of error detection (P^) is analogous to sensitivity. It is
defined as the probability of detecting er-
ror of a given magnitude. The probability
of false rejection (Pfr) is analogous to
1—specificity. The Pfr is defined as the
probability of detecting an error when
only the inherent error of the method is
present. Just as sensitivity and specificity
can guide clinicians in ordering and interpreting laboratory tests, so can the Ped and
Pfrguide laboratorians in designing quality control procedures and interpreting
control data. V
The ideal quality control procedure
yields a P^ of 100% (or 1.00) and a Pfrof
0% (or 0.00). Unfortunately, such performance characteristics are usually impossible. There tends to be a direct relationship between Ped and Pf„ and adjustments that increase the P^ frequently increase the P(r. Quality control procedures
with a high probability of error detection
and low probability of false rejection are
the product of judicious combinations of
control techniques and rules.
Computer simulation studies are recommended to accurately quantitate the
P^ and Pfr of a given quality control procedure. These studies are sometimes referred to as "Monte Carlo" simulations,
and involve the use of a "random number
generator" to produce values that simulate control data.4 The steps involved in
the usual type of simulation experiment
are listed below:
1. Collect the real data upon which
the simulation will be based.
2. Characterize the data as to their distribution, mean value, and standard deviation.
3. Use an appropriate pseudo-random
number computer program to generate
sets of values having the required distribution.
4. Apply the quality control rule or
procedure under study to the simulated
data, tabulating the proportion of out-ofcontrol values detected. This is the Pfr.
5. Add error in various increments to
the simulation model. It is usual to add error in multiples of the method's longterm standard deviation, frequently from
0.5 to 5.0 SD in 0.5 increments. Systematic error can be simulated by adjusting the
mean of the simulated values. Random
error can be simulated by adding the appropriate increment of error to the standard deviation. Separate simulation experiments are done for random error and
systematic error.
6. At each level of error, apply the
quality control rule orprocedure under
J-3S
SYSTEMATIC ERROR (S)
Fig 5. l 3s control rule for systematic error using mull tie controls.
study to the simulated data, as in 4 above,
tabulating the proportion of out-of-control values (errors) detected. This represents the P^ for each level of error.
The tabulated results can be expressed
graphically as a plot of probability of rejection versus the size of error. Such a
graph is called a power function graph,
and is usually a sigmoid curve. The optimal control rule (Fig 1) would have a
low probability for detecting small increments of error. An error level is eventually
reached which results in a dramatic increase in out-of-control values, yielding a
steep rise to a P^, of 1.0. The curve then
flattens out, with the Pedfixedat 1.0. The
y-intercept is the Pfr, the probability of detecting "error" when no additional error
is present. Figure 2 shows a more realistic
power function curve. The y-intercept is
not zero, indicating that most quality control procedures have a small but definite
probability of false rejection. The steep
slope of the graph lessens prior to achieving a probability of error detection of 1.0,
indicating that many popular control
rules have a less than perfect error detection rate even for large analytical errors.
The performance characteristics of many
of the more common quality control
rules have been determined and are available in the literature.2-3-5-9 Figure 3 shows
the power function curve for the 1 ^ rule
for both random and systematic error. A
systematic error equivalent to a 3 SD shift
is detected about 83% of the time using
this rule. However, a Pfr of 0.05 indicates
that a false rejection signal is generated
about 5% of the time. Figure 4 shows the
power function curve for the 1^ control
rule for systematic error for various numbers of control observations. While the
Pjj increases with increasing N, the Pfr
shows an unacceptable increase. The 1^
control for systematic error has a lower
Pfr (Fig 4) but also has a lower P^. When
more than one control value is considered
(Fig 5), the P^ can be improved without
much change in the Pfr.
The Pjj and Pfr provide an adequate description of the power of a quality control
system to detect intermittent errors.
When errors persist from run to run, it is
useful to have an estimate of the number
of runs required to detect the error, referred to as "average run length" (ARL).10
Quality control procedures can be characterized by an average run length for acceptable quality (ARLa) and an average
run length for rejectable quality (ARLr).
The ARLa is the average number of runs
(or control observations) that occur before a rejection signal is generated when
there is no added error in the system. The
ARLr is the average number of runs (or
control observations) that occur before a
rejection signal is generated when added
error is present. If control procedures use
only the control observations in the current run, the ARL is inversely proportional to the probability of rejection (P).
Therefore the ARLa = 1/Pfr and ARLr =
Thus, a quality control procedure with
a Pft of 1% (0.01) and a P^ of 50% (0.50)
for a 2s shift, the ARL. will be 1/0.01 or
100, and the ARLr will be 1/0.50 or 2, ie,
an average of 100 runs will pass before a
false rejection signal is generated, and on
average a 2s shift will be detected during
the second run. Use of the ARL concept is
especially helpful when evaluating quality
control procedures based on patient data,
such as "running means" of patient results
or exponential smoothing techniques.
Persistent errors will have a cumulative
effect on such systems, and even small errors will eventually be detected.
In general, increasing the number of
control observations per run will increase
the P^ at the expense of an increasing Pfr.
One can often achieve an increase in P^,
without a significant increase in the Pfr by
judicious grouping of control rules into a
control procedure ("multi-rule"). For example, a combination of the 13s rule and
the 2^ rule greatly enhances the P^, without a significant increase in the Pf„ even
when multiple control observations are
used. The detection of a 2-standard deviation shift improves from less than 40%
with either rule alone to about 60% with
the Ijjl^ control procedure, using three
control observations. The probability of
false rejection remains less than 1%. An
additional strategy is to combine control
rules sensitive to random errors with
those sensitive to systematic error, enhancing the overall sensitivity of the procedure. The 'i-iJl^ procedure is a good
example, since the 1^ rule is sensitive to
random error, while the 2^ rule is preferable for systematic error. As control procedures become more complex, computer
simulation experiments assume a greater
importance in quantifying their power.
Predictions based on intuition are inadequate.
Laboratory Medicine June 1989
391
Judicious combinations of control
rules sensitive to random and systematic
error, designed in such a way that probability of error detection is enhanced while
probability of false rejection is minimized, are the current methods of choice
for quality control in clinical laboratory
medicine. The future will see more aggressive use of computer simulations for
the validation of existing control procedures and the development of new procedures using both assayed control material
and patient values. The goal should always be confident detection of significant
analytical errors without undue sensitivity
to the inherent error of the method •
392
Laboratory Medicine June 1989
References
1. Westgard JO, Barry PL, Hunt MR, et ah A multi-rule Shewhart chart for quality control in clinical
chemistry. Clin Chem 1981;27:493-501.
2. Westgard JO, Groth T: Power functions for statistical control rules. Clin Chem 1979;25:863-869.
3. Westgard JO, Groth T: Design and evaluation
of statistical control rules. Clin Chem
1981;27:1536-1545.
4. Groth T, Falk H, Westgard JO: An interactive
computer simulation program for the design of statistical control procedures in clinical chemistry.
Comput Programs Biomed 1981;13:73-86.
5. Westgard JO, Groth T, Aronnson T, et al: Performance characteristics of rules for internal quality control: Probabilities for false rejection and error detection. Clin Chem 1977;23:1857-1867.
6. Westgard JO, Groth T: Design and evaluation
of statistical control procedures: Applications of a
computer 'quality control simulator' program. Clin
Chem 1981;27:1536-1545.
7. Cembrowski GS, Westgard JO: Quality control
of multichannel hematology analyzers: evaluation
of Bull's algorithm. Am J Clin Pathol 1985;83:337345.
8. Lunetzky ES, Cembrowski GS: Performance
characteristics of Bull's multi-rule algorithm for
the quality control of multichannel hematology
analyzers. Am J Clin Pathol 1987;88:634-638.
9. Cembrowski GS, Lunetzky ES, Patrick CC, et
al: An optimized quality control procedure for hematology analyzers using retained patient specimens. Am J Clin Pathol 1988;89:203-210.
10. Westgard JO, Barry PL: Cost-Effeaive Quality
Control: Managing the Quality and Productivity
ofAnalytical Processes. Washington, DC, AACC
Press, 1986, p 72.