Mathematical Analysis of Cervical Cancer Screening Policies
(Does the Federal Requirement Make Any Sense?)
James C. Benneyan, Ph.D.
Northeastern University, Boston MA
[email protected]
ABSTRACT
This paper summarizes several recent analytic studies to investigate cervical cancer screening policies commonly
used in the examination of Pap smears for early detection. In particular, we investigate and compare:
a) the policy mandated by the Clinical Laboratory Improvement Amendments Act (CLIA), b) recently proposed
automated rescreening approaches, and c) possible improvements and alternatives to CLIA. These alternatives are
motivated by a desire to reduce the high costs associated with diagnostic errors and are similar to manufacturing
policies that use multiple 100% inspections to eliminate nonconforming product in the presence of inspection error.
Using mathematical and expected cost models, significant improvements in early detection and total costs are to be
shown possible by changing the policies currently mandated under CLIA. Additionally, the use of automated
rescreening devices recently approved by the FDA in many cases can be a move in the wrong direction in terms of
accuracy and cost, despite widespread marketing by the manufacturers to the contrary.
INTRODUCTION
Recent litigation and several large settlements due to false-negative Pap smear results have attracted considerable
media attention and public concern. For example, Robb (1993) stated that 15 of every 100 pathologists currently are
involved in a malpractice lawsuit, of which 10% directly concern a false-negative determination from a Pap smear.
In order to minimize future liability and total costs, the re-design of clinical laboratory policies should depend on the
ability to detect truly positive and truly negative patients and on all costs that are incurred as a result of these
policies. Motivated by these statistics and concerns, we therefore have derived several mathematical models as part
of an interdisciplinary effort to develop a thorough statistical and economic understanding of cancer screening
processes, to compare alternatives, and to identify the overall optimal policy. This approach is similar
philosophically to that taken in industry, in which probability and inspection cost models are used to determine
whether or not to inspect production and to identify the effects of multiple inspections of accepted or rejected items.
Currently, as an attempt to improve the accuracy of the overall screening process, the 1976 Clinical Laboratory
Improvement Act (amended 1988) requires that a pathologist randomly sample and rescreen a minimum of 10% of
all Pap smear slides that initially are judged to be negative by laboratory technicians (called cytotechnologists).
Additionally, 100% of slides judged to be positive by cytotechnologists are re-examined by a pathologist. An
editorial by Melamed and Flehinger (1992) in the medical journal Acta Cytologica, however, criticized the use of
this mandatory 10% random rescreening rule as a means of detecting unacceptable performance in Pap smear
screening. Numerous other authors also have questioned this policy since its original introduction in 1976 and
proposed alternative quality assurance policies. A study by Ashton (1989), however, indicated that most cervical
cytology laboratories continue to apply the 10% rescreening rule as a method of quality assurance, with some
facilities rescreening negative slides at rates higher than 10% (Koss, 1993).
More recently, automated rescreening systems have been advocated by their manufacturers (Koss et al 1994, Patten
et al 1996), although their value relative to CLIA has been questioned and point to the need for further analysis
(Hutchison et al 1996, O’Leary et al 1998). The models summarized below can be used to help study and compare
the effectiveness of CLIA, of rescreening rates other than 10%, multiple rescreenings, and of automated rescreening
in terms of overall sensitivity, specificity, and total costs. Results to-date suggest that significant improvements in
the detection of cervical cancer and in total costs are possible via alternate screening policies instead of the current
policy mandated under CLIA, with one to three repeated screenings by cytotechnologists and 0% or 100% of all
negative slides being rescreened by a pathologist (Benneyan, 1997). Furthermore, the proposed use of automated
rescreening has been shown in many likely scenarios to actually increase cost and reduce accuracy when compared
to either CLIA of the optimal non-automated policy (Benneyan 1997, Kaminsky et al 1997).
1
Proceedings of the 2000 IEMS Conference
PROBABILITY OF DETECTING TRULY POSITIVE OR TRULY NEGATIVE PATIENTS
For conventional (non-automated) screening processes, Benneyan (1997) and Benneyan and Kaminsky (1995)
developed mathematical models that determine the probability of detecting truly positive and truly negative patients,
based on the following assumptions and notation:
1.
Each cytotechnologist i and each pathologist has a sensitivity of 1˚-˚αci and 1˚-˚αp, respectively. Thus, the
probability that a pathologist will correctly classify a truly positive slide as positive is 1˚-˚αp and the probability
that the ith cytotechnologist will correctly classify a truly positive slide as positive is 1˚-˚αci. These sensitivities
are assumed to remain constant over time and to be independent of the rescreening rate.
2.
Each cytotechnologist i and each pathologist has a specificity of 1 - βci and 1 - βp, respectively. Thus, the
probability that the ith cytotechnologist will correctly classify a truly negative slide as negative is 1 - βci, the
probability that a pathologist will classify a truly negative slide as negative is 1 - βp. Again, these specificities
are assumed to remain constant over time and are independent of the rescreening rate.
3.
Each slide may be inspected repeatedly by a specified number, j, of independent cytotechnologists. A negative
reading by any cytotechnologist i < j causes a slide to be read by the next technologist. If any technologist
determines a slide to be positive, the slide is given to a pathologist for confirmation. If all j cytotechnologists
determine the slide to be negative, it is randomly selected at a rescreening rate of (r x 100)% by a pathologist.
With these assumptions, the probability of detecting truly positive and truly negative patients can be shown to be:
j


Probability of detecting a truly positive patient = p j = 1 − α p 1 − (1 − r )∏ α ci  , and
i =1


j


Probability of detecting a truly negative patient = q j = 1 − β p 1 − (1 − r )∏ 1 − β ci  .
i =1


(
)
(
)
Additionally, if n positive patients are screened independently, then the number of correct positive and correct
negative identifications can be modelled with binomial distributions with parameters n and pj and parameters n and
qj, respectively. As shown below, these mathematical models can be used to examine the magnitude of the expected
number of correct positive and negative identifications as a function of αci, αp, βci, βp, and the rescreening rate r.
Figure 1 illustrates the effect of the rescreening rate on the overall sensitivity for various scenarios, on the left using
just one cytotechnologist and on the right using two identical cytotechnologists (with αc1 = αc2 ).
Overall Policy Sensitivity After 1 Screening per Smear
Overall Policy Sensitivity After 2 Screenings per Smear
000
000
ab Technician
ensitivit
00
ab Technician
ensitivit
00
.95
.95
00
.75
.60
00
00
.75
.60
00
ound
00
ound
00
00
00
0
0
0
0
0
0
0
0
0
00
0
escreening Rate (%)
0
0
0
0
0
0
0
0
00
escreening Rate (%)
Figure 1: Number of Correct Diagnoses per 1000 Positive Patients (Pathologist Sensitivity = 0.95)
As these examples show, the mandated 10% rescreening policy does not significantly improve the detection of truly
positive patients when compared to a 0% rescreening policy. Additionally, significant improvements in the
detection of truly positive patients are possible by using more than one cytotechnologist. Finally, it can be shown
that the pathologist sensitivity, 1 - αp, is a mathematical upper bound on the overall sensitivity of any sequential
screening policy, regardless of the rescreening rate, whereas the overall specificity is bounded mathematically above
by the pathologist specificity, 1 - βp.
2
Proceedings of the IEMS Conference
Figure 2 summarizes similar results for the number of correct identifications of truly negative patients. Figures 1
and 2 also nicely illustrate the inherent tradeoff that confronts risk managers - namely that repeated screenings and
additional cytotechnologists decrease the rate of false-negatives, but increase the false-positive rate and screening
costs, making identification of the best policy difficult. These tradeoffs therefore are incorporated into the following
cost model in order to help determine the optimal minimum total cost laboratory screening policy.
FigureSpecificity
7b: Rescreening
2 CytotechnologistsScreenings
System
with 2with
Cytotechnologist
1000
Cytotechnologist
Specificity
900
0.95
800
0.75
700
0.60
Bound
600
500
1000
Number of Correct Diagnoses
per 1000 Negative Patients
Number of Correct Diagnoses
per 1000 Negative Patients
System Specificity with 1 Cytotechnologist Screening
Cytotechnologist
Specificity
900
0.95
800
0.75
0.60
700
Bound
600
500
0
10
20
30
40
50
60
70
80
90
100
0
10
20
30
Rescreening Rate (%)
40
50
60
70
80
90
100
Rescreening Rate (%)
Figure 2: Number of Correct Diagnoses per 1000 Negative Patients (Pathologist Specificity = 0.80)
TOTAL COST OF CANCER SCREENING POLICY
Benneyan (1997) and Benneyan and Kaminsky (1995) also developed several cost models to help determine the
total cost of any specific lab screening policy, based on the following additional assumptions and notation:
4.
Slides that are judged to be negative by all j cytotechnologists either are accumulated in a pool until that pool
reaches a pre-determined size m, at which time a random sample of size rm is selected for rescreening by a
pathologist, or are randomly selected to be rescreened immediately with probability r.
5.
The following costs may be incurred a random number of times per every n processed smears:
k1
k2
k3
k4
k5
=
=
=
=
=
per slide screening cost by a cytotechnologist,
per slide screening cost by a pathologist,
average cost per false positive result,
average cost per false negative result, and
per slide screening cost by a automated rescreening process.
Note that all of the realities of the true system are captured in the above assumptions and that the concept of a cycle
in assumption 4 defines a regenerative process. Using these costs and the previous notation, the expected value of
the cost per n processed smears for any particular values of j and r, denoted ECj,r, can be shown to be:
j −1 i−1
j −1 i−1
j −1
j −1








EC j , r = k1np 1 − α c1 + ∑ i ∏ α c k 1 − α ci  + j ∏ α c k  + k1n(1 − p)β c1 + ∑ i ∏ 1 − β c k β ci  + j ∏ 1 − β c k 




 k =1
 k =1
k =1
k =1
i = 2
i = 2


j
j
j




 j



+ k 2n(1 − p)1 − (1 − r) p∏ α ci + (1 − p)∏ 1 − β ci  + k 3n(1 − p)1 − ∏ 1 − β ci (1 − r) + k 4np1 − 1 − α p 1 − ∏ α ci (1 − r) .



 i=1
 i=1

 i=1

i=1


(
)
(
(
)
(
(
)
(
)
)
(
)
)
This cost model now can be used to evaluate a variety of screening policies for any specified of values of k1, k2, k3,
k4, αci, αp, βci, βp , p, and n. Moreover, the optimal rescreening rate r and number of cytotechnologists j which
minimize the total expected cost also can be found, such as by a simple half-interval search. For example, Table 1
compares the expected costs for four situations where the following values were assumed, based on discussions with
pathology practitioners, and assuming identical cytotechnologists such that αc1 = αc2 = αc3 = = αc5 :
k1 = $3.00, k2 = $9.00, k3 = $750.00, n = 100, and p = 0.015.
3
Proceedings of the IEMS Conference
Table 1. Examples of Total Expected Cost per 100 Smears and Identification of the Optimal Policy
Expected Cost per 100 Processed Smears
Average Cost per False-Negative = $5,000
$50,000
Number of
Technicians
Rescreen
Rate (%)
Technician Sens = .95
Pathologist Sens = .95
Technician Sens = .60
Pathologist Sens = .60
Technician Sens = .75
Pathologist Sens = .95
Technician Sens = .75
Pathologist Sens = .95
1
1
0
10 ("CLIA")
$ 1,273 *
$ 1,673
$ 5,337
$ 2,695
$ 22,102
$ 5,593
$ 2,953
$ 20,756
1
20
$ 2,072
$ 5,848
$ 3,210
$ 19,410
1
30
$ 2,472
$ 6,104
$ 3,467
$ 18,064
1
40
$ 2,871
$ 6,360
$ 3,725
$ 16,718
1
50
$ 3,271
$ 6,615
$ 3,982
$ 15,373
1
60
$ 3,670
$ 6,871
$ 4,239
$ 14,027
1
70
$ 4,070
$ 7,127
$ 4,497
$ 12,681
1
80
$ 4,470
$ 7,382
$ 4,754
$ 11,335
1
90
$ 4,869
$ 7,638
$ 5,011
$ 9,990
1
100
$ 5,269
$ 7,894
$ 5,269
$ 8,644
2
2
0
10
$ 1,434
$ 4,760 *
$ 5,102
$ 1,861 *
$ 2,230
$ 9,244
$ 1,845
2
20
$ 2,257
$ 5,444
$ 2,599
$ 9,180
2
30
$ 2,669
$ 5,785
$ 2,968
$ 9,149
2
40
$ 3,080
$ 6,127
$ 3,337
$ 9,117
2
50
$ 3,492
$ 6,468
$ 3,706
$ 9,085
2
60
$ 3,903
$ 6,810
$ 4,075
$ 9,053
2
70
$ 4,315
$ 7,152
$ 4,444
$ 9,021
2
80
$ 4,727
$ 7,493
$ 4,813
$ 8,989
2
90
$ 5,138
$ 7,835
$ 5,182
$ 8,957
2
100
$ 5,550
$ 8,176
$ 5,551
$ 8,926
3
3
0
10
$ 1,890
$ 4,804
$ 2,002
$ 6,379
$ 2,283
$ 5,168
$ 2,383
$ 6,660
3
20
$ 2,676
$ 5,532
$ 2,765
$ 6,941
3
30
$ 3,068
$ 5,896
$ 3,146
$ 7,223
3
40
$ 3,461
$ 6,260
$ 3,528
$ 7,504
3
50
$ 3,853
$ 6,624
$ 3,910
$ 7,786
3
60
$ 4,246
$ 6,988
$ 4,291
$ 8,067
3
70
$ 4,639
$ 7,352
$ 4,673
$ 8,348
3
80
$ 5,031
$ 7,716
$ 5,054
$ 8,630
3
90
$ 5,424
$ 8,080
$ 5,436
$ 8,911
3
100
$ 5,816
$ 8,444
$ 5,818
$ 9,193
4
:
0
:
$ 2,339
$ 5,082
$ 2,368
$ 5,994 *
:
:
:
:
4
100
$ 6,070
$ 8,697
$ 6,071
$ 9,446
5
:
0
:
$ 2,766
$ 5,440
$ 2,775
$ 6,212
:
:
:
:
100
$ 6,310
$ 8,938
$ 6,312
$ 9,687
5
Cost per single technician evaluation = $3
Cost per single pathologist evaluation = $9
Cost per false-positive = $750
$ 9,212
Technician Specificity = 95%
Pathologist Specificity = 95%
Incidence Rate of Cervical Cancer = 1.5%
* = Optimal policy
4
Proceedings of the IEMS Conference
DISCUSSION
The scenario in column 3 represents fairly accurate technicians and pathologists with high sensitivities and
specificities, resulting in the minimal total cost policy being a single cytotechnologist screening and a pathologist rescreening rate of 0% (not 10%). Note that a savings of approximately $400 per 100 smears would result from
changing from current practice to this optimal policy, equating to roughly $208 million per year if extrapolated to
the 52 million smears examined annually in the U.S. Under CLIA, however, this policy is not permitted. It also is
interesting that the policy of single screening with a 10% rescreening rate, which is the minimum permitted under
CLIA, has a higher expected cost than 2 cytotechnologist screenings with a 0% rescreening rate.
The scenario in column 4 uses the same costs and high specificities, but now with lower sensitivities (more falsenegatives) for both the technicians and pathologists. These results indicate that the optimal policy now is to use 2
cytotechnologists, again with a rescreening rate of 0%, at a savings of over $832 per 100 smears, or roughly $418
million and 93,600 fewer false-negatives per year. In this case, the multiple sequential technician screenings
counteract their low individual sensitivities, although the expected cost of the optimal policy is considerably higher
than for the previous example. The overall cost of poor individual sensitivity therefore can be quite high. The
example in column 5 may be the most likely in terms of sensitivities, again resulting in the optimal policy being to
use 2 independent screenings and a 0% rescreening rate. In this case, changing from CLIA to this optimal policy
would result in $1,092 savings per 100 smears and an increase in overall process sensitivity by over 20%, or roughly
$568 million and 120,400 fewer false-negatives per year.
For completeness, column 6 illustrates a more extreme false-negative cost, resulting in 4 independent evaluations
and 0% rescreening being optimal by several magnitudes, with a savings over CLIA of almost $15,000 per 100
smears and an increase in overall process sensitivity by over 28%, or roughly $7.4 billion per year and 164,000
fewer false-negatives per year. This last case illustrates that scenarios theoretically can exist where something more
than double rescreening is optimal. This can happen, for example, when the cost of false-negatives is extremely
high relative to that of false-positives and to that of a pathologist examination, as might be the case when testing for
life threatening medical conditions. Note that high false-negative costs very dramatically increase the total cost of
any laboratory policy, with less rescreening resulting in higher total costs.
$10.00
Inspection Cost
Accepted NC Cost
$8.00
Rejected NC Cost
Rejected C Cost
(Sale Price)
$6.00
Total Expected Profit
$4.00
$2.00
$0.00
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15
($2.00)
($4.00)
Total Exp Cost per 100 Smears
The above results are similar to industrial experiences where multiple inspections significantly improve overall
outgoing quality in the presence of inspection error, the optimal number of re-inspections often is between 1 to 3,
and partial inspection of manufactured items never results in minimum costs. In fact, Benneyan and Kaminsky
(1995) showed that the optimal screening policy for any combination of sensitivities, specificities, incidence rate,
and number of cytotechnologists also always must employ either 0% or 100% rescreening. That is, in no case will
partial rescreening ever result in the best possible laboratory policy. This is a very significant result in light of
current laboratory practices and federal requirements to the contrary. An algorithm also has been developed to
identify the optimal amount of rescreening in any particular scenario that minimizes all competing costs, again very
similar to industrial inspection research. For example, Figure 5 compares an example of the identification of the
optimal inspection policy on the left with the identification of the optimal cervical cancer screening policy on the
right.
$25,000
Total Cost @ 0% Rescreening Rate
$20,000
Total Cost @ 100% Rescreening Rate
$15,000
$10,000
$5,000
$0
1°
2
3
4*
5
6
7
8
9
10
Number of Cytotechnologist Screenings
Number of Inspections (j)
Figure 5: Examples of Optimal Redundant Industrial Inspection and Cancer Screening Policies
5
Proceedings of the IEMS Conference
CONCLUSION
The models summarized in this paper can assist risk managers, clinicians, and regulatory bodies to design more
effective and minimum cost screening policies and requirements. Preliminary exploration across a range of
scenarios developed from the literature suggests that significant improvements are possible by revising the CLIA
legislation, estimated between 90,000 to 160,000 fewer false-negatives and $400 to $850 million savings per year
nationwide. Additionally, these models have been used to show that:
•
•
•
•
CLIA mathematically never is optimal by any criteria and always increases total costs;
Overall CLIA sensitivity never can be improved beyond certain mathematical bounds;
No amount of partial rescreening is ever optimal; and
In some plausible scenarios, 2-4 multiple evaluations of each smear is optimal.
Although not the focus here, similar results have shown that the proposed use of automated rescreening technology
recently approved by the FDA can dramatically increase overall costs without significantly improving sensitivity,
despite widespread marketing by manufacturers to the contrary (Benneyan 1997 1998, Kaminsky et al 1997). It is
significant that these theoretic results also closely agree with a recent independent large empirical study (O Leary et
al 1998).
REFERENCES
Ashton PR: "American Society of Cytotechnology Quality Assurance Survey Data: Summary Report". Acta
Cytologica 33:451-454, 1989.
Benneyan JC: Optimal Policies for Clinical Laboratory Quality Control, AHCPR Grant Summary ROSHS0932901, 1998, www.coe.neu.edu/~benneyan/papers/ahcpr_summary.pdf.
Benneyan JC: Some Approaches to Quality in the Presence of Inspection Error, PhD dissertation, Industrial
Engineering and Operations Research, University of Massachusetts, Amherst, 1997.
Benneyan JC, Kaminsky FC: "Statistical and Economic Models for Analysis and Optimal Design of Laboratory
Screening Policies for Cervical Cancer", Annals of Operations Research, 67:235-285, 1995.
"Clinical Laboratory Improvement Amendments of 1988, PL 100-578", Congressional Record. 134: 3828-63, 1988.
Hutchinson ML, Inhorn SL, Papillo J, Lapidus SN: A Perspective on Modern Quality Control Methods , Acta
Cytologica., 1995.
Kaminsky FC, Benneyan JC, Mullins DL: "Mathematical Models for Evaluating Overall Process Sensitivity,
Specificity, and Cost of Automated Rescreening in Cervical Cytology", Acta Cytologica 41(1):209-223, 1997.
Kaminsky FC, Benneyan JC: "Alternative Inspection Policies in Pap Smear Testing for the Detection of Cervical
Cancer", presented at the First International Applied Statistics in Medicine conference, 1995.
Koss LG: "Cervical (Pap) Smear: New Directions". Cancer 71:1406:1412, 1993.
Koss LG, Lin E, Schreiber K, Elgert P, Mango L: "Evaluation of the Papnet™ Cytologic Screening System for
Quality Control of Cervical Smears", Anatomic Pathology 101(2): 220-229, 1994.
Melamed MR, Flehinger BJ: "Reevaluation of Quality Assurance in the Cytology Laboratory". Acta Cytologica
36:61-65, 1992.
O Leary TJ, Tellado M, Bucker S-B, Ali IS, Stevens A, Ollayos CW: PAPNET-Assisted ReScreening of Cervical
Smears: Cost and Accuracy Compared with a 100% Manual Rescreening Strategy , JAMA 279:235-237, 1998.
Patten SF Jr, Lee JSJ, Nelson AC: "NeoPath, Inc.: NeoPath AutoPap 300 Automatic Pap Screener System", Acta
Cytologica 40(1):45-52, 1996.
Robb JD: "The Pap Smear is a Cancer Screening Test: Why Not Put the Screening Error Rate in the Report?".
Diagnostic Cytopathology 9:485-486, 1993.
6
Proceedings of the IEMS Conference