Anatomic Pathology / DISCREPANCY ANALYSIS AND OUTCOMES
Frequency and Outcome of Cervical Cancer Prevention
Failures in the United States
Stephen S. Raab, MD,1 Dana Marie Grzybicki, MD, PhD,1 Richard J. Zarbo, MD, DMD,2
Chris Jensen, MD,3 Stanley J. Geyer, MD,4 Janine E. Janosky, PhD,5 Frederick A. Meier, MD,2
Colleen M. Vrbin,1 Gloria Carter, MD,1 and Kim R. Geisinger, MD6
Key Words: Papanicolaou test; Cervical cancer; Patient safety; Colposcopy
DOI: 10.1309/97JHG6GLY69BVF4Y
Abstract
We measured the frequency and outcome of cervical
cancer prevention failures that occurred in the
Papanicolaou (Pap) and colposcopy testing phases
involving 1,646,580 Pap tests in 4 American hospital
systems between January 1, 1998, and December 31,
2004. We defined a screening failure as a 2-step or
greater discordant Pap test result and follow-up biopsy
diagnosis. A total of 5,278 failures were detected
(0.321% of all Pap tests); 48% and 52% of failures
occurred in the Pap test and colposcopy phases,
respectively. Missed squamous cancers (1 in 187,786
Pap tests), glandular cancers (1 in 19,426 Pap tests),
and high-grade lesions (1 in 6,870 Pap tests)
constituted 4.1% of all failures. Unnecessary repeated
tests or diagnostic delays occurred in 70.8% and 63.9%
of failures involving high- and low-grade lesions,
respectively. We conclude that cervical cancer
prevention practices are remarkably successful in
preventing squamous cancers, although a high
frequency of failures results in low-impact negative
outcomes.
The success of cervical cancer prevention depends on
optimizing the performance of a number of unique testing
steps. Much of the medical literature has focused on the
Papanicolaou (Pap) test screening portion of cervical cancer
prevention, and numerous studies have evaluated the accuracy
of the Pap test and investigated the root cause of Pap test failures.1-3 Despite the well-recognized success of the Pap test in
reducing the incidence and mortality rates of cervical cancer,1,2
the Pap test portion of cervical cancer prevention services has
been the phase most radically transformed by new technologies and the effect of governmental regulations.4-9
Pap test screening failures may occur in any testing phase
from ordering, performing, processing, interpreting, reporting,
and then acting on the result of a test. Yet, if cervical cancer
prevention is viewed from the perspective of the total testing
process,10 the steps of initial patient recruitment, screening, and
follow-up also are vitally important for cancer prevention. It is
well documented that the majority of women in whom cervical
cancer develops never had a Pap test or did not have a recent
Pap test.11,12 The frequency and the effect of failures in phases
of the cervical cancer prevention testing cycle beyond the initial screening examination have been less well studied.
One method of detecting Pap test failures is by using the
cytologic-histopathologic (CH) correlation method by comparing the Pap test and follow-up cervical biopsy diagnoses.7,13,14 The College of American Pathologists and others
documented the usefulness of CH correlation in identifying
overall performance and error-prone steps in Pap testing.13,14
However, CH correlation also may be used to detect failures in
the follow-up colposcopy testing cycle.13 In the ASCUS-LSIL
[Atypical Squamous Cells of Undetermined Significance–LowGrade Squamous Intraepithelial Lesion] Triage Study (ALTS),
Am J Clin Pathol 2007;128:817-824
© American Society for Clinical Pathology
817
DOI: 10.1309/97JHG6GLY69BVF4Y
817
817
Raab et al / DISCREPANCY ANALYSIS AND OUTCOMES
Ferris et al15 reported that the mean colposcopist sensitivity
for detecting cervical intraepithelial neoplasia (CIN) 2 or 3
was lower than expected, indicating that failures in the colposcopy phase of testing are not uncommon.
Our primary goal in this study was to establish a baseline
level of cervical cancer screening failures occurring in the Pap
test and the colposcopy phases of testing using the CH correlation method. Our approach considered that each test in the
prevention process was unique and that each false-negative or
false-positive diagnosis had the potential to result in unnecessary delay or overtesting and/or overtreatment or undertreatment. We measured the frequency and impact of false-negative and false-positive Pap test and colposcopy biopsy diagnoses on patient care.
Materials and Methods
Cervical Cancer Prevention
In most developed nations, cervical cancer prevention
involves the successful performance of several distinct tests,4-9,16
and, conceptually, these tests conjointly may be viewed as a
total testing process.10 Women undergo a screening test, such
as a Pap test, follow-up colposcopy for specific Pap test
epithelial cell abnormality diagnoses,4,17 and a second clinic
visit for lesion excision for specific histopathologic diagnoses.18-20 The implementation of new Pap test technologies,
screening intervals, and follow-up protocols was variably
standardized during our study time frame. In general, women
who had a Pap test diagnosis of LSIL; high-grade SIL (HSIL);
atypical glandular cells; cancer, atypical squamous cells
(ASC), cannot exclude HSIL; or ASC, undetermined significance (ASC-US) (with a positive high-risk human papilloma
virus [HPV] DNA test result) were referred for colposcopy.6,17
Study Population
The study population included all women who received a
Pap test between January 1, 1998, and December 31, 2004, in
4 health care systems. This study is part of a larger project
funded by the Agency for Healthcare Research and Quality
and by the Centers for Disease Control and Prevention as a
means to study and improve patient safety in diagnostic testing and screening.13 The health care systems were the
University of Pittsburgh Medical Center, Pittsburgh, PA;
Henry Ford Health System, Detroit, MI; Western
Pennsylvania Hospital, Pittsburgh; and the University of Iowa
Healthcare, Iowa City; they were deidentified for the purposes of this study. All sites obtained institutional review board
approval for project performance. The 4 institutions adopted
liquid-based cytology between 2000 and 2004 and high-risk
HPV DNA testing for ASC-US diagnoses between 2002 and
2004. Our goal was not to specifically study the implementation of these technologies because they were introduced variably and gradually.
Definitions of Discrepancy and Test Failure
We defined a discrepancy as a difference between the cytologic and histopathologic diagnoses that indicated the presence
or absence of a pathologic entity or a definite difference in the
degree to which the pathologic condition was judged to be present.11 Cytologic and histopathologic diagnostic schema are different, and we developed a scaled hierarchy of categories to
determine if a discrepancy occurred ❚Table 1❚.13 We used the
2001 Bethesda System to classify Pap test results and the CIN
system to classify histopathologic diagnoses.17,21
The CH correlation discrepancy proportion depends on the
method of case detection.13 Correlation processes were standardized at the 4 study sites in 2002 ❚Table 2❚, and the effectiveness of cervical cancer screening before (n = 1,083,221
❚Table 1❚
Discordant and Nondiscordant Diagnoses
Histopathologic Specimen
Papanicolaou test diagnosis
NILM or unsatisfactory
ASC-US
ASC-H
Low-grade SIL (CIN 1)
High-grade SIL (CIN 2/3)
Dysplasia
Squamous cell carcinoma
AGC, favor neoplasia
AGC
AIS
Adenocarcinoma
≥ 2 steps)
Discordant Diagnoses (≥
Nondiscordant Diagnoses (<2 steps)
CIN 1 or higher (including dysplasia, not
otherwise specified)
CIN 2 or higher
Negative
Negative or cancer
Negative
Negative
Negative or CIN 1
Negative or any CIN
Any CIN
Negative or CIN 1
Negative or CIN 1
Negative
Negative or CIN 1
CIN 1 or higher
CIN 1 or CIN 2/3
CIN 1 or higher
CIN 1 or higher
CIN 2 or higher
AIS or adenocarcinoma
Negative or AIS or adenocarcinoma
CIN 2 or higher or AIS or adenocarcinoma
CIN 2 or higher or AIS or adenocarcinoma
AGC, atypical glandular cells; AIS, adenocarcinoma in situ; ASC-H, atypical squamous cells, cannot exclude HSIL; ASC-US, atypical squamous cells of undetermined
significance; CIN, cervical intraepithelial neoplasia; NILM, no evidence of intraepithelial lesion or malignancy; SIL, squamous intraepithelial lesion.
818
818
Am J Clin Pathol 2007;128:817-824
DOI: 10.1309/97JHG6GLY69BVF4Y
© American Society for Clinical Pathology
Anatomic Pathology / ORIGINAL ARTICLE
Pap tests) and after (n = 563,359 Pap tests) standardization
was evaluated separately. We defined a test failure as a 2-step
or greater discrepancy, because in most cases, one of the diagnoses did not accurately describe the presence or absence of a
disease in a patient.13,21,22 We examined only 2-step or greater
discrepancies because 1-step discrepancies were exceedingly
common, generally were associated with minimal impact on
clinical outcome, and were associated with poorly reproducible adjudication of failure cause.13 Thus, some 1-step differences with clinical import (eg, HSIL Pap test and CIN 1
histopathologic specimen) were not examined.
The review pathologist performed a root cause analysis
by reviewing all microscopic slides associated with the discrepant Pap test and histopathologic biopsy results. The
pathologist then adjudicated if the failure was secondary to the
preanalytic (sampling) or analytic (screening or interpretation)
phases of the Pap or colposcopy testing cycles.14 An analytic
failure was a failure in diagnostic categorization by the individual who originally screened or examined the case. A preanalytic failure was a failure in which diagnostic material was
not present on the slides of one case and present on the slides
of the other. Our previous work has shown that traditional
methods of adjudication of test failure root cause exhibited
moderate to low reproducibility. However, this adjudication
method may be used as a first step in exploring more detailed
analysis of failure cause.
We separately examined the relationship between specific diagnostic categorical differences and impact on care for
the following:
1. High-grade lesions (adenocarcinoma in situ and HSIL)
and cancers (squamous carcinoma and endocervical adenocarcinoma) not detected by Pap testing. The ability of the Pap
test to detect some glandular lesions is controversial,23,24 and,
therefore, discrepant cases of endometrial vault adenocarcinomas were excluded from our analysis. Because CH correlation
traditionally is not used to compare Pap test and histopathologic diagnoses with long intervals between specimen
procurement,14 we excluded women who had specimens
procured more than 6 months apart (eg, false-negative Pap test
results based on a 1-year look back). We determined the number of women who had a benign Pap test result with cancer or
high-grade lesion follow-up result.
2. High-grade lesions and cancers not detected in
histopathologic specimens when the Pap test result was abnormal. Because high-grade lesions and cancers are unlikely to
regress in an interval of 6 months or less, most of these discrepancies were secondary to failures in the colposcopy testing phase.
3. Disease misclassification on Pap tests and histopathologic specimens, excluding specimens from women who had
a missed cancer or high-grade lesion. This number represented the diagnostic noise in the screening system and was the
number of underdiagnosed or overdiagnosed lesions (excluding underdiagnosed cancers and high-grade lesions). We recognized LSIL disease regression complicated the test failure
adjudication process.25 We evaluated the cases of women who
had an original and review LSIL Pap test and follow-up
benign histopathologic diagnoses in 2 ways. First, we considered the failure as a preanalytic colposcopy failure regardless
of postcolposcopy follow-up because this discrepant pair
resulted in at least clinical confusion and patient anxiety.
Clinicians did not know whether the LSIL actually had
regressed and responded variably. Second, we only considered
the cases of women who had an abnormal Pap test or
❚Table 2❚
Prestudy and Consensus Methods of Cytologic-Histopathologic Correlation Detection and Adjudication Method
Step
Difference
Cytotechnologist
Prescreening
Reviewer
Adjudication Method
of Error Cause
6-mo wide using previous
month histopathologic
specimens to search for
cytologic cases; search
performed monthly
2-step
disagreement
No
1 of 3 pathologists
Cases shown to
original pathologist
for input; final
decision based on
reviewer
4-mo wide search
Any
disagreement
2-step
disagreement
Any
disagreement
Yes
1 pathologist
None
Yes
2 pathologists
None
No
Pathologist signing
Case shown to
out histopathologic
original cytologist
specimen
if disagreement
Yes
1 pathologist
Site
Case Retrieval Method
Search Interval
A
B
Computer search for all
correlating histopathologic and cytologic
specimens followed
by manual review for
those not correlating
Same as A
C
Same as A
D
12-mo search; cases
reviewed bimonthly
Reviewed daily
Cytologic case identified
when triggered by
positive histopathologic
specimen
Consensus Computer search for
6-mo wide using previous
all correlating histomonth histopathologic
pathologic and cytologic specimens to search for
specimens followed by
cytologic cases
manual review
2-step
disagreement
Final decision based
on reviewer
Am J Clin Pathol 2007;128:817-824
© American Society for Clinical Pathology
819
DOI: 10.1309/97JHG6GLY69BVF4Y
819
819
Raab et al / DISCREPANCY ANALYSIS AND OUTCOMES
histopathologic biopsy follow-up as preanalytic failures with
the remainder falling in an indeterminate failure class.
Impact on Patient Care
We performed clinical record review on all cases with Pap
test–histopathologic specimen discrepancies that had original
or review diagnoses of HSIL/CIN 2 or greater and a random
sample of 10% of all discrepancies that had original or review
diagnoses of less than HSIL/CIN 2.13 A 10% clinical record
review was performed on the latter subset because preliminary
studies showed that women with these discrepancies had very
similar assessments of outcomes and that the annual volume of
these discrepancies was prohibitively large. We extrapolated
the reviewed sample proportions of outcomes to all unreviewed
cases. The clinical record reviewer recorded additional tests
ordered, additional treatment protocols initiated, morbidity or
mortality related to additional tests or treatments, and delays in
diagnosis. A pathologist assessed the clinical impact of the discrepancies using the categories shown in ❚Table 3❚. This
scheme was based on error impact schema previously published in the medical literature.13,26,27 We note that other classification schemes use the term harm to indicate a negative clinical outcome that may be graded in degree.26,27
Data Analysis
All data analyses were performed using SPSS software
(version 14, SPSS, Chicago, IL). Data initially were analyzed
descriptively. Discrepant case frequencies were calculated for
different categories of initial Pap test or histopathologic diagnoses. Proportions of discrepant cases occurring during specific intervals were calculated as n/d, with n the number of
discrepant cases and d the total number of Pap tests examined
during the same interval. Appropriate denominator values
were retrieved from existing electronic laboratory information
systems using query tools designed to extract Pap test workload data.
Differences in discrepancy proportions were examined by
using the Fisher exact test or the χ2 test. Statistical significance was assumed at a P value of .05 or less.
Discrepant Pap test–histopathologic case pairs were
obtained concurrently or nonconcurrently. For nonconcurrent
discrepant case pairs, the Pap test was done during a separate
clinic or office visit before colposcopy. Concurrently obtained
discrepant specimens were obtained during the same colposcopy procedure. The practice of obtaining a Pap test specimen at the time of colposcopy with biopsy was variable;
therefore, we performed separate analyses including and
excluding these data.
Results
The number of all and nonconcurrent testing failures is
shown in ❚Table 4❚ and ❚Table 5❚, respectively. Standardized
CH correlation resulted in greater discrepancy detection compared with nonstandardized methods (P < .001). Excluding
concurrent case pairs, the proportion of women who had a Pap
test and a CH correlation discrepancy was 0.5% (or 1 in 204
Pap tests); 0.2% of the total (1 in 596 Pap tests) had a discrepancy secondary to an analytic failure. In the standardized time
frame, more women had a discrepancy secondary to a colposcopic failure compared with a Pap test failure.
❚Table 3❚
Severity Classification of Discrepancy
Classification
Description
No harm
The clinician acted regardless of an erroneous diagnosis; Example: A patient had a Pap test and
histopathologic biopsy during same colposcopic session; Pap test diagnosed as NILM and biopsy
tissue as CIN 1 (discrepancy adjudicated as a Pap test sampling error); clinician acted on the
histopathologic diagnosis regardless of the Pap test diagnosis
The clinician intervened before harm occurred or the clinician did not act on an erroneous diagnosis;
Example: Pap test diagnosed as HSIL and follow-up cervical biopsy diagnosed as benign (adjudicated
as a histopathologic sampling error); clinician proceeded with a therapeutic procedure regardless of
the histopathologic diagnosis
Near miss
Negative clinical outcome events
Minimal (grade I)
Mild (grade II)
Moderate (grade III)
Severe (grade IV)
Further unnecessary noninvasive diagnostic test(s) performed (eg, Pap test, blood test, or noninvasive
radiologic examination)
Delay in diagnosis or therapy of ≤6 mo
Unnecessary invasive further diagnostic test(s) (eg, tissue biopsy, reexcision, angiogram, or radionuclide study)
Delay in diagnosis or therapy of >6 mo
Minor morbidity owing to delay in therapy or unnecessary further diagnostic efforts or therapy
predicated on presence of (unjustified) diagnosis
Moderate morbidity owing to delay in therapy or (otherwise) unnecessary further diagnostic efforts or
therapy predicated on presence of (unjustified) diagnosis
Loss of life, limb, or other body part or long-lasting morbidity (>6 mo)
CIN, cervical intraepithelial neoplasia; HSIL, high-grade squamous intraepithelial lesion; NILM, no evidence of intraepithelial lesion or malignancy; Pap, Papanicolaou.
820
820
Am J Clin Pathol 2007;128:817-824
DOI: 10.1309/97JHG6GLY69BVF4Y
© American Society for Clinical Pathology
Anatomic Pathology / ORIGINAL ARTICLE
The number of nonconcurrent false-negative Pap test
results (original Pap test diagnosed as no evidence of intraepithelial lesion or malignancy [NILM]) and histopathologic
specimens (original histopathologic specimen diagnosed as
benign) is shown in ❚Table 6❚. CH correlation standardization
resulted in increased detection of discrepancies (P < .001). In
the poststandardized cohort, only 3 squamous cell carcinomas
(5.33 × 10–4% or 1 in 187,786 Pap tests) had an NILM Pap test
result, and none was adjudicated as an analytic failure. The
proportion of women who had an adenocarcinoma diagnosis
on a histopathologic specimen and an NILM Pap test result
was far more common (0.005% or 1 in 19,426 Pap tests with
1 in 46,947 Pap tests having an analytic failure). As a result of
CH correlation standardization, the number of benign
histopathologic specimens with LSIL Pap test results
increased by 3-fold, and 75% of these discrepancies were secondary to a preanalytic colposcopy failure.
The majority of false-negative colposcopy failures (bottom of Table 6) were associated with an LSIL Pap test result.
In this analysis, all cases in which the Pap diagnosis was
adjudicated as LSIL and the colposcopy diagnosis was not
adjudicated as an analytic failure were considered preanalytic colposcopy failures. Review of the follow-up of these
cases showed that 61.0% had a follow-up abnormal Pap test
or histopathologic diagnosis, 30.5% had no abnormal diagnoses, and 8.5% were lost to follow-up. Thus, 33.3% of
women with follow-up may have had LSIL regression before
colposcopy, reducing the total number of false-negative
histopathologic diagnoses from 2,104 to 1,710 cases.
The impact of failures is shown in ❚Table 7❚. We separately examined the impact of 2 discrepancy categories: HSIL/CIN
2+ and LSIL/CIN 1 and lower. In the pre-CH correlation standardized cohorts, more women had no-harm outcomes for all
discrepancy categories. In the standardized cohorts, there was a
❚Table 4❚
Total Cytologic-Histologic Correlation Discrepancies Including Concurrent Cases*
Papanicolaou Test Failures
Year
All Cases
1998-2001
2002-2004
Total
LSIL/CIN 1–
1998-2001
2002-2004
Total
HSIL/CIN 2+
1998-2001
2002-2004
Total
Colposcopy Failures
Total
Total
Preanalytic
Analytic
Total
Preanalytic
Analytic
1,837
3,441
5,278
1,083 (59.0)
1,653 (48.0)
2,736 (51.8)
694 (37.8)
1,016 (29.5)
1,710 (32.4)
389 (21.2)
637 (18.5)
1,026 (19.4)
754 (41.0)
1788 (52.0)
2,542 (48.2)
443 (24.1)
1372 (39.9)
1,815 (34.4)
311 (16.9)
416 (12.1)
727 (13.8)
1,517
3,051
4,568
954 (62.9)
1,508 (49.4)
2,462 (53.9)
662 (43.6)
974 (31.9)
1,636 (35.8)
292 (19.2)
534 (17.5)
826 (18.1)
563 (37.1)
1,543 (50.6)
2,106 (46.1)
294 (19.4)
1,159 (38.0)
1,453 (31.8)
269 (17.7)
384 (12.6)
653 (14.3)
320
390
710
129 (40.3)
145 (37.2)
274 (38.6)
32 (10.0)
33 (8.5)
65 (9.2)
97 (30.3)
112 (28.7)
209 (29.4)
191 (59.7)
245 (62.8)
436 (61.4)
149 (46.6)
213 (54.6)
362 (51.0)
42 (13.1)
32 (8.2)
74 (10.4)
CIN, cervical intraepithelial lesion; CIN 1–, CIN 1 or lower; CIN 2+, CIN 2 or higher; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous
intraepithelial lesion.
* Data are given as number (percentage).
❚Table 5❚
Total Cytologic-Histologic Correlation Discrepancies Excluding Concurrent Cases*
Papanicolaou Test Failures
Year
All Cases
1998-2001
2002-2004
Total
LSIL/CIN 1–
1998-2001
2002-2004
Total
HSIL/CIN 2+
1998-2001
2002-2004
Total
Colposcopy Failures
Total
Total
Preanalytic
Analytic
Total
Preanalytic
Analytic
1,379
2,763
4,142
766 (55.5)
1,125 (40.7)
1,891 (45.7)
439 (31.8)
559 (20.2)
998 (24.1)
327 (23.7)
566 (20.5)
893 (21.6)
613 (44.5)
1,638 (59.3)
2,251 (54.3)
386 (28.0)
1,259 (45.6)
1,645 (39.7)
227 (16.5)
379 (13.7)
606 (14.6)
1,110
2,402
3,512
659 (59.4)
992 (41.3)
1,651 (47.0)
416 (37.5)
526 (21.9)
942 (26.8)
243 (21.9)
466 (19.4)
709 (20.2)
451 (40.6)
1,410 (58.7)
1,861 (53.0)
260 (23.4)
1,059 (44.1)
1,319 (37.6)
191 (17.2)
351 (14.6)
542 (15.4)
269
361
630
107 (39.8)
133 (36.8)
240 (38.1)
23 (8.6)
33 (9.1)
56 (8.9)
84 (31.2)
100 (27.7)
184 (29.2)
162 (60.2)
228 (63.2)
390 (61.9)
126 (46.8)
200 (55.4)
326 (51.7)
36 (13.4)
28 (7.8)
64 (10.2)
CIN, cervical intraepithelial lesion; CIN 1–, CIN 1 or lower; CIN 2+, CIN 2 or higher; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion.
* Data are given as number (percentage).
Am J Clin Pathol 2007;128:817-824
© American Society for Clinical Pathology
821
DOI: 10.1309/97JHG6GLY69BVF4Y
821
821
Raab et al / DISCREPANCY ANALYSIS AND OUTCOMES
shift of outcome classification from no harm to near miss. For
the nonconcurrent and concurrent HSIL/CIN 2+ groups, negative outcomes were observed in 70.8% to 22.7% of all failures, and severe negative outcomes were identified in a small
minority of cases. In the LSIL/CIN 1 or lower groups, negative outcomes were identified in 63.9% and 15.1% of the
nonconcurrent and concurrent discrepant cases, respectively;
the negative outcome was generally minimal or mild.
cancers, less effective in preventing adenocarcinomas, and
associated with a relatively large yearly number of lowgrade test failures with women having an excessive number
of repeated Pap tests, colposcopic procedures with unnecessary biopsies, anxiety, and delays in diagnosis. These data
also indicate that the majority of test failures, particularly
failures in diagnosing high-grade lesions, occur more frequently in the colposcopy rather than the initial Pap testing
cycle of cervical cancer prevention.
The level of squamous cancer prevention produced by
the American screening system is remarkable, and our data
corroborate the findings of numerous previously published
studies.1,2,16,18,19 Good screening tests trade off sensitivity
Discussion
These correlation failure data show that cervical cancer
prevention services are highly effective in preventing squamous
❚Table 6❚
Nonconcurrent False-Negative Pap Test and Colposcopic Examination Results
1998-2001
False-negative Pap test results
Adjudicated histopathologic diagnosis
SCC
Adenocarcinoma or AIS
CIN 2/3
CIN 1
Total
False-negative colposcopic results
Adjudicated cytologic diagnosis
SCC
Adenocarcinoma or AIS
HSIL
LSIL
Atypical (ASC-H or AGC)
Total
2002-2004
All Years
Total
Preanalytic
Analytic
Total
Preanalytic
Analytic
Total
Preanalytic Analytic
5
24
118
203
350
3
18
100
148
269
2
6
18
55
81
3
29
82
355
469
3
17
61
257
338
0
12
21
98
131
8
53
200
558
819
6
35
161
405
607
2
18
39
153
212
0
0
124
399
13
536
0
0
99
261
13
373
0
0
25
138
0
163
1
3
189
1,228
147
1,568
1
3
165
920
147
1,236
0
0
24
308
0
332
1
3
313
1,627
160
2,104
1
3
264
1,181
160
1,609
0
0
49
446
0
495
AGC, atypical glandular cells; AIS, adenocarcinoma in situ; ASC-H, atypical squamous cells, cannot exclude HSIL; CIN, cervical intraepithelial neoplasia; HSIL, high-grade
squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; Pap, Papanicolaou; SCC, squamous cell carcinoma.
❚Table 7❚
Impact of Discrepancy on Patient Care
HSIL/CIN 2+
% Discrepancy Type/Impact
Including concurrent case pairs
No harm
Near miss
Negative clinical outcome
Minimal
Mild
Moderate
Severe
Unknown
Excluding concurrent case pairs
No harm
Near miss
Negative clinical outcome
Minimal
Mild
Moderate
Severe
Unknown
LSIL/CIN 1 or Lower
1998-2001
2002-2004
1998-2001
2002-2004
23.2
16.5
3.2
25.2
57.2
12.8
8.9
34.9
36.3
23.2
0.0
0.4
0.4
39.6
25.8
0.3
2.1
3.8
19.0
10.9
0.0
0.0
0.0
36.8
14.3
0.0
0.4
4.9
23.0
9.8
3.1
22.3
50.8
10.4
3.4
28.7
41.7
25.0
0.0
0.0
0.5
42.3
26.0
0.3
2.2
3.8
25.2
13.6
0.0
0.0
0.0
45.9
17.5
0.0
0.5
4.0
CIN, cervical intraepithelial neoplasia; CIN 2+, CIN 2 or higher; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion.
822
822
Am J Clin Pathol 2007;128:817-824
DOI: 10.1309/97JHG6GLY69BVF4Y
© American Society for Clinical Pathology
Anatomic Pathology / ORIGINAL ARTICLE
for specificity to maximize the detection of at-risk people.
However, it is false to assume that there is a value competition between detecting high-risk lesions and preventing the
95.9% of test failures detected by the CH correlation
process. The cervical cancer screening industry generally
has not focused on reducing these other test failures, which,
assuming 60 million Pap tests per year in the United States,28
occur in more than 285,000 women. Many American laboratories do not track or follow up the majority of CH correlation discrepancies,29 and the focus has been on increasing
Pap test squamous intraepithelial lesion detection through
Pap test technologies5,6,8,9 and regulation7 and not on preventing other failure types, particularly those related to colposcopy failures.
Determining the cause of test failure has been challenging because of observer bias and lack of adequate tools
and frontline skills to perform root cause analysis. Not
unexpectedly, our consortium found that the cause of test
failures generally was multifactorial and secondary to a
host of factors, including organizational structure and lack
of standardization, not simply practitioner lack of skill.30,31
Some pathologists understandably have advocated a medical-legal, panel-review–based approach to CH correlation,32
although this process has resulted in a measure of reproducibility of adjudicated cause rather than a focus on the
interplay of contributing factors. We found that many test
failures were secondary to poor specimen interpretability
composed of factors in specimen collection, processing, and
diagnostic interpretation.30 Adjudication of test failure into
preanalytic or analytic steps and Pap test or colposcopy
phases serves as a basis for quality improvement study and
is not meant to be punitive.13,30 Our data indicate that phases other than the Pap test interpretation phase need to be
addressed from a quality improvement perspective.
Interestingly, cervical cancer screening was far less
effective in the prevention of endocervical adenocarcinoma
than in the prevention of squamous carcinoma. We found
that for every 100,000 Pap tests, approximately 5 endocervical adenocarcinomas were missed. Endocervical adenocarcinomas are increasing in incidence,33 and in the United
States, missed adenocarcinomas are becoming an increasing
source of litigation.34 Not all experts believe that the Pap test
was designed to detect adenocarcinoma,35 and they attribute
false-negative Pap test results as secondary to patient-related
factors. Only rare studies have shown a decrease in mortality due to adenocarcinoma in screened populations.36 This
belief limits quality improvement because our findings indicate that these failures are secondary to combinations of factors, including patient-related factors as a contributing cause.
Our study has several limitations. First, our case selection criteria reduced our discrepancy proportion because we
excluded 1-step discordances and cases of women who had
a discrepant Pap test–histopathologic specimen separated by
more than 6 months. In the United States, the CH correlation
method traditionally has been used to evaluate case pairs
obtained within a relatively short time of each other.13,14
Second, we could not control for preanalytic variation in
screening intervals and follow-up protocols, including colposcopy referral rates and performance of concurrent Pap
tests with histopathologic biopsy. Because correlation practices were not standardized before 2002 and liquid-based
and high-risk HPV technologies were gradually introduced
during this time, we were not able to evaluate how technologies affected failure frequency.
We recognize that assessment of the impact of test failure on care was subjective,37 particularly for women who
may have lesions that regressed.38 However, excluding
cases of possible LSIL regression only reduced the overall
number of discrepant cases by 393. Some argue that cervical cancer screening may be viewed as a set of tests in
which the ultimate goal is the detection of disease rather
than the agreement of individual step diagnoses. In contrast,
we considered the diagnosis of each individual Pap or
histopathologic test as a unique step revealing information
that may conflict or concur with the preceding or follow-up
test diagnosis. We believe that the individual test approach
is a more patient-centric view of safety, and diagnostic discrepancies, at a minimum, may cause confusion and anxiety. An individual test focus also allows for more specific
quality improvement targeting.
The relatively high CH correlation discrepancy proportion justifies the development of error reduction interventions
in cervical cancer screening and further study of patient preference. Quality improvement in cervical cancer screening to
date primarily has not consisted of process improvement and
work redesign. Paradoxically, interventions that decrease the
false-negative Pap test result proportion may result in higher
discrepancy proportions with more women undergoing colposcopy with noncorrelating biopsy results. We note that our
error detection system did not evaluate other process breakdowns such as postanalytic recall failures or the preanalytic
problem of women who are not screened.
From the Departments of 1Pathology and 5Family Medicine and
Clinical Epidemiology, University of Pittsburgh School of
Medicine, Pittsburgh, PA; 2Pathology, Henry Ford Health System,
Detroit, MI; 3Pathology, University of Iowa Healthcare, Iowa City;
4Pathology, Western Pennsylvania Hospital, Pittsburgh; and
6Pathology, Wake Forest School of Medicine, Winston-Salem, NC.
Supported by grant HS13321-01 from the Agency for
Healthcare Research and Quality, Rockville, MD, and a grant from
the Centers for Disease Control and Prevention, Atlanta, GA.
Address reprint requests to Dr Raab: Dept of Pathology,
University of Pittsburgh School of Medicine, 5230 Centre Ave,
Pittsburgh, PA 15232.
Am J Clin Pathol 2007;128:817-824
© American Society for Clinical Pathology
823
DOI: 10.1309/97JHG6GLY69BVF4Y
823
823
Raab et al / DISCREPANCY ANALYSIS AND OUTCOMES
References
1. Kitchener HC, Castle PE, Cox JT. Chapter 7: achievements
and limitations of cervical cytology screening. Vaccine.
2006;24(suppl 3):S63-S70.
2. Saslow D, Runowicz CD, Solomon D, et al. American Cancer
Society guideline for the early detection of cervical neoplasia
and cancer. Cancer. 2002;52:342-362.
3. Renshaw AA. An accurate and precise methodology for
routine determination of the false-negative rate of
Papanicolaou smear screening. Cancer. 2001;93:86-92.
4. Wright TC Jr, Cox JT, Massad LS, et al. 2001 Consensus
guidelines for the management of women with cervical
cytological abnormalities. JAMA. 2002:287:295-304.
5. Davey E, Barratt A, Irwig L, et al. Effect of study design and
quality on unsatisfactory rates, cytology classifications, and
accuracy in liquid-based versus conventional cervical cytology:
a systematic review. Lancet. 2006;367:122-132.
6. Arbyn M, Sasieni P, Meijer CJ, et al. Chapter 9: clinical
applications of HPV testing: a summary of meta-analyses.
Vaccine. 2006;24(suppl 3);S78-S89.
7. Department of Health and Human Services, Health Care
Financing Administration. Medicare, Medicaid and CLIA
programs: regulations implementing the Clinical Laboratory
Improvement Amendments of 1988 (CLIA), 57 Federal
Register 7146 (1992) (codified at 42 CFR §493).
8. Brown AD, Garber AM. Cost-effectiveness of three methods
to enhance the sensitivity of Papanicolaou testing. JAMA.
1999;281:347-353.
9. Lehtinen M, Herrero R, Mayaud P, et al. Chapter 28: studies
to assess the long-term efficacy and effectiveness of HPV
vaccination in developed and developing countries. Vaccine.
2006;24(suppl 3):S233-S241.
10. Lundberg GD. Acting on significant laboratory results
[editorial]. JAMA. 1981;245:1762-1763.
11. Stenkvist B, Soderstrom J. Reasons for cervical cancer despite
extensive screening. J Med Screen. 1996;3:204-207.
12. Kenter GG, Schooderwald EM, Koelma IA, et al. The
cytological screening history of 469 patients with squamous
cell carcinoma of the cervix: does interval carcinoma exist?
Acta Obstet Gynecol Scand. 1996;75:400-403.
13. Raab SS, Grzybicki DM, Janosky JE, et al. Clinical impact and
frequency of anatomic pathology errors in cancer diagnosis.
Cancer. 2005;104:2205-2213.
14. Jones BA, Novis DA. Cervical biopsy–cytology correlation:
a College of American Pathologists Q-Probes study of 22,439
correlations in 348 laboratories. Arch Pathol Lab Med.
1996;120:523-531.
15. Ferris DG, Litaker MS, ALTS Group. Prediction of cervical
histologic results using an abbreviated Reid Colposcopic Index
during ALTS. Am J Obstet Gynecol. 2006;194:704-710.
16. van den Akker-van Marle ME, van Ballegooijen M, van
Oortmarssen GJ, et al. Cost-effectiveness of cervical cancer
screening: comparison of screening policies. J Natl Cancer Inst.
2002;94:193-204.
17. Solomon D, Davey D, Kurman R, et al. The 2001 Bethesda
System: terminology for reporting results of cervical cytology.
JAMA. 2002;287:2114-2119.
18. Eddy DM. Screening for cervical cancer. Ann Intern Med.
1990;113:214-226.
19. Goldie SJ, Kim JJ, Myers E. Chapter 19: cost effectiveness of
cervical cancer screening. Vaccine. 2006;24(suppl 3):S164S170.
824
824
Am J Clin Pathol 2007;128:817-824
DOI: 10.1309/97JHG6GLY69BVF4Y
20. Colgan TJ, Clarke A, Hakh N, et al. Screening for cervical
disease in mature women: strategies for improvement. Cancer.
2002;96:195-203.
21. Flowers LC, McCall MA. Diagnosis and management of
cervical intraepithelial neoplasia. Obstet Gynecol Clin North
Am. 2001;28:667-684, viii.
22. Kohn LT, Corrigan JM, Donaldson MS. To Err Is Human:
Building a Safer Health System. Washington, DC: National
Academy Press; 1999.
23. Wang SS, Sherman ME, Hildesheim A, et al. Cervical
adenocarcinoma and squamous cell carcinoma incidence
trends among white women and black women in the United
States for 1976-2000. Cancer. 2004;100:1035-1044.
24. Etherington IJ, Luesley DM. Adenocarcinoma in situ of the
cervix: controversies in diagnosis and treatment. J Lower Genit
Tract Dis. 2001;5:94-98.
25. Ostor AG. Natural history of cervical intraepithelial neoplasia:
a critical review. Int J Gynecol Pathol. 1993;12:186-192.
26. Battles JB, Kaplan HS, Van der Schaaf TW, et al. The
attributes of medical event reporting systems: experience with
a prototype medical event reporting system for transfusion
medicine. Arch Pathol Lab Med. 1998;122:231-238.
27. Rubin G, George A, Chinn DJ, et al. Errors in general
practice: development of an error classification and pilot study
of a method for detecting errors. Qual Saf Health Care.
2003;12:443-447.
28. Sirovich BR, Welch HG. The frequency of Pap smear
screening in the United States. J Gen Intern Med.
2004;19:243-250.
29. Vrbin CM, Grzybicki DM, Zaleski MS, et al. Variability in
cytologic-histologic correlation practices and implications on
patient safety. Arch Pathol Lab Med. 2005;129:893-898.
30. Raab SS, Stone CH, Wojcik EM, et al. Use of a new method
in reaching consensus on the cause of cytologic-histologic
correlation discrepancy. Am J Clin Pathol. 2006;126:836-842.
31. Raab SS, Vrbin CM, Grzybicki DM, et al. Errors in thyroid
gland fine-needle aspiration. Am J Clin Pathol. 2006:125:873882.
32. Renshaw AA, Young ML, Holladay EB. Blinded review of
Papanicolaou smears in the context of litigation. Cancer.
2004;102:136-141.
33. Vinh-Hung V, Bourgain C, Vlastos G, et al. Prognostic value
of histopathology and trends in cervical cancer: a SEER
population study. BMC Cancer. 2007;7:164 [Epub ahead
of print].
34. Cox JT. AGUS Pap smears: a follow-up strategy. Obgyn
Manage. 1998;27:74-97.
35. van Aspert-van Erp AJ, Smedts FM, Vooijs GP. Severe
cervical glandular lesions and severe cervical combined
lesions: predictive value of the Papanicolaou smear. Cancer.
2004;102:210-217.
36. Nieminen P, Kallio M, Hakama M. The effect of mass
screening on incidence and mortality of squamous and
adenocarcinoma of cervix uteri. Obstet Gynecol. 1995;85:10171021.
37. Gandhi TK, Kachalia A, Thomas EJ, et al. Missed and delayed
diagnoses in the ambulatory setting: a study of closed
malpractice claims. Ann Intern Med. 2006;145:488-496.
38. Jones BA, Novis DA. Follow-up of abnormal gynecologic
cytology: a College of American Pathologists Q-Probes study
of 16,132 cases from 306 laboratories. Arch Pathol Lab Med.
2000;124:665-671.
© American Society for Clinical Pathology