Statistical Identification of Potential Selection Bias
in Retrospective Chart Reviews
James E. Signorovitch, PhD1, William M. Reichmann, PhD1, Nanxin Li, PhD1, Zhimei Liu, PhD2,
Yanni Hao, PhD2, Jose Ricardo Perez, MD2
Analysis Group, Inc., Boston, MA, USA; 2Novartis Pharmaceuticals Corporation, East Hanover, NJ, USA
1
PRM21
BACKGROUND AND OBJECTIVES
• Retrospective chart reviews provide valuable data on real-world treatment
patterns and outcomes
• It is important to obtain a representative sample of charts; randomization or
sequential selection procedures are often specified for this purpose
• Physicians recruited from different commercially-maintained panels, or from different
institutions, may have different levels of compliance with chart sampling procedures
• Motivated by divergent results from two chart reviews conducted with different panel
vendors, we sought to identify and validate a statistical test for selection bias in retrospective
chart samples
MOTIVATING EXAMPLE
METHODS
• Two retrospective chart reviews for patients with metastatic renal cell carcinoma
(mRCC) were conducted with US-based oncologists recruited by different panel
vendors in 2011 and 20141,2
• Both studies employed similar retrospective cohort designs and sample selection
procedures (Figure 1)
• Included patients were required to have initiated second-line targeted
therapy for mRCC during an index window that closed ~1.5 years prior to the
date of chart review
• Participating physicians were instructed to randomly select up to five eligible
patient charts, and were provided with a randomization algorithm
• Overall survival (OS) was measured as the time from initiation of secondline therapy to death, with censoring at loss to follow-up or the end of data
availability
• OS in the 2011 study was consistent with benchmarks based on published
observational studies and randomized controlled trials, which had medians
ranging from 12.3 to 16.6 months after the initiation of second targeted therapy
(among patients receiving first targeted therapy with treatments other than
cytokines or bevacizumab)3-5
• However, OS in the 2014 study appeared surprisingly prolonged, with the
median not yet reached by 25 months
• We hypothesized that while some physicians in the 2014 study may have followed the random
selection procedure, others may have selected conveniently available charts for their recentlyseen patients
• This would result in length-biased sampling, also called survivor bias, in which patients alive
at the time of chart review, and therefore having longer OS, are more likely to be sampled
(Figure 2)
• We used latent class analysis to test whether charts obtained in 2014 represented a
homogeneous sample of physicians vs. an admixture of two types of physicians, each with a
different distribution of OS for their patients
• A Cox proportional hazards model was used, with latent classes defined at the physicianlevel and baseline hazard functions allowed to differ by class
• The model adjusted for multiple patient-level risk factors (age, sex, type of first targeted
therapy, progression while on first targeted therapy, type of second targeted therapy,
Eastern Cooperative Oncology Group (ECOG) performance status, duration of mRCC, liver
metastasis) and for physician years of practice and academic vs. community practice
• Models with one vs. two latent classes of physicians were compared using the Akaike
information criterion (AIC) and the Bayesian information criterion (BIC)
• OS and risk factor profiles were compared across classes of physicians, with classes assigned by
the fitted model
Figure 2. Example Survival Times in Random vs. Length-Biased Samples
Figure 1. Retrospective Cohort Design
Randomization procedure
•
•
Physicians were instructed
to search for an eligible
patient whose last name
started with a randomly
generated letter
After one eligible chart was
identified, or if no such
charts were identified,
another random letter was
provided for the next
search for an eligible
patient
Date of
chart review
Index
window
Eligible patients
starting secondline treatment[1,2]
Date of
chart review
Index
window
Random
sampling
of 4 patients
Patient alive at date
of chart review
Patient deceased
before date of
chart of review
Length-based
sampling
of 4 patients
1.5 years
Time
Notes:
[1] In the 2011 study, the index window was 10/2009 to 6/2010 and the chart review was conducted in 12/2011.
[2] In the 2014 study, the index window was 2/2012 to 1/2013 and the chart review was conducted in 7/2014.
Time è
RESULTS
1.00
1.00
Class 2 (65%)
0.75
OS by Class (Figure 3)
• Class 1 included 35% of the physicians
and had median OS of 13.9 months,
consistent with the 2011 study1 and
with benchmarks drawn from other
publications3-5
• Class 2 included 65% of the physicians
and had abnormally prolonged OS
estimated at over 90% after 25 months
0.75
0.50
Benchmark range
for median OS3-5
0.25
0.00
0.00
5
10
15
Years of practice, mean (SD)
Practice setting, n (%)
Community
Academic
Number of eligible patient charts, mean (SD)
Number of charts selected for review, mean (SD)
Number of charts censored for OS, mean (SD)
Selected only censored charts, n (%)
Selected only recently seen patients, n (%)*
Used EMR to abstract data, n (%)
5
10
1.00
Class 2, ECOG 0
Class 2, ECOG 1
0.75
Class 2, ECOG 2+
Class 1, ECOG 0
0.50
0.25
Class 1, ECOG 1
Class 1, ECOG 2+
0.00
10
15
Months
15
20
25
Table 2. Patient Characteristics by Class
Class 1
(N=111)
15.2 (6.1)
Class 2
(N=207)
13.7 (6.1)
90 (81%)
21 (19%)
19.8 (20.6)
3.6 (1.4)
1.7 (1.5)
9 (8%)
2 (2%)
92 (83%)
148 (71%)
59 (29%)
28.7 (27.1)
3.7 (1.5)
3.6 (1.5)
177 (86%)
90 (43%)
170 (82%)
*Patients were defined as recently seen if the last follow-up date was within 90 days of chart
abstraction.
Figure 4. Impact of ECOG on OS by Physician Class in the 2014 Chart Review
Survival
0
Months
Table 1. Physician Characteristics by Class
• The absolute effect of ECOG status, an important prognostic factor, differed
substantially between physician classes; patients with the worst ECOG status (2+) in
physician class 2 had longer OS compared to patients with the best ECOG status (0)
in physician class 1
• However, based on the fitted Cox proportional hazards model, the effects of
ECOG status were in the expected directions in each class, with lower ECOG status
associated with lower hazards of death:
• In class 1, hazard ratios were 0.13 for ECOG 0 vs. 2 and 0.39 for ECOG 1 vs. 2
• In class 2, hazard ratios were 0.07 for ECOG 0 vs. 2 and 0.23 for ECOG 1 vs. 2
Note: Kaplan-Meier curves.
25
Class 1 (35%)
Note: Kaplan-Meier curves.
Effect of ECOG Performance Status on OS by Class (Figure 4)
5
20
Benchmark range
for median OS 3-5
Months
• Physicians in class 2 had fewer years
of practice, and were more likely to
practice in academic centers, see more
mRCC patients, choose only censored
patients for chart abstraction, and choose
more recently seen patients compared
to physicians in class 1; there was no
difference in use of electronic medical
records (EMR) vs. paper-based charts
(Table 1)
• The patients selected by class 2 physicians
tended to be younger and had fewer
prognostic indicators for increased
mortality, including less progression
while on first targeted therapy and a
smaller number of metastatic sites; the
magnitude of these differences was not
large (Table 2)
0
0.50
0.25
0
Physician and Patient
Characteristics by Class
Survival
• The model with two physician classes fit
the data significantly better than the one
class model (AIC reduced from 2447 to
2278; BIC reduced from 2474 to 2333)
• When physicians were assigned to classes
based on the fitted model, the average
posterior probability in assigned classes
was greater than 85%, indicating good
confidence in classification
Figure 3. OS Before and After Physician Classification in the 2014 Chart Review
All Physicians
Physicians Stratified by Class
Survival
Number of Classes
20
25
Age (years) at mRCC diagnosis, mean (SD)
Gender, n (%)
Male
Female
Progressed on first targeted therapy, n (%)
ECOG performance status, n (%)
0
1
2+
Duration of mRCC < 12 months, n (%)
Liver metastasis, n (%)
DISCUSSION
• Results are consistent with the existence of two
types of physicians
• One class of physicians is selecting a
random, representative sample of charts as
instructed, resulting in expected durations
of OS
• Another larger class is selecting recentlyseen patients with artificially prolonged OS
• It is unlikely that between-physician
differences in patient risk profiles or treatment
patterns could fully explain these findings
• Analyses adjusted for multiple baseline
characteristics
• Patients showed only small baseline
differences between physician classes
• The highly favorable OS for the most severe
ECOG group (ECOG 2+) in the apparently
biased class 2 is difficult to explain without
sampling bias. For comparison, based
on the Memorial Sloan-Kettering Cancer
Center (MSKCC) risk criteria among secondline patients, best-case scenario OS is 10
months for patients with ECOG 2+ and no
other risk factors6
• Despite selection bias, it is possible for the
estimated effects of risk factors to have the
correct direction; effects estimated within the
unbiased class should be more reliable than
those in the biased class
• The present study considered OS in oncology
charts, however the methods can be extended
to other outcomes, treatment patterns, and
therapeutic areas
Class 1
(N=402)
60.8 (10.2)
Class 2
(N=771)
59.2 (10.2)
265 (66%)
137 (34%)
330 (82%)
520 (67%)
251 (33%)
594 (77%)
139 (35%)
183 (46%)
80 (20%)
86 (21%)
130 (32%)
264 (34%)
373 (48%)
134 (17%)
273 (35%)
206 (27%)
CONCLUSIONS
• Physicians recruited from different
commercially-maintained panels
may differ in their compliance with
sample selection instructions
• Latent class analysis can provide
a helpful tool for detecting and
screening-out potential selection
bias in retrospective samples of
patient charts
REFERENCES
1. Wong MK, Yang H, Signorovitch JE, et al. Comparative
outcomes of everolimus, temsirolimus and sorafenib
as second targeted therapies for metastatic renal cell
carcinoma: a US medical record review. Curr Med Res
Opin. 2014;30(4):537–45
2. Vogelzang NJ, Pal SK, Signorovitch JE, et al.
Comparative effectiveness of everolimus (EVE) and
axitinib (AXI) for 2nd-line treatment of metastatic renal
cell carcinoma (mRCC) in the US: a retrospective chart
review. Poster presented in 2015 American Society
of Clinical Oncology (ASCO) Genitourinary Cancer
Symposium
3. Ko JJ, Xie W, Kroeger N, et al. The International
Metastatic Renal Cell Carcinoma Database Consortium
model as a prognostic tool in patients with metastatic
renal cell carcinoma previously treated with first-line
targeted therapy: a population-based study. Lancet
Oncol. 2015;16(3):293–300
4. Motzer RJ, Escudier B, Tomczak P, et al. Axitinib versus
sorafenib as second-line treatment for advanced renal
cell carcinoma: overall survival analysis and updated
results from a randomised phase 3 trial. Lancet Oncol.
2013;14(6):552–62
5. Hutson TE, Escudier B, Esteban E, et al. Randomized
phase III trial of temsirolimus versus sorafenib as
second-line therapy after sunitinib in patients
with metastatic renal cell carcinoma. J Clin Oncol.
2014;32(8):760–7
6. Motzer RJ, Mazumdar M, Bacik J, et al. Survival and
prognostic stratification of 670 patients with advanced
renal cell carcinoma. J Clin Oncol. 1999;17:2530–40
Presented at the ISPOR 20th Annual International Meeting, Philadelphia, PA, USA, May 16–20, 2015