Estimating beyond the trial-represented population
by incorporating studies with self-selected
treatments
Eloise E. Kaizar
Department of Statistics
Ohio State University
February 26, 2016
Joint work with: Joel Greenhouse, Kelly Kelleher,
Taylor Pressler Vydra, Howard Seltman
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
1 / 18
Does antidepressant use cause suicide among
adolescents?
No single study could definitively answer
Available randomized evidence (Hammad, et al. 2006)
24 studies of various psychological disorders
4582 subjects
No completed suicides
Instead adjudicated “suicidality events"
n=87 (2%) experienced suicidality
Goal: Estimate the ’average’ safety of antidepressant use
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
2 / 18
Outline
1
Are Studies Generalizable?
2
Combining RCT and Observational Data
3
Extensions
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
3 / 18
Are Studies Generalizable?
Who do RCTs actually study?
Studies consistently show that it is usual for RCTs to exclude half
or more of the population of interest due to eligibility criteria
Asthma, > 57% (Travers, et al, 2007)
Alcohol treatment, 6 − 70% (Humphreys and Weisner, 2000)
Antidepressants, mean=66% (Zimmerman, et al, 2004)
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
4 / 18
Are Studies Generalizable?
Assessing Consistency with Target Populations
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
5 / 18
Are Studies Generalizable?
Assessing Consistency with Target Populations, cont.
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
6 / 18
Combining RCT and Observational Data
General Methods for Combining RCT and
Observational Data
Three-level meta-analysis (Prevost, Abrams & Jones, 2000)
Response surface (Rubin, 1990)
Bias-adjusted model, e.g.,
Confidence Profile Method (Eddy, Hasselblad & Shachter, 1990)
Multiple bias models (Greenland, 2005)
Cross Design Synthesis (US GAO, 1992)
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
7 / 18
Combining RCT and Observational Data
Cross Design Synthesis
a
b
Nonrandomized
In
te
m rn
od al
er val
at id
or ity
s
(Ohio State University)
y
dit
ali
l v tors
a
n a
ter er
Ex mod
Subjects select
treatment
Treatment effect
ect
Treatment eff
Randomized
Subjects randomized
to treatment
External validity moderators
Estimating Beyond the Trial
Ross-Royall Symposium
8 / 18
Combining RCT and Observational Data
Framework
Use RCT to estimate effect size in the group it represents
Rely on strong internal validity
Use observational data to estimate the generalizability bias
Rely on strong external validity
Put the two estimators together to estimate the population
average effect size
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
9 / 18
Combining RCT and Observational Data
Simplest Cross Design Synthesis
Linear Generalizability Bias
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
10 / 18
Combining RCT and Observational Data
Simple Example: Insulin Pump Use
Question: Is insulin pump use on average effective in improving
metabolic control in the total population of diabetic patients?
Outcome: Mean A1C level (lower is better)
Control: Insulin injections
Goal: Estimate the average treatment effect for use in policy
decision making
Issue: RCTs exclude the noncompliant (< 4 checks per day)
Doyle, et al (2004), Paris, et al (2009)
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
11 / 18
Combining RCT and Observational Data
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
12 / 18
Combining RCT and Observational Data
Assumptions for Unbiasedness
No confounding within the levels of inclusion
U⊥
⊥ T |X or E(Y |T , U, X ) = g(T , X )
Socioeconomic status is independent of the choice to use an insulin
pump, within monitoring groups, OR
Socioeconomic status does not influence metabolic control.
Confounding due to inclusion criteria and other variables are
separate
U⊥
⊥ X |T and U ⊥
⊥ X and E(Y |T , U, X ) = g1 (T , X ) + g2 (T , U)
The frequency of glucose monitoring is independent of
socioeconomic status and does not moderate its effect on
metabolic control.
Note that unlike the experimental estimator, the inclusion criterion
can moderate treatment, as long as this is separate from the other
variables.
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
13 / 18
Combining RCT and Observational Data
CDS Properties
Bias
CDS is better whenever the scaled difference in expected treatment
selection error is smaller than the difference in generalizability bias
(sample selection error)
Variance
CDS is never better, although the difference is expected to be small
whenever the observational data is large
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
14 / 18
Extensions
Improving the Plausibility of Assumption 1
Assumption 1: No confounding within the levels of inclusion
Approach: Use existing methods (e.g., propensity scores) to
reduce/eliminate confounding within levels of inclusion
Simulation
Three standard normal covariates, constant correlation
One influences treatment assignment
Another influences exclusion.
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
15 / 18
Extensions
Estimating Generalizability Bias
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
16 / 18
Summary
Summary
CDS offers a framework for thinking about estimating average
treatment effects in populations of interest that are not
represented by RCTs.
Simple CDS models rely on two rather restrictive (relative to ideal)
assumptions.
More work is necessary to relax such assumptions and improve
its usefulness.
CDS often replaces one data limitation with several others:
Data availability (including treatment in the observational study)
Data harmonization
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
17 / 18
References
References
Doyle E A, Weinzimer SA, Steffen AT, Ahern JAH, Vincent M, Tamborlane WV. 2004. A randomized, prospective trial
comparing the efficacy of continuous subcutaneous insulin infusion with multiple daily injections using insulin glargine.
Diabetes Care. 27: 1554-1558.
Eddy DM, Hasselblad V, Shachter R. 1990. A Bayesian method for synthesizing evidence: the confidence profile method.
International Journal of Technology Assessment in Health Care. 6: 31-56.
Greenland S. Multiple-bias modelling for analysis of observational data. 2005. Journal of the Royal Statistical Society:
Series A. 168: 267-306.
Kaizar EE. 2011. Estimating treatment effect via simple cross design synthesis. Statistics in Medicine. 30(25):2986-3009.
Kaizar EE. 2015. Incorporating Both Randomized and Observational Data into a Single Analysis. Annual Review of
Statistics and Its Application. 2:49-72
Paris CA, Imperatore G, Klingensmith G, Petitti D, Rodriguez B, Anderson AM, Schwartz ID, Standiford DA, Pihoker C.
2009. Predictors of insulin regimens and impact on outcomes in youth with type 1 diabetes: the SEARCH for Diabetes in
Youth study. Journal of Pediatrics. 155: 183-189.
Pressler T and Kaizar EE, 2013. The use of propensity scores and observational data to estimate randomized controlled
trial generalizability bias. Statistics in Medicine. 32(20):3552-3568.
Prevost TC, Abrams, KR, Jones DR. 2000. Hierarchical models in generalized synthesis of evidence: an example based
on studies of breast cancer screening. Statistics in Medicine. 19: 3359-3376.
Rubin, D. 1990. A new perspective on meta-analysis. In K. M. Wachter & M. L. Straff, eds., The Future of Meta-Analysis.
Russel Sage Foundation, 155-165.
U.S. General Accounting Office. 1992. Cross-design synthesis: A new strategy for medical effectiveness research
(GAO/PEMD-92-18). U.S. General Accounting Office, Washington, DC.
http://archive.gao.gov/d31t10/145906.pdf.
(Ohio State University)
Estimating Beyond the Trial
Ross-Royall Symposium
18 / 18