PCORI Methodology Standards:
Academic Curriculum
© 2016 Patient-Centered Outcomes Research Institute. All Rights Reserved.
Module 3b: Background
Category 9: Adaptive and
Bayesian Trial Designs
Prepared and presented by Gary Rosner, ScD
Example of Bayesian Inference in a Randomized Clinical Trial
 Levamisole and Fluorouracil for Adjuvant Therapy
of Resected Colon Carcinoma (Moertel, et al., 1990)
 Adjuvant therapy with levamisole (LEV) and
fluorouracil (5-FU) significantly reduced cancer
recurrence compared with no adjuvant therapy
in early studies
 Survival now the primary endpoint
 Randomized 1,296 patients to:
• No therapy (observation),
• LEV alone, or
• LEV + 5-FU
 Result—LEV + 5-FU reduced the overall death
rate among Stage C patients by 33% (p = 0.0064)
Levamisole
R
A
N
LEV + 5-FU
D
Observation
Source: Moertel, C. G., Fleming, T. R., Macdonald, J. S., et al. (1990). Levamisole and fluorouracil for adjuvant therapy of resected colon carcinoma. The New
England Journal of Medicine, 322(6), 352–358. http://doi.org/10.1056/NEJM199002083220602
3
Clinical Trial Example
 Primary outcome:
 Survival
 Parameter:
 Hazard ratio
 (OBS:LEV or OBS:LEV+5-FU)
 Alternative:
 Hazard ratio = 1.35
 One-sided 0.05 test  380 deaths
Source: Moertel, C. G., Fleming, T. R., Macdonald, J. S., et al. (1990). Levamisole and fluorouracil for adjuvant therapy of resected colon carcinoma. The New
England Journal of Medicine, 322(6), 352–358. http://doi.org/10.1056/NEJM199002083220602
4
Bayesian Design and Analysis of a Randomized Clinical Trial
 Bayesian analysis of Spiegelhalter, et al., “Bayesian Approaches to Randomized Trials”
 There is a range of possible hazard ratios with associated probabilities a priori
 Prior probabilities can be thought of as prior beliefs or measures of uncertainty about
more likely and less likely hazard ratios
 The prior uncertainty may be informed by results from prior studies, experience,
etc.
 Furthermore, we may characterize two different priors for the hazard ratio:
 A skeptic’s prior:
• The hazard ratio is most likely around 1 (no treatment difference)
 An enthusiast’s prior:
• The hazard ratio most likely favors the new treatment (i.e., > 1)
Source: Spiegelhalter, D. J., Freedman, L. S., Parmar, M. K. B. (1994). Bayesian approaches to randomised trials. J Roy Stat Soc Series A, 157:357–387.
5
Graphs of Prior Distributions
 Characterize initial beliefs in relative
treatment effects
 Skeptic may believe treatments have
roughly the same effect, on average
 Enthusiast likely thinks one treatment
leads to better outcomes, on average
Source: Spiegelhalter, D. J., Freedman, L. S., Parmar, M. K. B. (1994). Bayesian approaches to randomised trials. J Roy Stat Soc Series A, 157:357–387.
Courtesy of JSTOR.
6
Data
 Observed hazard ratios:
 LEV+5-FU versus Observation:
• Hazard ratio = 0.40 (192 deaths)
 Data show convincing evidence that LEV+5-FU
is better than control, but …
 … 78% probability that the treatment is
clinically superior
Source: Spiegelhalter, D. J., Freedman, L. S., Parmar, M. K. B. (1994). Bayesian approaches to randomised trials. J Roy Stat Soc Series A, 157:357–387.
Courtesy of JSTOR.
7
Graph Showing Updated Knowledge About
the Hazard Ratio: Prior and Posterior
 Enthusiast more convinced by the data
 Prior: ~50% probability > equivalent
 Posterior: ~75% probability > equivalent
 Skeptic impressed but convinced by data?
 Prior: 50% probability < equivalent
 Posterior: 1.5% probability < equivalent
• LEV+5-FU no worse!
 Prior: ~6% probability > equivalent
 Posterior: ~36% probability > equivalent
 Posterior mean = 0.25
• Within range of equivalence
Source: Spiegelhalter, D. J., Freedman, L. S., Parmar, M. K. B. (1994). Bayesian approaches to randomised trials. J Roy Stat Soc Series A, 157:357–387.
Courtesy of JSTOR.
8
Interim (Predictive) Analysis of LEV+5-FU Versus Observation
 Should the trial
continue until 380
deaths (+ ~190)?
Decision (based on
final 99% interval)
 Predict likely
outcomes
Bayesian
(skeptic)
Likelihood
Obs > LEV+5-FU
0.000
0.000
LEV+5-FU not superior
0.004
0.000
Equivocal
0.407
0.091
Obs not superior
0.590
0.845
LEV+5-FU superior
0.000
0.064
Source: Spiegelhalter, D. J., Freedman, L. S., Parmar, M. K. B. (1994). Bayesian approaches to randomised trials. J Roy Stat Soc Series A, 157:357–387.
Courtesy of JSTOR.
9
Relationship of Bayesian Inference to Frequentist Inference
 With “noninformative” prior, Bayesian and frequentist inference pretty similar
 Interim analyses not problematic for Bayesian (no adjustment needed)
 Including data from multiple sources (e.g., other studies) straightforward for Bayesian
10
Adaptive Design Definition From FDA Draft Guidance
“Adaptive design clinical study is defined as a study that includes a
prospectively planned opportunity for modification of one or more
specified aspects of the study design and hypotheses based on analysis of
data (usually interim data) from subjects in the study.”
Source: FDA. (February 2010). Guidance for Industry: Adaptive Design Clinical Trials for Drugs and Biologics. Available at:
http://www.fda.gov/downloads/Drugs/.../Guidances/ucm201790.pdf. Accessed September 1, 2015.
11
Why Might We Consider Adaptive and Bayesian Clinical
Trials for Patient-Centered Outcomes Research?
 Adaptive and Bayesian clinical trial designs allow the trial to change in response to
accumulating data
 Less effective treatment arms may drop from consideration
 Enrollment criteria may change to enrich the study population for patients deriving
benefit and to stop enrolling patients who may not be benefitting or who may be
suffering harm
• Can lead to providing randomized evidence to support individualizing therapies
12
Types of Adaptive Designs
 Group sequential
 Allows for early stopping
 Sample size re-estimation
 Increase sample size to achieve significance
 Response adaptive
 Randomization probabilities change over time to favor better treatments
13
Adaptive Designs
 Early adaptive designs:
 Sequential designs
• Sequential medical trials (Armitage, 1960)
 Frequentist properties (control error probabilities)
• Wrong design considerations
 Error probabilities rather than likelihood principle (Anscombe, 1963)
 Other examples:
 Play-the-winner rules
Sources:
Armitage, P. (1960). Sequential Medical Trials. Springfield, Ill.: Thomas.
Anscombe, F. J. (1963). Sequential medical trials (Com: p384-387). Journal of the American Statistical Association, 58:365–383.
14
Adaptive Design Controversy: ECMO Example
 Michigan study favored extracorporeal membrane oxygenation (ECMO)
 Urn scheme randomization
 Lack of firm credibility
• Nine babies received ECMO and lived
• One baby received non-ECMO and died
Source: Bartlett, R. H., Roloff, D. W., Cornell, R. G., Andrews, A. F., Dillon, P. W., and Zwischenberger, J. B. (1985). Extracorporeal circulation in neonatal
respiratory failure: a prospective randomized study. Pediatrics, 76, 479-487.
15
Adaptive Design Controversy: ECMO Example
 Michigan study favored extracorporeal membrane oxygenation (ECMO)
 Urn scheme randomization
 Lack of firm credibility
• Nine babies received ECMO and lived
• One baby received non-ECMO and died
 Ware’s compromise design (Boston Children’s Hospital)
 1:1 randomization until four deaths
 Then all get other treatment until four deaths
• Favored ECMO
Source: O'Rourke, P. P., Crone, R. K., Vacanti, J. P., et al. (1989). Extracorporeal membrane oxygenation and conventional medical therapy in neonates with
persistent pulmonary hypertension of the newborn: a prospective randomized study. Pediatrics, 84, 957-963.
16
Adaptive Design Controversy: ECMO Example
 Michigan study favored extracorporeal membrane oxygenation (ECMO)
 Urn scheme randomization
 Lack of firm credibility
• Nine babies received ECMO and lived
• One baby received non-ECMO and died
 Ware’s compromise design (Boston Children’s Hospital)
 1:1 randomization until four deaths
 Then all get other treatment until four deaths
• Favored ECMO
 UK study
 1:1 randomization until enrolled target sample size
• Favored ECMO
Source: UK Collaborative ECMO Trial Group. (1996). UK collaborative randomised trial of neonatal extracorporeal membrane oxygenation. The Lancet, 348, 75-82.
17
Summary
 We discussed Bayesian inference applied to a randomized clinical trial
 We reviewed an adaptive trial that did not convince the scientific community
 Methodology standards will help ensure a valid and convincing adaptive and
Bayesian randomized clinical trial
18