A Bayesian single-arm study design
accounting for uncertain historical
control rates
Simon Wandel, Expert Stat Methodologist
Novartis Pharma AG, Basel
PSI conference 2017
• Co-authors
– Qing Liu, Novartis
– Steven Green, Novartis
– Beat Neuenschwander, Novartis
• Acknowledgements
– Satrajit Roychoudhury, Novartis
Global Drug Development
PSI 2017 - Simon Wandel
Agenda
General background and problem statement
Accounting for uncertainty, bias: meta-analytic approach
Case study in pediatric high-grade glioma (HGG)
Conclusions
Global Drug Development
PSI 2017 - Simon Wandel
General background and
problem statement
Background
• Single-arm studies
– frequently used in rare diseases, particularly in Oncology
– Monzon et al. (2015): 80% of ph II trials in neoplasms are single-arm
– Swallow et al. (2015)
– 44% of EMA Oncology approvals in the last decade based on single-arm trials
– > 50% of FDA accelerated approvals based on single-arm trials
• Challenges
– between-study heterogeneity
– potential bias, e.g. selection bias
Global Drug Development
PSI 2017 - Simon Wandel
Single-arm studies matter
• Example 1: ceritinib in ALK-positive non-small cell lung
cancer
– FDA approval (2014) based on single-arm study
The safety and efficacy of ceritinib is primarily based on data from Study
CLDK378X2101... The efficacy and safety of ceritinib was established in an
open-label, single-arm study enrolling 163 patients... (FDA Summary Review)
• Example 2: pembrolizumab for unresectable / metastatic
melanoma previouslty treatead with ipilimumab
– FDA approval (2014) based on study cohort without control
The biologics license application ... relies on the results of a single,
randomized (1:1), open-label, dose-ranging, multicenter cohort (Cohort B2)...
Among the 173 patients ... treated in this sub-study (Cohort B2), 89 received
pembrolizumab 2 mg/kg every 3 weeks... and 84 received pembrolizumab 10
mg/kg Q3W. (FDA Summary Review)
Global Drug Development
PSI 2017 - Simon Wandel
Common approach to single arm
studies
• Often a binary endpoint, i.e. binomial model
𝑟𝑇 ~𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙(𝑟𝑇 , 𝑝𝑇 )
• Defining a historical control rate 𝑝𝐶
• Goal: designing a study with reasonable power to show
that 𝑝𝑇 > 𝑝𝐶 under an assumed alternative response
rate 𝑝𝑇,𝐴
• Implementation: metric for success based on
– p-value (Frequentist)
– posterior probability (Bayesian)
Global Drug Development
PSI 2017 - Simon Wandel
Common approach for single arm
studies
• 𝑝𝐶 is a critical design assumption
– usually assumed fixed
– based on (some) historical evidence, not necessarily on a
comprehensive, systematic review
• This approach ignores uncertainty, in particular
– between-study heterogeneity
– bias
• (How) can we do better?
Global Drug Development
PSI 2017 - Simon Wandel
Accounting for uncertainty,
bias: meta-analytic approach
Setup for meta-analytic framework
• Assume
– data on the control treatment from j = 1, … , 𝐽 historical studies
– data on the test treatment from the new study j = ∗
• Work with the following data structure
Study (j)
𝒓𝒋,𝑪 / 𝒏𝒋,𝑪
𝒓𝒋,𝑻 / 𝒏𝒋,𝑻
1
... / ...
NA
2
... / ...
NA
...
... / ...
NA
J
... / ...
NA
*
NA
... / ...
Global Drug Development
PSI 2017 - Simon Wandel
Meta-analytic framework
• Hierarchical (meta-analytic) model
𝒓𝟏,𝑪 , 𝒏𝟏,𝑪
𝜽𝟏,𝑪
𝒓𝟐,𝑪 , 𝒏𝟐,𝑪
𝜽𝟐,𝑪
𝒓𝑱,𝑪 , 𝒏𝑱,𝑪
𝜽𝑱,𝑪
?
– data (sampling) model: 𝑟𝑗,𝐶 ~𝐵𝑖𝑛 𝑛𝑗,𝐶 , 𝑝𝑗,𝐶
– parameter model:
𝜃𝑗,𝐶 ~𝑁 𝜇, 𝜏 2
with 𝜃𝑗,𝐶 = 𝑙𝑜𝑔𝑖𝑡(𝑝𝑗,𝐶 )
𝜃∗,𝐶 ~𝑁 𝜇, 𝜏 2 , with 𝜃𝑗,𝐶 = 𝑙𝑜𝑔𝑖𝑡(𝑝𝑗,𝐶 )
– prior distributions: for 𝜇, 𝜏
Global Drug Development
PSI 2017 - Simon Wandel
Meta-analytic framework
• Hierarchical (meta-analytic) model
𝒓𝟏,𝑪 , 𝒏𝟏,𝑪
𝜽𝟏,𝑪
𝒓𝟐,𝑪 , 𝒏𝟐,𝑪
𝜽𝟐,𝑪
𝒓𝑱,𝑪 , 𝒏𝑱,𝑪
𝜽𝑱,𝑪
𝝁, 𝝉
– data (sampling) model: 𝑟𝑗,𝐶 ~𝐵𝑖𝑛 𝑛𝑗,𝐶 , 𝑝𝑗,𝐶
– parameter model:
𝜃𝑗,𝐶 ~𝑁 𝜇, 𝜏 2
with 𝜃𝑗,𝐶 = 𝑙𝑜𝑔𝑖𝑡(𝑝𝑗,𝐶 )
𝜃∗,𝐶 ~𝑁 𝜇, 𝜏 2 , with 𝜃𝑗,𝐶 = 𝑙𝑜𝑔𝑖𝑡(𝑝𝑗,𝐶 )
– prior distributions: for 𝜇, 𝜏
Global Drug Development
PSI 2017 - Simon Wandel
Meta-analytic framework
• Hierarchical (meta-analytic) model
𝒓𝟏,𝑪 , 𝒏𝟏,𝑪
𝜽𝟏,𝑪
𝒓𝟐,𝑪 , 𝒏𝟐,𝑪
𝜽𝟐,𝑪
𝝁, 𝝉
𝒓𝑱,𝑪 , 𝒏𝑱,𝑪
𝒓∗,𝑪 , 𝒏∗,𝑪
𝜽𝑱,𝑪
𝜽∗,𝑪
– data (sampling) model: 𝑟𝑗,𝐶 ~𝐵𝑖𝑛 𝑛𝑗,𝐶 , 𝑝𝑗,𝐶
𝜃𝑗,𝐶 ~𝑁 𝜇, 𝜏 2
with 𝜃𝑗,𝐶 = 𝑙𝑜𝑔𝑖𝑡(𝑝𝑗,𝐶 )
𝜃∗,𝐶 ~𝑁 𝜇, 𝜏 2
– prior distributions: for 𝜇, 𝜏
with 𝜃∗,𝐶 = 𝑙𝑜𝑔𝑖𝑡(𝑝∗,𝐶 )
– parameter model:
Global Drug Development
PSI 2017 - Simon Wandel
Meta-analytic framework
• The hierarchical model
– allows a comparison between treatment and control for the actual study
based on the predicted control rate 𝜃∗,𝐶
– accounts for between-study heterogeneity 𝜏
– assumes similarity between the historical and actual control rate
• The similarity assumption may be violated (bias)
• A bias model (Pocock 1976) may address this
– data (sampling) model: 𝑟𝑗,𝐶 ~𝐵𝑖𝑛 𝑛𝑗,𝐶 , 𝑝𝑗,𝐶
– parameter model:
𝜃𝑗,𝐶 = 𝜃∗,𝐶 + 𝛿𝑗,𝐶 with 𝜃…,𝐶 = 𝑙𝑜𝑔𝑖𝑡(𝑝…,𝐶 )
𝛿𝑗,𝐶 ~𝑁 0, 𝜏𝛿 2
– prior distributions: for 𝜃∗,𝐶 , 𝜏𝛿
– sensitivity analysis for systematic bias: 𝛿𝑗,𝐶 ~𝑁 𝑚𝛿 , 𝜏𝛿 2
Global Drug Development
PSI 2017 - Simon Wandel
Prior distributions for the metaanalytic model
• Need prior distributions for
– population mean 𝜇
– between-trial heterogeneity 𝜏
• Prior for 𝜇: unproblematic, weakly informative
• Prior for 𝜏: important, scale dependent!
– difficult to infer 𝜏, especially for few studies (Friede et al. 2016, 2017,
Spiegelhalter et al. 2004)
– prior should reflect plausible values
– e.g., on logit-scale 𝜏~HN 0.5 or 𝜏~HN 1
Global Drug Development
PSI 2017 - Simon Wandel
Case-study in pediatric highgrade glioma (HGG)
Pediatric high-grade glioma (HGG)
• Pediatric high grade glioma (Fangusaro et al. 2012,
Ostrom et al. 2015)
– tumor of the central nervous system (CNS)
– characterized by aggressive clinical behavior
– rare disease, estimated yearly incidence of 0.85/100’000
– genetic mutations highly relevant (Jones et al. 2016)
– very limited treatment options
• High need for novel treatments!
Global Drug Development
PSI 2017 - Simon Wandel
Case study
• Phase II study in pediatric HGG
– in a selected population (specific mutation)
– treatment: combination of two novel agents (both unapproved in
pediatric HGG)
– preliminary single-agent data from a phase I study
• Key questions for decision making:
– Is the combination effective compared to standard of care?
– Is the combination potentially better than the single agent?
• Binary endpoint: objective response rate (ORR)
Global Drug Development
PSI 2017 - Simon Wandel
Statistical design specifications
• Meta-analytic model as presented before (no bias)
𝑟1,𝐶 ~𝐵𝑖𝑛 𝑛1,𝐶 , 𝑝1,𝐶
𝑟∗,𝑇 ~𝐵𝑖𝑛 𝑛∗,𝑇 , 𝑝∗,𝑇
𝑙𝑜𝑔𝑖𝑡(𝑝1,𝐶 )~𝑁 µ, τ2
𝑙𝑜𝑔𝑖𝑡(𝑝∗,𝐶 )~𝑁 µ, τ2
• Prior distributions
𝑝∗,𝑇 ~𝐵𝑒𝑡𝑎(2/3,1)
µ ~𝑁(0, 22 )
τ ~𝐻𝑁(0.5)
• Bayesian success criteria
– statistically convincing: 𝑃(𝑝∗,𝑇 > 𝑝∗,𝐶 𝐷𝑎𝑡𝑎 ≥ 0.90
– clinically relevant: posterior median of 𝑝∗,𝑇 at least 0.4
Global Drug Development
PSI 2017 - Simon Wandel
Example scenario
• Historical single-agent data: 6 / 19 (32%)
• Study size: n = 40
• Two analyses
– simple analysis using a fixed historical control rate of 32%
– meta-analytic approach using the historical data
• Assumed data: 17 responders in 40 patients
Analysis
Posterior probability
𝑷 𝒑∗,𝑻 > 𝒑∗,𝑪 𝑫𝒂𝒕𝒂)
Posterior median of
𝒑∗,𝑻
simple
0.71 
0.43 
meta-analytic
0.92 
0.43 
Global Drug Development
PSI 2017 - Simon Wandel
Example scenario
• Why this difference...?
– meta-analytic approach discounts the historical data by ~ 50%
simple analysis
3
0
1
2
Density
4
5
6
meta-analytic approach
0.0
0.2
0.4
0.6
pT
Global Drug Development
PSI 2017 - Simon Wandel
0.8
1.0
The difference can be remarkable
• Simple approach
– minimal response rate for success: 𝑟∗,𝑇 = 17
– maybe 32% is not the «right» threshold; some would argue e.g. for the
upper bound of an 80% CI?
– somewhat arbitrary...
• Meta-analytic approach
– minimal response rate for success: 𝑟∗,𝑇 = 24
• Key challenge: high uncertainty in historical control rate
– usually, much more information for standard of care
– remember: here, we compare against an unapproved single-agent
Global Drug Development
PSI 2017 - Simon Wandel
Operating characteristics
• Operating characteristics (type I error, power)
– can be time-consuming for Bayesian designs involving MCMC sampling
– in situations like the one here, quite simple
• Recipe
– for a given sample size 𝑛∗,𝑇 , find the minimal 𝑟∗,𝑇 𝑐𝑟𝑖𝑡 for success
– then it is a simple binomial calculation: 𝑃(𝑟∗,𝑇 ≥ 𝑟∗,𝑇 𝑐𝑟𝑖𝑡 |𝑝𝐴 )
– Example: for 𝑛∗,𝑇 = 40, 𝑟∗,𝑇 𝑐𝑟𝑖𝑡 = 24
𝒑𝑻
Probability of successful study
0.40
< 1%
0.60
57%
0.65
80%
0.70
94%
Global Drug Development
PSI 2017 - Simon Wandel
Conclusions
Conclusions
• Single-arm studies
– have an important role in rare diseases (especially Oncology)
– should be analyzed accounting for their inherent limitations, ideally in a
formal way
• Bayesian meta-analytic framework
–
–
–
–
between-study heterogeneity
bias
effect of control had control been included (predictive effect)
for main ideas and other applications, see also Neuenschwander et al.
2010, Schmidli et al. 2014
• Other potential approaches
– propensity score matching
– joint analysis accounting for covariates
– natural disease history models
Global Drug Development
PSI 2017 - Simon Wandel
References
• Fangusaro J. Pediatric high grade glioma: a review and update on tumor clinical characteristics and biology.
Front Oncol. 2012
• FDA summary review for ceritinib, available at:
https://www.accessdata.fda.gov/drugsatfda_docs/nda/2014/205755Orig1s000SumR.pdf
• FDA summary review for pembrolizumab, available at:
https://www.accessdata.fda.gov/drugsatfda_docs/nda/2014/125514Orig1s000SumR.pdf
• Friede T, Röver C, Wandel S, Neuenschwander B. Meta-analysis of two studies in the presence of
heterogeneity with applications in rare diseases. Biom J [epub ahead of print]
• Friede T, Röver C, Wandel S, Neuenschwander B. Meta-analysis of few small studies in orphan diseases. Res
Synth Methods 2017
• Jones C, Karajannis MA, Jones DT et al. Pediatric high-grade glioma: biologically and clinically in need of new
thinking. Neuro Oncol. 2017
• Monzon JG, Hay AE, McDonald GT et al. Correlation of single arm versus randomised phase 2 oncology trial
characteristics with phase 3 outcome. Eur J Cancer 2015
• Neuenschwander B, Capkun-Niggli G, Branson M, Spiegelhalter DJ. Summarizing historical information on
controls in clinical trials. Clin Trials. 2010
Global Drug Development
PSI 2017 - Simon Wandel
References
• Ostrom QT, Gittleman H, Fulop J, et al. CBTRUS Statistical Report: Primary Brain and Central Nervous System
Tumors Diagnosed in the United States in 2008-2012. Neuro Oncol. 2015
• Pocock SJ. The combination of randomized and historical controls in clinical trials. Journal of Chronic Diseases.
1976
• Schmidli H, Gsteiger S, Roychoudhury S et al. Robust meta-analytic-predictive priors in clinical trials with
historical control information. Biometrics 2014
• Spiegelhalter DJ, Abrams KR, Myles JP Bayesian Approaches to Clinical Trials and Health-Care Evaluation .
Chichester, UK: Wiley & Sons 2004.
• Swallow E, Signorovitch J, Yuan Y, Kalsekar A. Indirect comparisons for single-arm trials or trials without
common comparator arms. Workshop: ISPOR 20th Annual International Meeting 2015
Global Drug Development
PSI 2017 - Simon Wandel