Use of “conditional assurance” to evaluate futility criteria and
event targets in cardiovascular outcomes trials:
An example using elicited priors
Adam Crisp, GSK
PSI Conference,
May 2017
1
Background
• Routine at GSK to calculate “assurance” at key development
milestones
• Assurance
– PoS, weighted over a prior distribution
– If “success” = sig p-value, Assurance = “expected power”
– Success definition can be generalised in terms of target effect
• Benefits
–
–
–
–
Relationship between PoS and sample-size
Potential importance of futility interims
Transparency of prior assumptions
Support investment decisions
• Focus for today
– Outcomes trial with a futility interim – insight to cost/risk architecture
2
Scenario: Cardiovascular Outcomes Trial
Primary Endpoint: “MACE”
Primary objective:
HR 95%CI < 1.3
Base case:
Fixed event target (E=611,
powered for non-inf)
Option: continue to ≥ 1000 events if superiority not
futile
Concern: high risk and cost of progressing to failed
superiority
Target: characterise options in terms of PoS and risk
discharge
 Group-sequential theory
Enablers:
 Use of elicited priors
 R code for calculating multi-level integrals
3
Overview of decision-making framework
First analysis at
611 events
MACE
superiority
futile*?
Yes
No
Submit for
non-inferiority
Continue to
1000 events
*Futile: interim benefit <10%
MACE
non-inferior/
superior?
4
Evaluating Design Performance
 Power and Minimum Detectable Effect
 Operating characteristics of futility rule conditional on true
effect
• But there are gaps…
Overall PoS / Assurance
Useful?
P(Go at Interim)
Very useful
(likelihood of additional spend
being incurred)
P(Success | Go at Interim)
“Conditional Assurance”
All require a
Prior
Very useful
(how much risk does interim
discharge)
5
Technical Foundations
• Ei events at each analysis
• θTrue : true log HR, 𝜃𝑖 estimated log HR at each analysis
• Go at interim if 𝜃1 < 𝜃𝐺𝑜 [e.g. 𝜃𝐺𝑜 = log(0.9), at least 10% benefit]
• Let 𝑍𝑖 = 𝜃 𝑖 / 4/𝐸𝑖 and 𝑍𝐺𝑜 = 𝜃𝐺𝑜 / 4/𝐸𝑖
• Joint distribution of Z = (Z1, Z2) is bivariate normal#
• MACE superiority if Go at interim and p < 0.05 at end
zgo  z / 2
P(MACE superiority | True) 
f ( z1, z 2 | True) dz
 


Calculation: R function pmvnorm ideal, as can
handle arbitrary number of interims
#Details
in back-up slide
6
Assurance: “unconditional” PoS
• Prior π(θ) for θ=log(HRTrue) and “success” = sig p-value
zgo  z / 2
Assurance    ( )   f ( z1, z 2 |  ) dz d




Direct numerical integration possible via combination of R
functions, e.g.
– integrate / Vectorize / pmvnorm
– integrate / sapply / pmvnorm
7
Combining integrate with pmvnorm
• Problem:
zgo  z / 2
Assurance    ( )   f ( z1, z 2 |  ) dz d




pmvnorm: mean is a 2x1 vector
dependent on true HR
• Solution:
integrate: will fire random-length
vectors of true HR values at its target
function: by default this will break
pmvnorm
– Need pmvnorm to accept vector-based input of arbitrary length
– Example R functions: Vectorize and sapply
8
Using integrate, Vectorize and pmvnorm
define the integrand for the assurance calculation:
predprob <- function(theta){
density for prior: - can be any elicited distribution represented as R function
px <- dnorm(theta,mean=…,sd=…)
power for given true effect:
pwr <- pmvnorm(lower=…,upper=…,mean=theta*sqrt(e/4),sigma=sig)
overall value of the integrand:
rslt <- px*pwr
return(rslt)
}
assurance <- integrate(Vectorize(predprob),-Inf,Inf)$val
Use of “sapply” is more elegant and defensive coding, but more involved to write
9
Prior Elicitation example
10
Mapping elicited HR priors to log HR priors
• If we have an elicited prior for the HR itself, π(HR)
then prior for θ=log(HR) is π(eθ) x eθ
Jacobian of the
transformation
11
Completing the design assessment
Overall PoS/Assurance?
Integrate group-sequential
likelihood over the prior
Probability of incurring
post-interim spend?
i.e. P(Go)?
Integrate P(Go | θ)
over the prior
How much risk discharged by
interim?
i.e. P(Success | Go at Interim)?
P(Go and Success) / P(Go)
= Assurance / P(Go)
“Conditional Assurance”
12
Illustrative cost/risk profile for design
611 MACE
Base case
1000 MACE with
interim at 611
1300 MACE
with interim at 611
Alternate 1
Alternate 2
Minimum detectable RRR for
superiority (p<0.05)
15%
12%
10%
Overall PoS for superiority
31%
37%
40%
Probability of ‘go’ at interim#
-
50%
50%
Conditional PoS/Assurance
-
75%
80%
£M1
£M2
£M3
-
£M1+
£M1+
Total cost (£m)
Cost if ‘stop’ at interim (£m)
Insight to risk discharged by futility interim played
important role in discussions with key stakeholders
#assuming guidance
to stop if interim HR > 0.9 (i.e. <10% benefit)
13
Remarks
• For sequential designs, Assurance is a “means to an end”
– May indicate or support need for a futility interim but will not tell us
what the value of the futility interim is
• “Conditional Assurance” – value of futility interim
– Converts a sequential design into a series of “bets”
– PoS for each “bet” in the sequence determined by Prior and Design
– Overall cost/risk architecture of design can then be assessed
• Prior Elicitation an invaluable trial-design tool
– Feedback from experts generally very positive
– Makes assumptions more transparent than a “black-box” PoS exercise
14
Back-ups
15
Syntax of pmvnorm function
• R package mvtnorm includes the pmvnorm function
– integrates a multivariate normal distribution over an arbitrary number of
dimensions
• Syntax:
pmvnorm(lower=c(l1,l2,l3,….),upper=c(u1,u2,u3,…),mean=…,sigma=…)
lower integration
limits
upper integration
limits
• Infinite limits may be specified using key words –Inf or Inf
16
Details of bivariate-normal distribution
•
•
•
•
Interim analysis after E1 events, final analysis at E2 events
θTrue : true log HR
Zi – standardised Z value at ith analysis
Joint distribution of Z = (Z1, Z2)
 E1 / 4 

1


Z ~ N 

True, 

 E1 / E 2
 E 2 / 4 

E1 / E 2  

1 
17
R Integrate function
• Adaptive quadrature of functions of one variable over a finite
or infinite interval
– calculates area over adaptively redefined subintervals of the
integration domain
• Syntax: integrate(f, lower, upper,…)
• Arguments:
–
–
–
–
f is an R function (may be user-defined) taking a numeric argument
the target function must accept vector-based input
lower and upper define integration limits
other arguments available for fine control
18
Using integrate, sapply and pmvnorm
predprob <- function(theta){
map incoming vector to individual elements “y”
sapply(theta,function(y){
go on to define the integrand:
px <- dnorm(y,mean=…,sd=…)
pwr <- pmvnorm(lower=…,upper=…,mean=y*sqrt(e/4),sigma=…)
rslt <- px*pwr
return(rslt)
})
}
assurance <- integrate(predprob,-Inf,Inf)$val
19
20