The Incorporation of Meta-Analysis
Results into Evidence-Based
Decision Modelling
Nicola Cooper, Alex Sutton,
Keith Abrams, Paul Lambert, David Jones
Department of Epidemiology & Public Health,
University of Leicester.
CHEBS, Multi-Parameter Evidence Synthesis Workshop,
Sheffield, March 2002
Where we fit in with Tony’s intro
• Process
–  Model relationship between evidence & parameters
– Consistency check
• Uncertainty Panacea
–
–
–
–
 Statistical error
½ Evidence relates to parameters indirectly
Systematic errors
Data quality, publication bias, etc
METHODOLOGIC PRINCIPLE
1) Pooled estimates
Chan
Nabholtz
Sjostrom
Bonneterre
Combined
.1
.25
1
Odds - log scale
5
METHODOLOGIC PRINCIPLE
1) Pooled estimates
2) Distribution
Chan
mu.rsprtD sample: 12001
2.0
1.5
1.0
0.5
0.0
Nabholtz
Sjostrom
Bonneterre
-5.0
Combined
.1
.25
1
Odds - log scale
5
0.0
5.0
METHODOLOGIC PRINCIPLE
1) Pooled estimates
2) Distribution
Chan
mu.rsprtD sample: 12001
2.0
1.5
1.0
0.5
0.0
Nabholtz
Sjostrom
Bonneterre
-5.0
Combined
.1
.25
1
Odds - log scale
5
3) Transformation of
distribution to transition
probability (if required)
(i) time variables:
 ln[ 1  P(t 0, tj )] 

1  exp 
j


(ii) prob. variables:
1  [1  P(to, tj )]1 / j
0.0
5.0
METHODOLOGIC PRINCIPLE
1) Pooled estimates
2) Distribution
Chan
mu.rsprtD sample: 12001
2.0
1.5
1.0
0.5
0.0
Nabholtz
Sjostrom
Bonneterre
-5.0
Combined
.1
.25
1
Odds - log scale
3) Transformation of
distribution to transition
probability (if required)
(i) time variables:
 ln[ 1  P(t 0, tj )] 

1  exp 
j


(ii) prob. variables:
0.0
5.0
5
4) Application to model
Respond
Stable
Progressive
1  [1  P(to, tj )]1 / j
Death
EXAMPLES
1) Net Clinical Benefit Approach
• Warfarin use for atrial fibrillation
2) Simple Economic Decision Model
• Prophylactic antibiotic use in
caesarean section
3) Markov Economic Decision Model
• Taxane use in advanced breast cancer
MODELLING ISSUES COMMON TO
ALL EXAMPLES
• Bayesian methods implemented using Markov Chain
Monte Carlo simulation within WinBUGS software
• Random effect meta-analysis models used throughout
• All prior distributions intended to be ‘vague’ unless
otherwise indicated
• Where uncertainty exists in the value of parameters
(i.e. most of them!) they are treated as random
variables
• All analyses (decision model and subsidiary analyses)
implemented in one cohesive program
EXAMPLE 1: NET (CLINICAL) BENEFIT
Benefit
Harm
Threshold
Excess absolute risk (harm)
Reduction in absolute risk (benefit)
Net Benefit = (Risk level x Risk reduction) – Harm
Risk
•
Glasziou, P. P. and Irwig, L. M. An evidence based approach to
individualizing treatment. Br.Med.J. 1995; 311:1356-1359.
RE-ANALYSIS OF WARFARIN FOR NONRHEUMATIC ATRIAL FIBRILLATION
• Evidence that post MI, the risk of a stroke is
reduced in patients with atrial fibrillation by
taking warfarin
• However, there is a risk of a fatal
hemorrhage as a result of taking warfarin
• For whom do the benefits outweigh the
risks?
METHOD OUTLINE
1) Perform a meta-analysis of the RCTs to estimate
the relative risk for benefit of the intervention
2) Use this to check the assumption that RR does
not vary with patient risk
3) Check harm (adverse events) is constant across
levels of risk (use RCTs and/or data from other
sources) & estimate this risk
4) Place benefit & harm on same scale (assessment
of QoL following different events)
5) Apply model - need to predict patients risk
(identify risk factors and construct multivariate
risk prediction equations)
SOURCES OF EVIDENCE
Multivariate risk
equations
Net Benefit
M-A of RCTs
=
(risk of stroke x relative reduction in risk of stroke)
(risk of fatal bleed x outcome ratio)
M-A of RCTs/obs
studies
QoL study
0
2
4
6
8
10
Excess absolute risk of intracranial haemorrhage (%/year)
10
8
Embolic stroke
2
4
6
Intracranial haemorrhage
0
Reduction in absolute risk of embolic stroke (%/year)
Singer,D.E. Overview of the randomized trials to prevent stroke in atrial
fibrillation. Ann Epidemiol 1993;3:567-7.
4
6
8
10
Risk of embolic stroke in control arm (%/year)
12
EVALUATING THE TRADE-OFF BETWEEN STROKE
AND HEMORRHAGE EVENTS IN TERMS OF QOL
• QoL following a fatal bleed = 0
• Data available on QoL of patients following
stroke
Proportion
with index
greater
than
horizontal
axis value
Time trade-off index
– Glasziou, P. P., Bromwich, S., and Simes, R. J. Quality of life six months
after myocardial infarction treated with thrombolytic therapy. The Medical
Journal of Australia. 1994; 161532-536
-2.70
-2.65
6
4
2
-1.5
-1.0
-0.5
0.0
0.5
1.0
reduction in relative risk
0
50
100
150
200
250
300
Multivariate risk
equations
0.002
0.004
0.006
0.008
risk of bleed per year
Meta-analyses
of RCTs
(risk of stroke  relative reduction in risk of stroke)
(risk of fatal bleed  outcome ratio)
=
Meta-Analysis of
Net Benefit
0.010
0.012
0.014
Net Benefit (measured in stroke equivalents)
Mean
(s.e.)
Median
(95%
CrI)
Probability of
Benefit > 0
-0.0004
(0.15)
0.06
(-0.29 to
0.20)
54.2 %
Simulated PDF
2
17.6 (10.5
to 29.9)
1
12
0
68
3
4
5
6
Multivariate Risk Equation Data
No.
Thrombo Clinical No. of
% of
embolism
risk
patients cohort
rate (%
factors
per year
(95% CI))
2 or 3
0
20
40
60
80
100
Outcome ratio
QoL study
RCTs/obs studies
Risk of fatal
bleed per
year taking
warfarin:
0.52% (0.27
to 0.84)
0.4
-2.75
0.3
-2.80
0.2
-2.85
0.1
-2.90
Relative risk
reduction for
strokes taking
warfarin (1-RR):
0.23 (0.13 to 0.41)
0.0
-2.95
Risk of stroke
per year e.g.
for 1 or 2 clinical
risk factors:
6.0% (4.1 to
8.8)
0
0
2
4
6
8
10
EVALUATION OF NET BENEFIT
-0.8
-0.6
-0.4
2 or 3 Clinical factors
-0.2
0.0
0.2
0.4
Outcome ratio (1/QoL
reduction) Median
3.75 (1.07 to 50),
Mean 26.14,indicating
the number of strokes
that are equivalent to
one death
Net Benefit (Stroke Equivalents)
1.0
0.8
Probability of benefit > 0.95
0.6
0.4
0 1 or 2
>=3 No. of combined risk factors
0.2
0.0
-0.2
-0.4
Mean
Median
-0.6
0
1
2 or 3
No. of clinical risk factors
-0.8
-1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Risk of Stroke (Rate % per year)
1.0
“TAKE-HOME” POINTS 1
Net-benefit provides a transparent quantitative
framework to weigh up benefits and harms of an
intervention
Utilises results from two meta-analyses and allows
for correlation induced where studies included in both
benefit and harm meta-analyses
Credible interval for net benefit can be constructed
allowing for uncertainty in all model parameters


EXAMPLE 2: SIMPLE DECISION TREE
Use of prophylactic antibiotics to prevent
wound infection following caesarean section
No infection (1-p2)
Cost with antibiotics
Yes
Infection (p2)
Cost with antibiotics + Cost of treatment
Prophylactic antibiotics?
No infection (1-p1)
Cost with no antibiotics
No
Infection (p1)
Cost of treatment
METHOD OUTLINE
1) Cochrane review of 61 RCTs evaluating prophylactic
antibiotics use for caesarean section
2) Event data rare: use “Exact” model for RR
3) Meta-regression: Does treatment effect vary with patients’
underlying risk (pc)?
ln(RRadjusted ) = ln(RRaverage)+  [ln(pc) - mean(ln(pc))]
4) Risk of infection without treatment from ‘local’ hospital
data (p1)
5) Derive relative risk of treatment effect for ‘local’ hospital
(using regression equation with pc=p1)
6) Derive risk of infection if antibiotics introduced to ‘local’
hospital (p2)
p2 = p1 * RRadjusted
UNDERLYING BASELINE RISK
ln(relative risk)
fit
=0.24
(-0.28 to 0.81)
2
1.5
1
ln(Relative Risk)
.5
No treatment effect
0
-.5
-1
-1.5
-2
-2.5
-3
-2.5
-2
-1.5
-1
-.5
0
.5
ln(control group risk) centred on mean)
1
1.5
Local hospital event rate
RESULTS
Mean (95%
Credible
Interval)
Relative Risk, RRadjusted
0.30
(0.21 to 0.40)
Posterior distribution
rr[1] sample: 20000
10.0
7.5
5.0
2.5
0.0
0.1
Prob(wound
infection/placebo), p1
0.08
(0.06 to 0.10)
0.2
0.3
0.4
0.5
p1 sample: 20000
60.0
40.0
20.0
0.0
0.025
Prob(wound
infection/antibiotics), p2
0.02
(0.015 to 0.034)
0.075
0.1 0.125
p2[1] sample: 20000
100.0
75.0
50.0
25.0
0.0
0.0
0.02
0.04
RESULTS
RESULTS
(cont.)
Mean (95%
Credible
Interval)
Reduction in cost using
antibiotics
-£49.53
(-£77.09 to
-£26.79)
Posterior distribution
diff[1] sample: 20000
0.04
0.03
0.02
0.01
0.0
-150.0
Number of wound
infections avoided using
antibiotics per 1,000
Between study variance
(random effect in M-A), 2
53.09
(42.12 to
73.37)
0.30
(0.05 to 0.74)
-100.0
-50.0
nw dreduct[1] sample: 20000
0.06
0.04
0.02
0.0
20.0
40.0
60.0
80.0
tau.squared[1] sample: 20000
3.0
2.0
1.0
0.0
0.0
0.5
1.0
1.5
COST-EFFECTIVENESS PLANE
Control
dominates
Treatment
more effective
& more costly
20
Number of wound infections avoided per 1,000 caesarean sections
0
-20
0
20
40
60
80
-20
Treatment
dominates
-40
-60
-80
-100
-120
-140
Cost difference
Treatment
less effective
& less costly
100
SENSITIVTY OF PRIORS
ca terpillar plot
[1] Gamma(0.001,0.001)
[1]
on 2
[2] Normal(0,1.0-6)
truncated at zero on 
[2]
[3] Uniform(0,20) on 
[3]
-80.0
-60.0
-40.0
Cost difference
-20.0
“TAKE-HOME” POINTS 2
Incorporates M-A into a decision model
adjusting for a differential treatment effect with
changes in baseline risk
Meta-regression model takes into account the
fact that covariate is part of the definition of
outcome
Rare event data modelled ‘exactly’ (i.e.
removes the need for continuity corrections) &
asymmetry in posterior distribution propogated
Sensitivity of overall results to prior
distribution placed on the random effect term in



EXAMPLE 3: USE OF TAXANES FOR 2ND LINE
TREATMENT OF BREAST CANCER
Stages 1 & 2
In 2nd line
treatment
(cycles 1 to 3)
Stage 3
Respond
Stable
Progressive
Dead
Progressive
Dead
Treatment
cycles
(cycles 4 to 7)
Stage 4
Respond
Stable
(cycles 8 to 35)
Post Treatment
cycles
Respond
Stable
Progressive
Dead
METHOD OUTLINE
1) Define structure of Markov model
2) Identify evidence used to inform each model
parameter using meta-analysis where multiple sources
available
3) Transform meta-analysis results, where necessary,
into format required for model (e.g. rates into
transition probabilities)
4) Informative prior distributions derived from elicited
prior beliefs from clinicians
5) Evaluate Markov model
META-ANALYSES
Progression-free time
Time to response from stable
Time to progressive from response
Overall survival time
No. of
studies
3
1
1
3
Response rate
% moving directly to progressive at stage 2.
% with infections / febrile neutropenia
% hospitalised with infection / febrile neutropenia
% dying from infections / febrile neutropenia
% discontinue treatment due to adverse event
% with Neutropenia grades 3 & 4
% with Anaemia grades 3 & 4
% with Diarrhoea grades 3 & 4
% with Stomatis grades 3 & 4
% with vomiting grades 3 & 4
% with fluid retention grades 3 & 4
% with cardiac toxicity grades 3 & 4
4
1
3
1
1
3
2
2
3
3
2
3
1
Time in weeks
(95% Credible Interval)
25 (15 to 24)
12 (6 to 18)
35 (29 to 41)
53 (35 to 74)
Probabilities
0.43 (0.29 to 0.58)
0.13 (0.08 to 0.18)
0.18 (0.04 to 0.56)
0.08 (0.05 to 0.11)
0.01 (0.00 to 0.02)
0.16 (0.03 to 0.49)
0.94 (0.82 to 0.98)
0.03 (0.00 to 0.28)
0.09 (0.06 to 0.14)
0.08 (0.04 to 0.14)
0.03 (0.00 to 0.12)
0.05 (0.02 to 0.12)
0.00 (0.00 to 0.02)
TRANSITION PROBABILITIES
Transition Probabilities
(95% Credible Interval)
Infection/FN
Hospitalised due to infection/FN
Dying from infection/FN after hospitalisation
Discontinuation due to major adverse events
Adverse events – Neutropenia
0.09 (0.02 to 0.32)
0.04 (0.03 to 0.05)
0.00 (0.00 to 0.01)
0.04 (0.04 to 0.16)
0.50 (0.34 to 0.63)
Adverse events – Anaemia
0.01 (0.00 to 0.07)
Adverse events – Diarrhoea
0.02 (0.01 to 0.37)
Adverse events – Stomatis
0.02 (0.01 to 0.04)
Adverse events – Vomiting
0.01 (0.00 to 0.03)
Adverse events – Fluid retention
0.01 (0.00 to 0.03)
Adverse events – Cardiac toxicity
Transition directly to ‘progressive’ state
0.00 (0.00 to 0.01)
0.12 (0.08 to 0.18)
Transition ‘stable’ to ‘stable’
0.65 (0.44 to 0.75)
Transition ‘stable’ to ‘response’
0.16 (0.11 to 0.28)
Transition ‘stable’ to ‘progressive’
0.18 (0.11 to 0.37)
Transition ‘response’ to ‘response’
0.94 (0.93 to 0.95)
Transition ‘response’ to ‘progressive’
0.06 (0.05 to 0.07)
Transition ‘progressive’ to ‘progressive’
0.93 (0.79 to 0.96)
Transition ‘progressive’ to ‘death’
0.07 (0.04 to 0.21)
METHODOLOGIC PRINCIPLE
1) Pooled estimates
2) Distribution
Chan
mu.rsprtD sample: 12001
2.0
1.5
1.0
0.5
0.0
Nabholtz
Sjostrom
Bonneterre
-5.0
Combined
.1
.25
1
Odds - log scale
3) Transformation of
distribution to transition
probability (if required)
(i) time variables:
 ln[ 1  P(t 0, tj )] 

1  exp 
j


(ii) prob. variables:
0.0
5.0
5
4) Application to model
Respond
Stable
Progressive
1  [1  P(to, tj )]1 / j
Death
ELICITATION OF PRIORS
e.g. Response Rate
Taxane
0%
5%
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
40%
45%
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
40%
45%
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
x
x
x
x
x
10%
15%
20%
25%
30%
35%
x
x
x
x
x
x
x
x
x
x
x
x
x
x
20%
25%
30%
35%
Standard
0%
x
x
x
x
x
x
5%
10%
15%
RESPONSE RATE
80
50
40
60
30
40
20
20
10
Std. Dev = .87
Std. Dev = 17.28
Mean = -.6
Mean = 38
3.5
3.0
2.5
2.0
1.5
.5
1.0
-.0
-.5
-1.0
-1.5
-2.0
-2.5
-3.0
95
85
75
65
55
45
35
25
5
15
N = 280.00
-3.5
N = 280.00
0
0
logit (Response rate for docetaxel)
Response rate (docetaxel)
50
100
40
80
30
60
20
40
20
10
Std. Dev = .92
Std. Dev = 14.75
Mean = -1.0
Mean = 31
N = 300.00
logit (Response rate for doxorubicin)
2.5
1.5
.5
-.5
-1.5
-2.5
-3.5
-4.5
-5.5
-6.5
85
75
65
55
45
35
25
15
5
Response rate for doxorubicin
-7.5
N = 300.00
0
0
COST-EFFECTIVENESS PLANE
Bayesian (MCMC) Simulations
£10,000
£8,000
Incremental cost
£6,000
Doxorubicin
dominates
Docetaxel more
effective but
more costly
£4,000
£2,000
-0.50
-0.40
-0.30
-0.20
Docetaxel less
costly but less
effective
£0
-0.10
0.00
-£2,000
0.10
-£4,000
Incremental utility
0.20
0.30
0.40
Docetaxel
dominates
0.50
“TAKE-HOME” POINTS 3
Synthesis of evidence, transformation of
variables & evaluation of a complex Markov
model carried out in a unified framework
(facilitating sensitivity analysis)
Provides a framework to incorporate prior
beliefs of experts

FURTHER ISSUES
• Handling indirect comparisons correctly
•E.g. Want to compare A v C but evidence only
available on A v B & B v C etc.
•Avoid breaking randomisation
• Necessary complexity of model?
•When to use approaches 1,2,3 above?
• Use of predictive distributions
•Necessary when inferences made at ‘unit’ level
(e.g. hospital in 2nd example) rather than
‘population’ level?
• Incorporation of EVI
MODEL SPECIFICATION
Bayesian random effects M-A model specification: ln(RR)
ric ~ Binomial ( nic , pic )
rit ~ Binomial ( nit , pit )
 i  log( pic )
i  1,.....,61
log( pit )   i  min(  i , log( pic ))
 i ~ Normal ( Δ, 2 )
RRantibiotics  exp( Δ)
Prior distributions:
pic ~ Beta( ,  )
 ~ Uniform(1,100)
Δ ~ Normal (0,0.1)
 ~ Uniform(1,100)
τ 2 ~ InverseGam ma (0.001,0.001)
Warn et al 2002 Stats in Med (in press)