Minimum run-time requirements to reduce
Monte Carlo error in stochastic simulations
Volker Foos1, Phil McEwan4, Adam Lloyd2, James Palmer1, Mark
Lamotte3 and David Grant2.
1. IMS Health, Basel, Switzerland, 2. IMS Health, London, UK, 3. IMS Health,
Brussels, Belgium, 4. Center for Health Economics, Swansea University, UK
Mount Hood 2012 – Friday June 8, 2012
Background
The key consideration of PSA is to assess outcome variability
that can be attributed to parameter uncertainty
• In HEOR the role of probabilistic sensitivity analysis (PSA) is to
propagate the uncertainty of model predictions with respect to the
underlying parameter uncertainty.
• However, in Monte Carlo simulation parameter uncertainty coincides
with and cannot be distinguished from stochastic uncertainty (1st
order variability/random noise/Monte Carlo error [MCE]).
• The minimum criteria for PSA runs should be therefore to reduce 1st
order variability to such an extent that a meaningful assessment of
parameter uncertainty is possible.
• In Monte Carlo simulation, MCE can be reduced by increasing the
number of patients that are processed through the model.
Objective
The overall objective was to demonstrate the relation of 1st
and 2nd order variability for different degrees of parameter
uncertainty
The overall objective of our investigation was to demonstrate the
relation of 1st and 2nd order output variability for
1.
Increasing number of patients - to identify the number of
patients required to allow a meaningful assessment of parameter
uncertainty while keeping within reasonable levels of computation
time).
2.
Two different degrees by which input model parameters were
sampled using
• Standard deviations
parameter variability
(SD)
representing
a
larger
input
• Standard errors (SE) representing lower input parameter
variability
Materials and Methods
We used a well established and validated diabetes simulation
model to illustrate the degree of MCE vs. parameter uncertainty
• The IMS CORE diabetes model (CDM), was used to compare the
outcome variability for bootstrap simulations with 1,000 repetitions
for patient cohorts ranging from 500 to 100,000.
• Model projections were obtained evaluating the cost effectiveness of
two hypothetical interventions with differences in clinical
effectiveness of 0.5% HbA1c and a 2kg weight change in favor of
the treatment- vs. control arm and a 400 Euro difference in
treatment costs.
• Each simulation was performed in three ways
1. no parameter sampling
2. parameters were sampled around 5% of their means (intending to
represent variability for SE input)
3. parameters were sampled around 25% of their means (intending to
represent variability for SD input)
Materials and Methods
The degree of randomness contained in PSA confidence ranges
can be assessed by comparing confidence ranges of sampled
vs. non sampled analyses
• Output variability was classified as 1st or 2nd order and the amount of
MCE assessed by comparing confidence ranges of selected endpoints
in sampling vs. non sampling analyses
• The variability of outputs that is attributed to parameter sampling
can be assessed by comparing confidence sizes of 1st order
simulations vs. 2nd order simulations.
12
CI without
sampling
11
CI attributable
to parameter
uncertainty
CI with
sampling
cumulative inicidence (%)
10
Deterministic
model
9
=
8
7
}
6
Stochastic
model
5
4
}
3
0
0.5
1
1.5
2
2.5
3
3.5
• While stochastic models
don’t specifically behave in
this predictive way our
intention was to estimate
the run time (number of
patients) required to start
making predictive
assumptions on the
parameter uncertainty in
the hypothetical absence of
1st order variability.
Materials and Methods
The last assessment can be illustrated as follows…
• While confidence ranges decline with increasing number of patients (thereby reducing
MCE), the proportion of the confidence range that is attributable to parameter
uncertainty remains stable.
• Therefore the number of patient required to reduce 1st order variability below overall
outcome variability has to be identified.
• Depending on the selected endpoint, this assessment may be straight forward or not.
d_QA L Y P S A S D
d_QA L Y P S A S E
d_QA L Y no P S A
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0
500
-0.2
1000
2500
5000
10000
25000
50000
100000
-0
500
-0.2
1000
2500
5000
10000
25000
50000
100000
-0.4
-0.4
-0.4
-0.6
-0.6
-0.6
IC UR no P S A
500
-0.2
300000
300000
200000
200000
200000
100000
100000
100000
0
2500
5000
10000
25000
50000 100000
5000
10000
25000
50000
100000
0
0
1000
2500
IC UR P S A S D
IC UR P S A S E
300000
500
1000
500
1000
2500
5000
10000
25000
500
50000 100000
-100000
-100000
-100000
-200000
-200000
-200000
-300000
-300000
-300000
1000
2500
5000
10000
25000
50000 100000
Materials and Methods
We selected two different criteria to determine minimum run
time requirements
1. When stochastic uncertainty (CI of non PSA analysis) consistently
decreased below overall outcome variability (CI of PSA analyses)
2. When stochastic uncertainty (CI of non PSA analysis) decreased below
50% of overall outcome variability (CI of PSA analyses)
•
We focused on confidence ranges of the ICER per QALE gained
•
The overall criteria was that the prediction of stochastic
uncertainty was reliable so the confidence range in non PSA runs
decreased with increasing number of patients.
Results
Based on the 1st criteria, analyses of the confidence ranges
suggested a minimum number of 10,000 patients required to
run over 1000 bootstrap repetitions
• Confidence ranges in non PSA runs demonstrated an irregular pattern
(increased or maintained stable) with increasing patient number until 10,000
included patients.
• A clearer picture could have been obtained by increasing the number of
bootstrap repetitions
Incremental costrs per QALY - Width of confidence ranges in "no PSA",
PSA SE" and "PSA SD" analyses
450000
400000
Patients (N)
350000
300000
Width - No PSA
250000
Width - PSA SE
200000
Width - PSA SD
150000
100000
50000
0
500
1000
2500
5000
10000
Confidence width
25000
50000
100000
• Beyond 10,000 included
patients, both confidence
ranges for SE and SD
based PSA were
consistently lower vs.
those obtained in non
PSA
Results
When minimum run time requirements were defined according to
the 50% threshold of stochastic uncertainty contained in overall
uncertainty the number of required patient runs increased
• The proportion of stochastic uncertainty contained in overall ICER
variability was obtained from the ratio of confidence sizes of the no
PSA vs. PSA runs.
Proportion of MCE contained in overal ICER variability
% of no PSA in PSA SE
% of no PSA in PSA SD
1.2
110.3%
106.8%
96.7%
91.8%
1
• When SE inputs were used
in PSA the minimum run
time requirement increased
to 50,000 patients
86.2%
86.2%
79.6%
% MCE
0.8
79.6%
72.6%
0.6
• Run time requirements For
PSA based on SE input
remained at 10,000
included patients.
53.7%
45.0%
36.6%
0.4
32.2%
0.2
12.8%
9.4%
5.7%
0
500
1000
2500
5000
10000
Patients (N)
25000
50000
100000
• The question remains which
part of the overall
uncertainty is attributable
to 1st order and 2nd order.
A theoretical exercise was conducted to to evaluate the implications
two overlaying distributions (random noise and parameter uncertainty)
on the joint distribution
A ) V ariability due to parameter
s ampling
B) Random nois e
T wo ov erlay ing dis tributions (random nois e and
parameter unc ertainty ) and the joint dis tribution
C ) J oint Dis tribution
12
• A) mean = 50 +/5
S tandard Deviation
10
• B) mean = 0 +/10 to 0.5
8
6
• C) Joint distributio
was calculated as
A+B
15%
50%
4
2
0
89%
81%
71%
67%
63%
57%
52%
45%
37%
De g re e (%) of ra ndom noise within ove ra ll va ria bility
10
• 10000 random
draws were
generrated in MS
Excel with:
29%
20%
10%
• The SD of the joint
distributio was
compared to SD‘s
of distribution B to
assess %
overestimation
The proportion of the confidence range attributable to parameter
uncertainty was estimated from the relationship between 1st and
2nd order outcome variability obtained from this theoretical
exercise
R elations hip between proportions of 1s t order and
2nd order variability from overall outc ome variability
P erc ent 1s t order variability from overall outc om e
variability
120.00%
y = -3.0279x 3 + 3.9569x 2 - 2.0979x + 1.3021
R 2 = 0.9935
100.00%
96.85%
95.41%
% Parameter
uncertainty in
overall outcome
variability:
SE-PSA: 88.9%
89.36%
SD-PSA: 93.3%
80.00%
70.93%
66.73%
61.87%
57.53%
60.00%
% rnd from all
51.27%
P oly. (% rnd from
all)
44.10%
40.00%
37.58%
28.36%
20.00%
19.61%
10.13%
0.00%
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
% pa ra m e te r unc e rta inty from ove ra ll outc om e va ria bility
120.00%
• For SE based
PSA the
proportion of
parameter
uncertainty
from overall
uncertainty
was 88.9% (x
confidence
width)
• For SD based
PSA the
proportion
was 93.3%
Conclusion
Run time requirements to reduce Monte Carlo error are lower
whenever the uncertainty of included parameters is increased
• In an ideal world, an infinitely high number of patients should be applied in
stochastic models to reduce random noise to zero so parameter uncertainty
can be precisely assessed.
• We have shown that even if stochastic uncertainty is not fully eliminated it is
possible to predict the expected degree of parameter uncertainty.
• For this, run time has to be increased to the level that allows a reliable
interpretation of confidence ranges.
• The higher the underlying parameter uncertainty, the lower are the
requirements to reduce 1st stochastic uncertainty in order to assess
parameter uncertainty
• In the CORE Diabetes Model, 10,000 and 50,000 patients were required in
bootstrap simulations of 1000 repetitive runs to make reliable assumptions
on parameter uncertainty when SD and SE were used to represent parameter
uncertainty
Foos et al, ADA Philadelphia, 9 June 2012
12
Thank You!
Foos et al, ADA Philadelphia, 9 June 2012
13