PRM145
A Maximum Likelihood Simulation Technique For Estimating
Adverse Event Rates From Published Trials
Ronald C Wielage,1 Gregory P Samsa,2 Timothy M Klein,1 Michael Happich3
Decision Modeling Inc., Indianapolis , IN, USA; 2Duke University, Durham, NC, USA; 3Lilly Deutschland GmbH, Homburg, Germany
ABSTRACT
Figure 1. Distribution of diclofenac hypertension
matches in SAS and CafeSim MLSs.
METHODS (continued)
• Verification
OBJECTIVE
Clinical trial publications commonly report only adverse event (AE) rates
occurring above an arbitrary threshold. Our objective was to devise a metaanalysis technique that allowed trials to be included even when AE rates fell
below thresholds.
METHODS
A maximum likelihood simulation (MLS) was devised that assumed all AE
trial results lay in the same binomial distribution truncated by reporting
thresholds.
AE data from osteoarthritis trials were retrieved. The MLS was executed
using the random number generator and binomial distribution function of
CafeSim, a Java modeling toolkit. Ten million iterations, needed for
convergence, were run for each tenth of a percent up to the highest rate
reported. For each iteration the values generated from the binomial function
were compared to the published AE rates and/or thresholds. The rate with
the most matches was designated the point estimate (PE). The range from
the 2.5 to 97.5 percentiles of matches was the 95% confidence interval (CI).
Verification was conducted for 2 AEs of 2 compounds. Results for 2 AEs
reported in all etoricoxib trials were compared to Comprehensive MetaAnalysis (CMA) results. Results for 2 AEs below reporting thresholds of one
or more diclofenac trials were compared to results from equivalent SAS
code using RANBIN and PROC FREQ.
RESULTS
The MLS estimated PEs and CIs for the etoricoxib AEs within 0.001 of CMA
(hypertension PE = 0.058 (0.059 for MLS), CI [0.051, 0.065]). The MLS
executed in CafeSim estimated PEs and CIs for the diclofenac AEs within
0.002 of those estimated in SAS, identical for hypertension, (PE = 0.027, CI
[0.022, 0.032]). When trials reported widely differing rates the MLS
converged slowly. The MLS estimated 0.000 when no trials reported the AE
rate.
CONCLUSION
An MLS technique assuming a common binomial distribution may provide a
useful estimate of AE rates when they occasionally fall below reporting
thresholds.
BACKGROUND
450000
– Results for two AEs reported in all etoricoxib trials were
compared to results of a fixed effect model estimated using
Comprehensive Meta-Analysis (CMA), a statistics program for
meta-analysis.
– Results for two AEs below the reporting thresholds of one or
more diclofenac trials were compared to results from equivalent
SAS code using RANBIN and PROC FREQ.
• The influence of reporting thresholds was investigated.
– Results for diclofenac/hypertension were changed by
manipulating reporting thresholds from actual (5%) to 10%, 2%,
and 1%.
– In the worst case, the researcher might assume that the AE
rate for the comparator in that trial was zero.
• We created a meta-analysis method that attempts to incorporate
the reporting threshold as data in the estimation of AE rates when
they fall below reporting thresholds.
METHODS
•
A maximum likelihood simulation (MLS) was devised to estimate
AE rates.
– It adopted the hypothesis that all rates for a particular AE
seen in trials for a comparator lay in the same binomial
distribution.
–For trials that did not report the AE rate, it was assumed the
reporting thresholds truncated the distribution.
• The MLS was programmed in Java.
–Used the random number generator and binomial distribution
function (BDF) of CafeSim, a Java modeling toolkit.
–Incremented in tenths of one percent from 0.0% to greater
than the highest rate reported.
–The BDF was run with a unique random number for each trial
in each replication. Results from the BDF runs were compared
to the trial results / reporting thresholds. If the results were the
same as each reported trial result and below the reporting
threshold for the publications not reporting the AE rate, the
replication “matched”.
–Ten million replications were run at each rate, forming a
distribution of matches that allowed convergence when
reported AE rates varied widely among trials.
–The mode of the distribution was adopted as the point
estimate .
• All studies reporting etoricoxib AEs reported results for hypertension
and vomiting or did not specify a reporting threshold.
50000
– Results from CafeSim MLS therefore were compared to
conventional meta-analysis results.
0
– Results were very similar, even though CMA used a fixed effects
model and Baraf (2007) was much larger than the other trials.
• Some studies reporting diclofenac AEs did not report results for
hypertension or vomiting but specified a reporting threshold.
– Results from CafeSim MLS therefore are compared to the MLS
written in equivalent SAS code.
– Results are similar for the sets of MLS results.
Table 2 shows the results of manipulating reporting thresholds on two
studies in the diclofenac / hypertension estimate.
• As expected, raising the reporting thresholds above the rate
reported by Baraf (2007) does not raise the estimated AE rate, while
lowering them lowers the estimate.
• Only slight differences between the results with CafeSim and SAS.
Table 1. Selected results.
Comparators, AE, Studies, Rates and Results
Etoricoxib / Hypertension
n
Rate [95% CI]
Baraf (2007)
3593
0.0584
Leung 2002
224
0.0759
Puopolo 2007
224
0.0402
Reginster (2007)
446
0.0516
Weisenhutter (2005)
214
NA*
Results from CMA
0.058 [0.051, 0.067]
Results from MLS
0.059 [0.051, 0.065]
Etoricoxib / Nausea
n
Rate [95% CI]
Baraf (2007)
3593
0.0264
Leung 2002
224
0.0402
Puopolo 2007
224
0.0179
Reginster (2007)
446
0.0314
Weisenhutter (2005)
214
0.0140
Results from CMA
0.027 [0.023, 0.032]
Results from MLS
0.027 [0.023, 0.034]
Diclofenac / Hypertension
n
Rate [95% CI]
Baraf (2007)
3518
0.027
Sikes (2002)
212
< 0.05
Yocum (2000)
153
< 0.05
Results from CafeSim MLS
0.027 [0.022, 0.033]
Results from SAS MLS
0.027 [0.022, 0.033]
Diclofenac / Vomiting
n
Rate [95% CI]
Baraf (2007)
3518
NA*
Sikes (2002)
212
< 0.05
Yocum (2000)
153
0.0261
Results from CafeSim MLS
0.023 [0.010, 0.052]
Results from SAS MLS
0.025 [0.009, 0.053]
* No reporting threshold specified. Not included in the estimate.
Table 2. Influence of reporting threshold.
Rate [95% CI]
Actual data (reporting thresholds = 0.05)
0.027 [0.022, 0.033]
Reporting threshold = 0.10
0.027 [0.022, 0.033]
–Rates for 15 AEs.
Reporting threshold = 0.02
0.025 [0.021, 0.031]
–Ten different oral treatments including NSAIDs, opioids, and
antidepressants
Reporting threshold = 0.01
0.025 [0.021, 0.031]
• For data, AE rates were collected via a systematic literature
review of clinical trials in osteoarthritis.
–Reported AE rates/thresholds from 38 treatment arms.
SAS
CafeSim
Discussion
The need for a method such as this arises not only from space
limitations in medical journals. A general barrier to meta-analysis of
all kinds is the limited access to data surely collected and examined
by researchers, but neither reported in the initial publication of the
research nor shared by corresponding authors. The disciplines of
meta-analysis and economic modeling greatly depend on the quality
and availability of original research. Therefore all practitioners and
proponents of meta-analysis and economic modeling in health care
should urge that as much data as possible be posted online as
appendixes to original manuscripts. While the manuscript may be
limited to a few thousand words and a half dozen tables, the space
made available online is frequently almost limitless.
• Limitations
– When an AE occurs below the reporting threshold of all
publications the method estimates a rate of 0%.
– The method is observational in nature since it focuses on the
rate in a single treatment arm rather than the difference in
rates between arms.
Diclofenac / Hypertension
–The range from 2.5 to 97.5 percentiles of matches was
designated the 95% confidence interval.
200000
100000
Not shown – AEs for which only reporting thresholds were available
were estimated by the MLS at 0%.
– In doing so the researcher discards data from trials where
the AE rate was below the reporting threshold and what that
threshold was.
250000
A small subset of results is shown in Table 1.
– Usually this is met by setting an arbitrary minimum reporting
threshold of 1% to 10% of the patients in any arm.
– In such a situation the researcher may be tempted simply
not to include in the meta-analysis those trials which did not
report the AE.
300000
RESULTS
Figure 1 depicts the distribution of matches in the diclofenac hypertension MLS.
– In performing meta-analyses to populate the incidence rates
of AEs in such a model, the researcher may find that most or
all trials for one comparator may report the rate of a
particular AE, while most or all of the trials for another
comparator do not.
350000
150000
• Due to space limits and other factors, a clinical trial publication
typically reports a partial listing of the adverse events (AEs)
encountered in the trial.
• A researcher creating an economic model may be faced with
comparators with widely divergent safety profiles.
400000
1.7%
1.8%
1.9%
2.0%
2.1%
2.2%
2.3%
2.4%
2.5%
2.6%
2.7%
2.8%
2.9%
3.0%
3.1%
3.2%
3.3%
3.4%
3.5%
3.6%
3.7%
3.8%
3.9%
4.0%
1IMedical
– The method is not suitable for extension to a mixed
treatment comparison because of its observational nature.
– As the reporting threshold grows, the error of the estimate
grows.
– The number of replications needed for convergence grows
as the number of studies and the range of their results grow.
• Possible improvements
– An extension of the method to address the first limitation
above may be to assume a distribution between 0 and the
reporting thresholds of the publications, running a maximum
likelihood simulation using that distribution.
– If clinical trial reports focused on the differences of AE rates
between arms and the reporting threshold were applied to
that criteria, the method no longer would suffer from the
second limitation above.
Additional research is needed in this area. The method described
has significant limitations, as shown above. In addition, when all
available trials report results for the AE, conventional meta-analysis
methods will better accommodate larger numbers of trials and larger
differences in the AE rates reported by them. Conventional metaanalysis also preserves the relationship between rates seen in the
active treatment and placebo arms. Bayesian methods may also be
used in this case to perform mixed treatment comparison. When
rates in some trials fall below reporting thresholds, however,
consideration may be given to using this maximum likelihood
methodology.
CONCLUSIONS
• Unreported AEs typically occur below an arbitrary reporting
threshold.
• Reporting thresholds may provide additional data in the
meta-analysis of adverse event rates.
• When AEs with low frequency may affect evaluation of
alternative treatments an estimation using maximum
likelihood simulation can be created using reporting
thresholds when actual AE rates are not reported.
Acknowledgements:
Many thanks to Julie Myers who identified and retrieved adverse
event rates from numerous osteoarthritis trials. James Gahn
performed meta-analysis using Comprehensive Meta-Analysis.
Robert Klein reviewed the abstract and content of this poster.
–Total of 150 AE rates estimated.
ISPOR 15th Annual European Congress 3-7 November, 2012
Sponsored by Eli Lilly and Company