J Clin Epidemiol Vol. 52, No. 4, pp. 281–291, 1999
Copyright © 1999 Elsevier Science Inc. All rights reserved.
0895-4356/99/$–see front matter
PII S0895-4356(98)00159-0
Recursive Cumulative Meta-analysis: A Diagnostic for
the Evolution of Total Randomized Evidence from
Group and Individual Patient Data
John P. A. Ioannidis,1,* Despina G. Contopoulos-Ioannidis,2 and Joseph Lau3
1Therapeutics
Research Program, Division of AIDS, National Institute of Allergy and Infectious Diseases, National
Institutes of Health, Bethesda, Maryland; 2Department of Pediatrics, George Washington University School of
Medicine, Washington, DC; and 3Division of Clinical Care Research, New England Medical Center Hospitals,
Tufts University School of Medicine, Boston, Massachusetts
ABSTRACT. Meta-analyses of randomized evidence may include published, unpublished, and updated data in
an ongoing estimation process that continuously accommodates more data. Synthesis may be performed either
with group data or with meta-analysis of individual patient data (MIPD). Although MIPD with updated data is
considered the gold standard of evidence, there is a need for a careful study of the impact different sources of data
have on a meta-analysis and of the change in the treatment effect estimates over sequential information steps.
Unpublished data and late-appearing data may be different from early-appearing data. Updated information after
the end of the main study follow-up may be affected by cross-overs, missing information, and unblinding. The
estimated treatment effect may thus depend on the completeness and updating of the available evidence. To
address these issues, we present recursive cumulative meta-analysis (RCM) as an extension of cumulative metaanalysis. Recursive cumulative meta-analysis is based on the principle of recalculating the results of a cumulative
meta-analysis with each new or updated piece of information and focuses on the evolution of the treatment
effect as a more complete and updated picture of the evidence becomes available. An examination of the
perturbations of the cumulative treatment effect over sequential information steps may signal the presence of
bias or heterogeneity in a meta-analysis. Recursive cumulative meta-analysis may suggest whether there is a true
underlying treatment effect to which the meta-analysis is converging and how treatment effects are sequentially
altered by new or modified evidence. The method is illustrated with an example from the conduct of an MIPD
on acyclovir in human immunodeficiency virus infection. The relative strengths and limitations of both metaanalysis of group data and MIPD are discussed through the RCM perspective. J CLIN EPIDEMIOL 52;4:281–291, 1999.
© 1999 Elsevier Science Inc.
KEY WORDS. Meta-analysis, randomized trials, bias, study design, publication bias
INTRODUCTION
Meta-analyses of randomized trials try to incorporate all the
available evidence on a given topic to effect a research synthesis. Meta-analyses based on published group data may
be affected by the selective nonpublication or late publication of negative findings [1,2]. Such publication bias [1] and
publication lag [2] may lead to potentially larger treatment
effects in meta-analyses synthesizing early published evidence. The magnitude of the problem may be reduced with
inclusion of data from abstracts or communications with
experts in the field and with involvement of the pharmaceutical sponsors in the meta-analysis to ensure that no
studies are left unearthed. The direct involvement of inves-
*Address correspondence to: John P. A. Ioannidis, M.D., 23A Kalavryton
St., Ano Voula 16673, Greece.
Accepted for publication on 6 November 1998.
tigators and sponsors may also enable retrieval of individual
patient data from the pertinent studies for performing
meta-analyses of individual patient data (MIPD) [3]. Besides further harmonizing of the original trial databases,
MIPD often allows the inclusion of follow-up data beyond
the original follow-up of the original trial publications. This
may further diversify the treatment effect estimates compared with published group data. Thus in the process of performing a meticulous, comprehensive meta-analysis, one is
being faced with a sequential accumulation of pieces of information, each of which has its strengths and a set of new
problems.
While completeness of the evidence is highly desirable,
there is a need to quantify as accurately as possible how
much missing (or late-appearing) information may affect
the results of a meta-analysis. The relative benefits and
drawbacks of updated information are more controversial
and also need to be studied systematically. Finally, besides
282
the clear advantages of allowing more accurate time-toevent analyses and the development of prognostic models
on the basis of available covariates, there is considerable
debate on whether MIPD could actually lead to substantially different estimates of the treatment effects compared
with meta-analysis of the published literature (MPL) [4–6].
To address these issues systematically, we are presenting
RCM as an extension of the cumulative meta-analysis
method [7–9]. Cumulative meta-analysis addresses the impact of new studies on prior pooled results, whereas RCM
models the changes in the cumulative treatment effect as a
result of new studies, updating of old ones, or retrieval of
unpublished ones. Recursive cumulative meta-analysis may
be used while a comprehensive meta-analysis is being performed to investigate and present the results as a process of
accumulation of missing, updated, and new information. As
a diagnostic tool, RCM may help evaluate the effect of publication bias and publication lag and the relative merits of
different pieces of evidence included in a MIPD compared
with a MPL.
METHODS
Recursive Cumulative Meta-analysis
Recursive cumulative meta-analysis shows the evolution of
the estimate of the pooled treatment effect in a meta-analysis as this estimate is recalculated every time pieces of new,
updated, or more detailed evidence become available in discrete information steps. An information step may consist of
a new study or updated or more detailed results of a study
that is already included in the meta-analysis. New data are
included in the pooled calculations in the order in which
they are obtained, while more detailed, corrected, or updated data replace in the pooled calculations less detailed,
inaccurate, or limited follow-up data from the same study.
The amount of information added or changed in each information step may vary from nil (e.g., when updating
shows no new events or when individual patient data are
obtained to replace group data, but both have an identical
count of events) to the addition of new data from large trials or major updating. The following sources of data may be
incorporated: published data—a RCM limited to such data
is identical to the traditional cumulative meta-analysis
[7,8]; presented, but unpublished data—including data that
may be retrieved from meeting abstracts and the non–peerreviewed literature; published and presented data with updated
information—including follow-up beyond the original reports; and unpublished data. Thus, at each information
step, the RCM provides an estimate and confidence intervals for the treatment effect based on the most complete
and most accurate evidence available up to that point in
time. In other words, each step of the RCM represents the
results of a cumulative meta-analysis of the most complete
data available up to this specific step, and RCM may be
seen as a series of cumulative meta-analyses. The trajectory
J. P. A. Ioannidis et al.
of the treatment effect in the RCM shows whether the
treatment effect is unstable over information steps, is continuously diminishing over information steps, or is converging to a more stable estimate. Recursive cumulative metaanalysis can be implemented both for retrospective and for
prospective meta-analyses.
Pooling of the data from individual studies at each information step may be performed with standard methods
(fixed [10] and random effects [11] models for group data;
and with study-stratified proportional hazards models [12]
either unadjusted or adjusted for important predictors of
the outcome for time-to-event individual patient data; exact methods may depend on the form of the data [13]). Heterogeneity at each step can similarly be assessed with a traditional x2 statistic [13]. If the meta-analysis protocol
specifies an a priori emphasis on subgroup analyses and
meta-regression analyses (because it is strongly anticipated
that the treatment effect may be highly variable across subgroups of patients), then the RCM approach may be implemented separately for each one of the prespecified subgroups.
Theoretically, treatment effects even from highly effective treatments could be largely dissipated if data continued
to be updated after patients had crossed over (with loss of
the advantage of the treatment over the original control
arm) or (the extreme) until all patients had died. Therefore, the inclusion of unpublished information also needs to
be studied separately from information updates. We propose
that two different diagnostic approaches be used, one including all information steps in the RCM, and the other excluding the steps that reflect the addition of updated information beyond the main (original) follow-up of each study.
The latter diagnostic is strictly more appropriate for assessing the effect of publication lag and bias and heterogeneity.
We recommend that for practical purposes, RCM should
be used as a graphical test. For modeling potential trends,
several regression models for the treatment effect may be fit
to the data, including linear, logarithmic, and inverse functions of the information step number. However, the highly
correlated nature of the discrete information step estimates
means that drawing inferences from the statistical significance of the coefficients obtained from such regression approaches may not be valid. This problem may be partly bypassed by time series modeling, but this is unlikely to be
useful, unless the number of information steps is very large.
A different way to analyze with precision the trajectory of
the treatment effects over information steps is to calculate
the ratio of the cumulative treatment effect (e.g., odds ratio) at each information step compared with the treatment
effect at the previous step. In the absence of bias and genuine heterogeneity, the plot of this cumulative treatment effect ratio should show fairly symmetrical fluctuations around
1, as in some steps, the cumulative treatment effect would
increase and at others, it would decrease; also with more
evidence, this ratio should tend to stabilize toward 1. The
Recursive Cumulative Meta-analysis
graph would show fluctuations of decreasing height, and
the rapidity of the decrease in the height of the fluctuations
would depend on the amount of accumulated evidence and
its consistency over information steps. In the presence of
bias or heterogeneity, values substantially different than 1
would be observed and the fluctuations will not decrease
over information steps or may even increase.
ILLUSTRATIVE EXAMPLE
We performed a meta-analysis of updated individual patient
data on the clinical efficacy of high dose acyclovir in patients infected with human immunodeficiency virus (HIV).
The primary end point of interest was survival, and all randomized controlled trials with any death events were included
with a total of 1792 patients and 2947 patient-years of total
follow-up. The detailed results of the meta-analysis have
been published elsewhere [14]. Here, we present the process of
retrieving and updating the information used in the MIPD.
In 1994, we performed a preliminary meta-analysis of all
the available published pertinent data [15–19] that suggested that acyclovir provided a survival benefit. The finding was robust when data from meeting presentations were
283
also considered [20,21]. This meta-analysis was updated in
February of 1995, with the inclusion of the abstract results
of a trial that found no survival benefit from acyclovir [22].
After this presentation, which seemed to contradict previous beliefs among specialists in the field (even though the
data were not significantly heterogeneous among trials), we
decided to extend the meta-analysis protocol to include
data directly from the trialists and sponsors in a meta-analysis of individual patient data. The meta-analysis team contacted not only the industry sponsors but also a large number of experts in the field and asked them regarding the
potential for any unpublished studies. A total of almost 200
people were approached. The pharmaceutical sponsors were
then asked to retrieve the pertinent databases that had
been identified. Several experts from the United States, Europe, and Australia were contacted. The investigators of
each trial verified the validity of these unpublished data
and reviewed and verified all of the analyses of their pertinent databases. Moreover, all unpublished data were contributed as raw databases of individual patient data with extensive covariate information. These were analyzed by the
meta-analysis team, and the end points as well as the mode
of analysis had been prespecified in the protocol. All of
TABLE 1. Stepwise accumulation of randomized evidence for the effect of high-dose acyclovir on the survival of patients in-
fected with HIVa
Total available evidence
Information step
(Date)
1 (August 1991)
2 (Late 1991)
3 (Early 1992)
4 (July 1992)
5 (February 1993)
6 (May 1994)
7 (October 1994)
8 (January 1995)
9 (May 1996)
10 (July 1996)
11 (June 1997)
12 (July 1997)
13 (August 1997)
14 (August 1997)
15 (September 1997)
aSteps
Deaths/patients
Pieces of evidence
Published data on 6 months of follow-up
from H56-002 ARC
Preliminary 1-year follow-up of
H56-002 ARC and AIDS
Preliminary 1-year follow-up of H14-325
Meeting presentation of preliminary
data from P53
Published data on 1-year follow-up of
H56-002 ARC and AIDS
Published data on 1-year follow-up of
H14-325
Meeting presentation of updated data
of H56-002 ARC and AIDS
Meeting presentation of ACTG063
Retrieved updated IPD from H14-325
Retrieved updated IPD from
H56-002 ARC and AIDS
Retrieved updated IPD from ACTG063
Retrieved IPD from H56-002 KS
Retrieved updated IPD from P53
Retrieved updated IPD from ACTG010
Retrieved IPD from H14-326
(Acyclovir)
(Control)
Total deaths
(n)
(Rate)
Meta-analysis
P value
1/67
3/67
4
3.0
0.33
14/129
35/136
49
18.5
0.002
44/282
45/627
79/285
82/633
123
127
21.7
10.1
,0.001
,0.001
45/627
82/633
127
10.1
,0.001
42/627
81/633
123
9.8
,0.001
85/627
127/633
212
16.8
,0.001
137/796
233/796
233/796
181/798
263/798
263/798
318
496
496
19.9
31.1
31.1
0.003
0.027
0.027
233/796
239/819
245/819
246/838
247/895
263/798
268/821
272/821
275/843
276/897
496
507
517
521
523
31.1
30.9
31.5
31.0
29.2
0.027
0.037
0.059
0.046
0.047
1–5 are reconstructed to show the sequential appearance of the first evidence on acyclovir in HIV infection. The meta-analysis was first performed at
step 6 with the publication of data from H14-325. Not included in the calculations are three small studies that had no death events during their limited follow-up [18,26,27]. The meta-analysis P value corresponds to fixed effects calculations with the information available up to each step shown. Random effects
P values were very similar because there was no significant heterogeneity between studies at any step.
Abbreviations: ARC 5 AIDS related complex; AIDS 5 acquired immunodeficiency syndrome; KS 5 Kaposi’s sarcoma; IPD 5 individual patient data;
ACTG 5 AIDS Clinical Trials Group.
284
these efforts were made in order to avoid the controversy of
using summary “data on file” provided by the industry,
which previous investigators have suggested may be a biased selection of favorable data or data reported in the most
favorable fashion [23–25].
Updated databases as well as data on unpublished studies
were gradually retrieved with coordinated efforts at several
data archives. The retrieval, cleaning, and harmonization
of different trial databases proceeded at various speeds.
Eventually, updated individual patient data on five studies
were fully retrieved, cleaned, and analyzed by May 1996
(protocol H14-325), July 1996 (protocols H56-002 ARC
and H56-002 AIDS), June 1997 (AIDS Clinical Trials
Group protocol 063), and early August 1997 (protocol
P53). On further investigation, it was identified that updated individual patient data from one additional unpublished trial were available, as well as on one previously published pilot trial for which no mortality data had been
accumulated in the original publication. These data were
also searched in the archives of the sponsor and fully retrieved, cleaned, and analyzed by July 1997 (protocol H56002 KS) and mid-August 1997 (AIDS Clinical Trials Group
protocol 010). Meanwhile, the meta-analysis results were
J. P. A. Ioannidis et al.
continuously being updated as new data were becoming
available. A first draft of the meta-analysis manuscript accompanied with detailed tables for the main calculations
was distributed to all the investigators in late August 1997,
when it was believed that all the known trial databases had
been retrieved with individual patient data. Still, we made
further exhortations to the investigators and sponsors to
point to and retrieve potentially missing pieces of information. This resulted in the recollection of one more unpublished trial whose database was retrieved in September
1997 (protocol H14-326). We continued to exhort investigators even at this stage while revised versions of the manuscript were being distributed for feedback, and we continued to check with the pharmaceutical sponsor for other
unpublished or forgotten data. Even at this final stage, in
October 1997, two small (n 5 12 [26] and n 5 21 [27])
published trials were brought to our attention that had
been missed in the initial search because they had a different primary objective (pathogenesis and evaluation of the
efficacy of a different drug). Neither of these studies had
any death end points.
Table 1 shows the sequential information steps in the retrieval and updating of evidence for the meta-analysis. For
FIGURE 1. Graphic presentation of the recursive cumulative meta-analysis, showing the pooled odds ratio and confidence intervals
(95%) at each information step. The information steps correspond to the incorporation of pieces of evidence listed in Table 1. The estimate at each information step is the result of a cumulative meta-analysis of all the data available up to this specific information step.
The pooled estimate is calculated with the Mantel-Haenszel fixed effects model. DerSimonian and Laird random effects estimates were
practically identical, as there was no significant heterogeneity at any step of the cumulative retrieval meta-analysis (x2 for heterogeneity P . 0.1 at all steps).
Recursive Cumulative Meta-analysis
some studies, data are completely new at some point,
whereas for other studies, previous data are being replaced
over time with more precise (published in contrast to preliminary) or updated follow-up information. In some cases,
the retrieved individual patient data have exactly the same
counts of events as the previous summary data in the database (see, for example, steps 10 and 11 in Table 1).
As of December 1997, only four of the eight studies included in the meta-analysis had been published. Consistent
with our previous investigation [2] describing the extent of
publication lag for efficacy trials, all three trials showing a
survival benefit had been published, whereas four of the five
trials that did not show a significant survival benefit were
still unpublished 5 to 9 years after starting enrollment. One
of the four unpublished studies was eventually published in
mid-1998, almost a decade after starting enrollment [28].
Figure 1 shows a graphical representation of the RCM as
more complete, corrected, and updated information becomes available. As seen, early estimates of the treatment
effect based on early published trials tend to be larger than
later cumulative effects that give a more complete picture
of the whole evidence. An inspection of the RCM trajectory suggests that a meta-analysis limited to information
from the first few steps may be misleading, as there is strong
285
evidence that the accumulating information is continuously attenuating the magnitude of the treatment effect,
and there is no suggestion that the treatment effect is yet
converging to a more stable estimate. Thus, the RCM suggests that a meta-analysis of published data may be misleading, as the evidence has not converged yet and the conclusion of large treatment benefit that is highly significant
(e.g., P 5 0.0003 at step 6) would be premature. Moreover,
the RCM suggests that one may be misled in judging the results as conclusive simply on the basis of very extreme levels of statistical significance achieved in early steps (P 5
0.002 by step 2, P , 0.001 consistently in steps 3–6). By
contrast, the late additions of evidence do not seem to affect the magnitude of the treatment effect in any substantive way. The magnitude and confidence intervals of the
treatment effect seem to converge after the ninth information step. Evidence is probably more reliable at this stage
and suggests a cautious interpretation of the meta-analysis
results, which suggest the presence of a modest treatment
effect (pooled odds ratio 0.75), which is more likely to be
real, although the 95% confidence intervals extend almost
to no benefit.
When the updated follow-up steps were excluded to separate the effect of potential retrieval bias from the effect of
FIGURE 2. Graphic presentation of the recursive cumulative meta-analysis, excluding updated follow-up information beyond the original follow-up of each trial. The estimate at each information step is the result of a cumulative meta-analysis of all the data available up
to this specific information step. The pooled estimate at each step is calculated with the Mantel-Haenszel fixed effect model. DerSimonian and Laird random effects estimates were practically identical, as there was not heterogeneity at any step of the cumulative retrieval meta-analysis (x2 for heterogeneity P . 0.1 at all steps).
J. P. A. Ioannidis et al.
286
updated information, the RCM looked similar (Fig. 2).
Again, there was evidence that the estimate of the treatment effect converged after the eighth step. The magnitude
of the converging treatment effect seemed slightly larger
when updating steps were excluded, but a conservative interpretation would still be appropriate given the confidence
intervals.
Table 2 shows for comparison the results of typical metaanalyses based on published versus published and presented
versus all published and unpublished data. Each analysis is
reported with the inclusion or not of updated follow-up
data. The treatment effect seems to shrink across these categories as missing information is included and the updated
follow-up data tend to be even more conservative. Nevertheless, with the exception of the results of a meta-analysis
limited only to published data, the difference across the
other categories is rather small as far as the magnitude of
the treatment effect is concerned. Also shown are results of
MIPD analyses including all data either censored at the
specified follow-up of the primary trial analysis for each
study or including all updated follow-up information. There
is hardly any meaningful difference between summary and
individual level estimates once similarly complete data are
considered in the calculations.
Figure 3 shows the evolution of the ratio of the cumulative treatment effect (odds ratio) at each information step
divided by the treatment effect at the previous step. Large
initial fluctuations eventually lead to a more stabilized
treatment effect with little relative change between different steps. To put this plot into perspective, two more examples are given using data from two very well-known published cumulative meta-analyses: intravenous streptokinase
in acute myocardial infarction, a typical example in which
there is widespread consensus on the validity of the metaanalysis results and the intervention unquestionably works
[7] (Fig. 4); and magnesium salts in acute myocardial infarction [29], a typical example in which a mega-trial (ISIS-4
[30]) arrived at opposite conclusions compared with a previous meta-analysis, causing overt debate on the efficacy of
this intervention (Fig. 5) [31]. The fluctuations of the treatment effect of streptokinase expectedly diminish rapidly
over information steps. On the other hand, in the case of
magnesium, the treatment effect continues to fluctuate as
much or even more as more data are accumulated. Increasing fluctuations are noted even before the appearance of
ISIS-4 (added at information step 11 in the figure), the
mega-trail that found no benefit from magnesium in contrast with previous evidence. Persistence or increase in the
treatment effect fluctuations may be a signal of genuine
heterogeneity or bias, and this signal was apparent for magnesium very early on in the course of the accumulated evidence.
TABLE 2. Comparison of meta-analysis estimates
Death/patients
Meta-analysis of summary data
Without updated follow-up information
Published studies
Published and presented studies
All studies
With updated follow-up information
Published studies
Published and presented studies
All studies
Acyclovir
Control
Fixed effects
Random effects
41/282
95/796
99/876
78/285
136/798
138/875
0.43 (0.28–0.66)
0.61 (0.45–0.83)
0.64 (0.47–0.86)
0.43 (0.28–0.67)
0.56 (0.36–0.88)
0.60 (0.39–0.92)
41/282
137/796
247/895
78/285
181/798
276/897
0.43 (0.28–0.66)
0.64 (0.48–0.86)
0.75 (0.57–1.00)
0.43 (0.28–0.67)
0.64 (0.48–0.86)
0.76 (0.57–1.00)
Deaths/patients
Meta-analysis of individual patient dataa
Without updated follow-up information
With updated follow-up information
Pooled odds ratio
(95% CI)
Proportional hazards rate ratio
(95% CI)
Acyclovir
Placebo
Unadjusted
Adjusted
109/876
247/895
149/897
276/897
0.68 (0.53–0.87)
0.79 (0.66–0.93)
0.67 (0.52–0.85)
0.78 (0.65–0.93)
The status of the studies (published, presented, unpublished) is as of September 1997 (time of completion of the meta-analysis of individual patient data).
Fixed effects calculations are performed according to the Mantel-Haenszel model, and random effects calculations are performed according to the DerSimonian and Laird model. The two estimates are very similar because there was no significant heterogeneity in any case (x2 for heterogeneity P . 0.1). Unadjusted proportional hazard model estimates are stratified per study; adjusted estimates are also adjusted for baseline CD4 count (square root thereof), baseline
hemoglobin, age, gender, race, and AIDS diagnosis at baseline. The count of events when no updating is considered is slightly different in the group data
meta-analysis versus the MIPD, as in the MIPD, the results include all events that occurred in the originally specified follow-up, whereas a few such events
had not been included in the published analyses of the main follow-up of these studies. One study (ACTG010) is included only in meta-analyses using updated information, as no deaths were recorded in the initial follow-up report.
aIncludes all studies (both published and unpublished).
Abbreviations: AIDS 5 acquired immunodeficiency syndrome; CI 5 confidence interval; MIPD 5 meta-analysis of individual patient data.
Recursive Cumulative Meta-analysis
287
FIGURE 3. Graph of the cumulative treatment effect ratio over sequential information steps for the acyclovir recursive cumulative
meta-analysis. For each information step, the cumulative treatment effect ratio is defined as the cumulative odds ratio at that step divided by the cumulative odds ratio at the previous step. Values .1 suggest an overestimation of the treatment effect in the previous
step compared with the current one. Values ,1 suggest an underestimation of the treatment effect at the previous step compared with
the current one.
DISCUSSION
We have presented an extension of cumulative meta-analysis that can be used to evaluate the composite evidence on
a clinical topic as more information is being obtained, retrieved, and updated. Recursive cumulative meta-analysis
offers a diagnostic approach that can visualize the accumulating evidence as a continuum and can be helpful in assessing whether early estimates of the treatment effect change
over time; whether they are moderated or completely dissipated by overt or hidden publication bias [1] and publication lag [2] for negative results; and what effect updated follow-up has on the data. It deals in a systematic fashion with
issues that have been thought to be obstacles to objective
effect estimation in meta-analysis.
Clinical trials are typically designed with a frequentist
hypothesis in mind (the treatment is not better than control), while physicians usually want to know the magnitude
of the treatment effect for estimation purposes (how much
it works). The interchange of frequentist and estimation
approaches is not always without problems [32–34]. Moreover, there is a growing understanding that clinical evidence is a dynamic process, not a static estimation of a single
treatment effect at a single time-point [7,35,36]. Recursive
cumulative meta-analysis attempts to bridge this transition
from frequentist trials to unbiased estimation.
Until now, meta-analyses have been reported in the literature as final products with little or no information on
how information was gathered over time, as in most cases,
all of this information has been obtained from literature
searches performed at a specific time point. However, initiatives such as the Cochrane Collaboration [37] have introduced and will probably soon establish the concept that systematic reviews should be continuously renewed as new
evidence appears, and they are never final. In this process,
which is likely to be the future of meta-analysis, RCM is focusing on the evolution of treatment effects over time. Our
approach suggests that simply reporting a weighted aggregate may not be adequate, and details on the order in which
data appeared can offer insight on whether the treatment
effect is changing as a more complete picture evolves. The
implications are not limited to practitioners of meta-analysis, but to medicine at large, as the concept can be applied
to any meta-analysis regardless of subject matter. Fluctuations in the magnitude of the treatment effect may point to
the presence of either bias or genuine heterogeneity that
need to be scrutinized. As a general rule, meta-analyses
with excessive fluctuations, asymmetrical fluctuations, and
288
J. P. A. Ioannidis et al.
FIGURE 4. Graph of the cumulative treatment effect ratio over sequential information steps for a cumulative meta-analysis of intravenous streptokinase in myocardial infarction. This example uses a database that has been published and described in detail elsewhere
[7]. For each information step, the cumulative treatment effect ratio is defined as the cumulative odds ratio at that step divided by the
cumulative odds ratio at the previous step. Values .1 suggest an overestimation of the treatment effect in the previous step compared with the current one. Values ,1 suggest an underestimation of the treatment effect at the previous step compared with the current one.
those that do not reach values close to 1 for the cumulative
treatment effect ratio should be scrutinized for bias and heterogeneity. It would be imprudent to put specific values
that would define an “abnormal” signal in these properties
of the trajectory because we do not know what is the absolute unbiased truth in clinical research in order to correlate
it with specific trajectories. Although large simple trials
have been proposed as the gold standard, several investigations that have compared them to meta-analyses [38–40]
have not reached consensus on whether large trials can indeed be gold standards [41]. The RCM approach offers a
framework wherein current and future meta-analyses may
be presented and scrutinized and empirical evidence of the
properties of the trajectories of meta-analyses across different fields can be collected. It is thus a first step toward a
more robust interpretation of clinical evidence and how
such evidence is obtained.
It is possible that in many circumstances, the initial estimates of the treatment effect would gradually decrease over
sequential information steps before eventually converging.
There is strong prospective evidence that trials with nonsignificant results take longer to complete, submit, and pub-
lish [2]. On the other hand, studies with large treatment effects may be stopped early and are invariably published
quickly [2]. This would result in diminishing treatment effects over time in the RCM framework, while eventually
one would expect to be able to reach a more accurate estimate of the treatment’s efficacy once the complete evidence is accumulated. Empirical evidence among a small
number of meta-analyses of the published literature confirms that early trials tend to show larger estimates of the
treatment effect than late trials [42]. One should thus interpret cautiously the results of early meta-analysis when large
pieces of evidence are not yet in from ongoing trials.
Recursive cumulative meta-analysis may complement
and improve over sequential monitoring approaches that
have been proposed for meta-analyses based on the use of
alpha-spending functions [36]. Such approaches aim to
avoid premature claims on statistical significance when
only small portions of the total evidence are available and
statistical significance at an early look at the data may be
due to chance. However, alpha-spending function approaches
are not appropriate when there is bias in the order in which
the evidence is accumulated. In contrast to a single large
Recursive Cumulative Meta-analysis
289
FIGURE 5. Graph of the cumulative treatment effect ratio over sequential information steps for a cumulative meta-analysis of intravenous magnesium in acute myocardial infarction. This example uses a database that has been published and described in detail elsewhere [29]. For each information step, the cumulative treatment effect ratio is defined as the cumulative odds ratio at that step divided
by the cumulative odds ratio at the previous step. Values .1 suggest an overestimation of the treatment effect in the previous step
compared with the current one. Values ,1 suggest an underestimation of the treatment effect at the previous step compared with the
current one.
clinical trial in which there is no reason to believe that the
most favorable results will be systematically accumulated
first, in a meta-analysis, it is possible that this may happen.
Pogue and Yusuf [36] have suggested that alpha-spending
functions would not have claimed statistical significance
for a controversial meta-analysis (magnesium in acute myocardial infarction), whereas this would have been the case
in an undisputed effective medication (streptokinase in myocardial infarction). Still Egger and colleagues [43] have
recently shown that the inclusion of abstracts and unpublished data would have resulted in reaching robust significance for magnesium as well, even if an alpha-spending
function were to be used. Alpha-spending functions are
problematic in meta-analysis when there is bias or genuine
heterogeneity of the treatment effect. In contrast, the RCM
aims specifically at capturing this potential irregularity in
the evolution of the treatment effect over time.
Recursive cumulative meta-analysis may also offer a
means for the dissection of sources of disagreement between
MIPD and MPL in a consistent and standardized manner as
data are accumulated for an MIPD. There has been a long
debate on the relative merits of MIPD over MPL, but actual
data are scarce. Early evidence suggested that MIPD gives
significantly more conservative estimates than MPL [7], but
other investigators [8,9] did not observe any discrepancies.
In particular, summary-level or individual-level information seems to lead to the same estimates when based on the
same databases, as seen also in the example of the acyclovir
meta-analysis. Discrepant results probably arise either by
publication bias in MPL or retrieval bias in MIPD or by the
inclusion of updated information that differentiate the databases used by the two methods.
The collection of updated data beyond the original randomized follow-up presents several problems of implementation and interpretation. Such data should be carefully examined in each meta-analysis and the RCM graphs with
versus without the updating should be compared. The updated follow-up analyses are not appropriate when simply
the effects of intervention versus placebo are to be studied,
as many patients may cross-over once the main trial is over
and information is released to the participants. Cross-overs
may actually be influenced by the released trial results.
Still, the updated follow-up has its advantages, such as the
ability of assessing late outcomes of patients, and the com-
J. P. A. Ioannidis et al.
290
parisons are still protected by the initial randomization if
one is interested in making an assessment of simply whether
early versus deferred implementation of the tested intervention makes a difference in the long run. Updated data
are likely to offer more evidence and may even correct biased large estimates of the treatment effects based on interim analyses. They may give better evidence on the durability of a treatment effect, and for clinical purposes, it is
important to know whether the efficacy of early treatment
is time-limited and, if so, to determine the duration of its
benefit [44]. However, besides the effect of cross-overs, updated data may be of inferior quality, may suffer from more
extensive missing data [45], end points may not have been
verified as completely and as objectively as those during the
main follow-up (particularly for masked trials), and the decision to attempt some updating may be influenced by what
the early results have shown (e.g., updating may be performed preferentially on studies that show the larger early
benefits). The net effect of these forces on the treatment
effect is difficult to decipher. Generally, updated survival
data are probably more trustworthy than updated data on
more subjective end points such as disease progression or
change in health status and surrogate markers. This is the
reason that the acyclovir MIPD protocol was designed with
survival only as the primary end point. It is reassuring when
the results of the RCM are consistent regardless of whether
updated data are included or not, as in the case of the presented MIPD. When the contrary is observed, the robustness
of the updated data needs to be carefully scrutinized or the durability of the treatment benefit may need to be questioned.
In the absence of extensive empirical evidence in the relative validity of MPL and MIPD, strong statements about
their relative importance may be premature. Both can offer
very useful information, provided their strengths and limitations are understood. Meta-analysis of individual patient
data offers undoubtedly the strong advantages of performing risk modeling and more detailed time-to-event analyses,
but it is more costly and resource consuming and requires
the bona fide collaboration of a large number of investigators who are all enlightened to recognize the importance of
such endeavors [3]. Failure to gain approval for access to all
individual patient data may add another layer of bias peculiar to MIPD, and full sharing of all data, as was the case in
the presented MIPD, is often not attained. Publication bias,
retrieval bias, and the effects of updated information need
to be evaluated as a whole. This may be performed in the
RCM framework.
Finally, for some treatments, different patient populations may experience different treatment benefits from a
treatment [46–48]. If this is strongly suspected, we recommend that RCM is modified to adjust also for predictors
that may determine the magnitude of the treatment effect.
A separate RCM may then be performed for each subgroup
of patients that has been hypothesized in the meta-analysis
protocol to potentially experience a different treatment ef-
fect. In the acyclovir MIPD, we had overall observed no evidence that the magnitude of the survival benefit was different in patients at different levels of risk defined by predictors
such as baseline CD4 count, age, acquired immunodeficiency syndrome diagnosis at entry, and baseline hemoglobin level [14]. Of course, subgroup effects are harder to discern [49], but for some treatments, it may be more
imperative to adjust for subgroup effects. In particular, new
treatments are often likely to be tested first in patients at
high risk of the disease outcome and, if promising, then
tested in patients of lower risk. Trials with high-risk patients are also likely to accumulate the necessary number of
events earlier than trials of low risk patients. Empirical evidence shows that slow accumulation of events is a common
reason for the protracted conduct and late publication of
randomized trials [2]. The diversity of the studied patient
populations needs to be examined carefully to evaluate
whether the inclusion criteria of a meta-analysis may be too
broad and whether the RCM diagnostics should be applied
to all the studied populations or subgroups thereof.
We are grateful to several investigators who contributed data for the
meta-analysis of acyclovir and in particular to the principal investigators
Ann C. Collier, David A. Cooper, Brian G. Gazzard, Paul D. Griffiths, A. Paul Fiddian, Andrew T. Pavia, Michael S. Saag, Spotswood
L. Spruance, and Michael S. Youle. Christopher H. Schmid contributed valuable comments to the manuscript.
References
1. Dickersin K. The existence of publication bias and risk factors
for its occurrence. JAMA 1990; 263: 1385–1389.
2. Ioannidis JP. Effect of the statistical significance of results on
the time to completion and publication of randomized efficacy trials. JAMA 1998; 279: 281–286.
3. Stewart LA, Clarke MJ. Practical methodology of meta-analyses (overviews) using updated individual patient data. Cochrane Working Group. Stat Med 1995; 14: 2057–2079.
4. Stewart LA, Parmar MKB. Meta-analysis of the literature or
of individual patient data: Is there a difference? Lancet 1993;
341: 418–422.
5. Steinberg KK, Smith SJ, Stroup DF, Olkin I, Lee NC, Williamson GD, et al. Comparison of effect estimates from a metaanalysis of summary data from published studies and from a
meta-analysis using individual patient data for ovarian cancer
studies. Am J Epidemiol 1997; 145: 917–925.
6. Franzosi MG, Santoro E, Santoro L, and ACE-Inhibitor Collaborative Group. Prospective meta-analysis using individual
patient data vs. meta-analysis of published reports: The case
of ACE-inhibitors in myocardial infarction. Control Clin
Trials 1997; 18: 183S.
7. Lau J, Antman EM, Jimenez-Silva J, et al. Cumulative metaanalysis of trials of treatments for myocardial infarction. N
Engl J Med 1992; 327: 248–254.
8. Lau J, Schmid CH, Chalmers TC. Cumulative meta-analysis
builds evidence for exemplary medical care. J Clin Epidemiol
1995; 48: 45–57.
9. Berkey CS, Mosteller F, Lau J, Antman EM. Uncertainty of
the time of first significance in random effects cumulative
meta-analysis. Control Clin Trials 1996; 17: 357–371.
10. Mantel N, Haenszel W. Statistical aspects of the analysis of
Recursive Cumulative Meta-analysis
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
data from retrospective studies of disease. J Natl Cancer Inst
1959; 22: 719–748.
Der Simonian R, Laird NM. Meta-analysis in clinical trials.
Control Clin Trials 1986; 7: 177–188.
Cox DR, Oakes D. Analysis of Survival Data. London:
Chapman and Hall; 1984.
Lau J, Ioannidis JP, Schmid CH. Quantitative synthesis in
systematic reviews. Ann Intern Med 1997; 126: 820–826.
Ioannidis JP, Collier AC, Cooper DA, Corey L, Fiddian AP,
Gazzard BG, et al. Clinical efficacy of high-dose acyclovir in
patients with human immunodeficiency virus infection. J Infect Dis 1998; 178: 349–359.
Cooper DA, Pedersen C, Aiuti F, Vilde JL, Ruhnke M, Pehrson PO, et al. The efficacy and safety of zidovudine with or
without acyclovir in the treatment of patients with AIDSrelated complex. AIDS 1991; 5: 933–943.
Cooper DA, Pehrson O, Pedersen C, Moroni M, Oksenhehdler
H, Rozenbaum W, et al. The efficacy and safety of zidovudine
alone or as cotherapy with acyclovir for the treatment of patients with AIDS and AIDS-related complex: a double-blind,
randomized trial. AIDS 1993; 7: 197–207.
Youle MS, Gazzard BG, Johnson MA, Cooper DA, Hoy JF,
Busch H, et al. Effects of high-dose oral acyclovir on herpesvirus disease and survival in patients with advanced HIV disease: a double-blind, placebo-controlled study. AIDS 1994; 8:
641–649.
Chavanet P, Malet J, Wladner A, Aho S, Buisson M, Biefeld
P, et al. A double-blind randomized placebo trial on very high
doses of acyclovir in weakly symptomatic HIV-patients. Cancer Detect Prev 1990; 14: 669–673.
Collier AC, Bozzette S, Coombs RW, Causey DM, Schoenfeld DA, Spector SA, et al. A pilot study of low-dose zidovudine in human immunodeficiency virus infection. N Engl J
Med 1990; 323: 1015–1021.
Lavelle J, Lang W, Lefkowitz L, et al. A randomized trial comparing zidovudine alone and in combination with acyclovir
for treatment of early symptomatic HIV infection. In: Abstracts of the VIII International Conference on AIDS, Amsterdam, 1992. (Abstract PoB 3585)
Cooper DA, Gazzard GB, Stevens J, Bell AR. The effect of
acyclovir cotherapy on survival in individuals with advanced
HIV disease. In: Interscience Conference on Antimicrobial
Agents and Chemotherapy, 1994 Oct 4–7. (Abstract 61)
Collier AC, Schoenfeld DA, Bourland D, Hirsch M, Davis
LG, Corey L. Prospective comparative study of acyclovir and
zidovudine versus zidovudine alone in patients with AIDS. In:
Abstracts of the 2nd National Conference on Human Retroviruses and Related Infections, January 29–February 2,
1995. (Abstract 383)
Smith GD, Egger M. Meta-analysis. Unresolved issues and future developments. BMJ 1998; 316: 221–225.
Song F, Freemantle N, Sheldon TA, House A, Watson P,
Long A, et al. Selective serotonin inhibitors: Meta-analysis of
efficacy and acceptability. BMJ 1993; 306: 683–687.
Nakielny J. Effective and acceptable treatment for depression.
BMJ 1993; 306: 1125.
Mulder JW, de Wolf F, Goudsmit J, et al. Long-term zidovudine treatment of asymptomatic HIV-1 infected subjects. Antiviral Res 1990; 13: 127–138.
Luxton JC, Williams I, Weller I, Crawford DH. Epstein-Barr
virus infection of HIV-seropositive individuals is transiently
suppressed by high-dose acyclovir treatment. AIDS 1993; 7:
1337–1343.
Collier AC, Schoenfel D, Hirsch M, et al. Lack of effect on
survival of high-dose acyclovir and zidovudine compared with
291
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
zidovudine alone for acquired immunodeficiency syndrome.
Antiviral Ther 1998; 3: 25–32.
Schmid CH, Lau J, McIntosh M, Cappelleri JC. An empirical
study of the effect of the control rate as a predictor of treatment
efficacy in a meta-analysis of clinical trials. Stat Med. 1998; 17:
1923–1942.
ISIS-4 Collaborative Group. ISIS-4: A randomized factorial
trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58050 patients with suspected acute myocardial infarction. Lancet 1995; 345: 669–685.
Borzak S, Ridker PM. Discordance between meta-analyses
and large-scale randomized, controlled trials. Ann Intern
Med 1995; 123: 873–877.
Hughes MD, Freedman LS, Pocock SJ. The impact of stopping rules on heterogeneity of results in overviews of clinical
trials. Biometrics 1992; 48: 41–53.
Hughes MD, Pocock SJ. Stopping rules and estimation problems in clinical trials. Stat Med 1988; 7: 1231–1242.
Pocock SJ, Hughes MD. Practical problems in interim analyses, with particular regard to estimation. Control Clin Trials
1989; 10(Suppl. 4): 209S–221S.
Lau J, Ioannidis JP, Schmid CH. Summing up evidence: One
answer is not always enough. Lancet 1998; 351: 123–127.
Pogue JM, Yusuf S. Cumulative evidence from randomized
trials: Utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials 1997; 18: 580–593.
Bero L, Rennie D. The Cochrane Collaboration. JAMA
1995; 274: 1935–1938.
Villar J, Carroli G, Belizan JM. Predictive ability of meta-analysis of randomized controlled trials. Lancet 1995;345: 772–776.
Cappelleri JC, Ioannidis JPA, Schmid CH, et al. Large trials
versus meta-analyses of small trials: How do their results compare? JAMA 1996; 276: 1332–1338.
LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F.
Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997; 337: 536–542.
Ioannidis JPA, Cappelleri JC, Lau J. Issues in comparisons between meta-analyses and large trials. JAMA 1998; 279: 1089–
1093.
Rothwell PM, Robertson G. Meta-analyses of randomised
controlled trials. Lancet 1998; 350: 1181–1182. (Letter)
Egger M, Smith GD, Sterne JAC. Meta-analysis: Is moving
the goal post the answer? Lancet 1998; 351: 1517.
Ioannidis JP, Cappelleri JC, Lau J, Skalnik PR, Melville B,
Chalmers TC, et al. Early or deferred zidovudine therapy in
HIV-infected patients without an AIDS-defining illness: A
meta-analysis. Ann Intern Med 1995; 122: 856–866.
Ioannidis JP, Bassett R, Hughes MD, Valbenting PA, Sacks
HS, Lau J. Predictors and impact of patients lost to follow-up
in a long-term trial of early versus deferred antiretroviral
treatment. J Acquir Immune Defic Syndr Hum Retrovirol
1997; 16: 22–30.
McIntosh M. The population risk as an explanatory variable
in research synthesis of clinical trials. Stat Med 1996; 15:
1713–1728.
Ioannidis JP, Cappelleri JC, Schmid CH, Lau J. Impact of epidemic and individual heterogeneity on the population distribution of disease progression rates: An example from trials of
human immunodeficiency virus infection. Am J Epidemiol
1996; 144: 1074–1085.
Ioannidis JP, Lau J. The impact of high risk patients on the results of clinical trials. J Clin Epidemiol 1997; 50: 1089–1098.
Yusuf S, Wites J, Probstfield J, Tyroler HA. Analysis and interpretation of treatment effects in subgroups of patients in
randomized clinical trials. JAMA 1991; 266: 93–98.