outcomes assessment
Enhancing Meta-analysis by Considering the Correlation
between Two Outcomes
Julie Roïz, MSc, Director, UK Operations, Creativ-Ceutical Ltd., London, UK; Julie Dorey, MSc, Manager, US Operations,
Creativ-Ceutical USA Inc., Chicago, IL, USA; and Anne-Lise Vataire, MSc, PhD candidate, Pharmacoeconomics, University of Lyon
1, Villeurbanne, France
KEY POINTS
• Two models have been proposed for bivariate meta-analysis.
• Bivariate meta-analysis is recommended when heterogeneity is limited.
• Bivariate techniques show potential for conducting meta-analysis with missing data.
Introduction
Meta-analysis is a statistical technique used to pool
quantitative endpoints (e.g. treatment effects such
as odds-ratios) from independent clinical studies.
Meta-analysis provides a higher strength of evidence
than single-study analysis, allowing decision makers
to make decisions on combined results instead of
multiple results from multiple studies.
Meta-analysis can be carried out using either fixedeffect or random-effect models. Fixed-effect models
assume homogeneity between individual studies.
This means no interaction between the treatment
effect and each study. Differences in study results
are due to sampling and random error. Randomeffect models account for some heterogeneity
between individual studies. Differences between
study results are not only due to chance, but also to
actual differences between studies (design, patient
characteristics, environmental factors, etc.).
In meta-analysis, two types of variability exist:
the intra/within variance and the inter/between
variance. Let be the estimate (^) of the treatment
effect
in each individual study (i). We assume:
is the within-study variance, it represents
the variance due to sampling fluctuations. is
the between-study variance. It represents the
variance due to interaction between the treatment
effect and each study.
Bivariate Meta-Analysis
Bivariate data are data composed of two variables.
Hence, bivariate meta-analysis is meta-analysis of
data that include two variables. In the presence of
two outcomes, the standard approach consists of
analysing the two variables separately, regardless
of the relationship between the two measures.
Bivariate meta-analysis is a statistical technique
that allows simultaneous meta-analysis of the
two variables, taking into account the correlation
between them.
Bivariate meta-analysis should be conducted when
there is a high correlation between variables (for
example, sensitivity and specificity of a diagnostic
test, risk of vertebral and non-vertebral fractures,
etc.) in order to improve statistical power and
obtain more robust results analyses in terms of
precision of estimate and reduction of bias. The
bivariate approach preserves the two-dimensional
nature of the data by analyzing the two outcomes
jointly and by incorporating any correlation that
might exist between these two measures. Bivariate
meta-analysis also allows decision makers to take
decision jointly on two outcomes.
Bivariate meta-analysis is now being used more
often. We searched the literature for applications of
bivariate meta-analysis, and identified 46 published
studies as of May 2014. The main finding is that
the first study was published in 2006, and that the
number grew steadily since then. The second was
that most studies are conducted in obstetrics where
imaging plays an important role and is linked to the
third finding: the outcomes most often analyzed
jointly are sensitivity and specificity of tests.
Bivariate meta-analyses were conducted from a low
number of studies (less than 10) to a large number
of studies (more than 40).
Bivariate Meta-Analysis models
The general model was published by Riley et al.
in 2007 [1]. It was designed as a random-effect
model, and requires five parameters per study:
the treatment effects on both variables (presented
as mean changes, odds-ratios, proportions, etc.),
their two variances, and the correlation coefficient
between the two outcomes. The parameterization is
as follows:
A second model was proposed by Van Houwelingen
et al. in 2002 [2]. Its main advantage over the
general model is that it does not require informed
correlation coefficients for all included studies. Both
treatment effects and their variance are enough for
its estimation:
In this second model, the within-study correlation
can be interpreted as the global influenza of
one outcome on the other. The between-study
correlation
on the other hand can be
interpreted as the influenza of the error for one
outcome onto the error for the other.
Experiments
Two experiments were implemented to investigate:
• When to use bivariate meta-analysis to improve
robustness of analysis over univariate
• If bivariate meta-analysis can help when in
presence of missing data
Experiment 1 – Univariate or bivariate
meta-analysis
To investigate if and when bivariate meta-analysis
performs better than two univariate analyses–in
terms of precision of estimate and reduction of
bias–we designed and conducted an experiment
based on simulations of multiple meta-analysis
scenarios. We varied the number of included
studies, the degree of correlation and the presence
of heterogeneity.
We simulated 50 sets of a number of individual
studies, reporting odds-ratios (ORs) of a mean
of 0.65 for two outcomes. The number of studies
by meta-analysis varied between 8 and 15. Two
assumptions were considered for the correlation
between OR for the two outcomes: 0.1 and 0.7.
Two models were estimated for each simulated
data set: univariate random-effect meta-analytic
models for each outcome independently, and a
bivariate random-effect meta-analytic model based
on the Riley model.
Results for the various simulation scenarios are
presented in Table 1. >>
Volume 20 Number 5 SEPTEMBER/OCTOBER 2014 ISPOR CONNECTIONS 15
Table 1. Results for various degrees of heterogeneity,
correlation, and number of included studies
8 studies
15 studies
No heterogeneity
Low correlation (0.1)
Univariate preferred
High correlation (0.7)
Bivariate preferred
Low correlation (0.1)
Bivariate preferred
High correlation (0.7)
Bivariate preferred
Heterogeneity
Low correlation (0.1)
Univariate preferred
High correlation (0.7)
Univariate preferred
Low correlation (0.1)
Bivariate preferred
High correlation (0.7)
Bivariate preferred
Bivariate models performed better in presence of
high correlation, or could capture even low values
of correlation when the number of studies was
large. In presence of heterogeneity, however, the
complexity of the bivariate model took power and
added bias. Heterogeneity remains the first hurdle
to tackle when performing meta-analysis.
Experiment 2 – Bivariate meta-analysis
and missing data
We then designed a second experiment to
investigate if and when bivariate meta-analysis
offers advantages over than two univariate analyses
when data is missing for one of the two correlated
endpoints.
We simulated 50 sets of 25 fictitious studies,
reporting odds-ratios for two outcomes. Logarithms
of odds ratios (log OR) were assumed to follow a
bivariate normal distribution with mean of -0.5,
corresponding to ORs of 0.606 and a variance of 0.25
for both outcomes. The number of patients by study
varied between 50 and 500. Three assumptions
were considered for the correlation between mean
log OR for the two endpoints: 0, 0.5, and 1.
For the complete data, two models
were estimated for each simulated
data set: univariate random-effect
meta-analytic models for each
outcome independently, and a
bivariate random-effect metaanalytic model based on the van
Houwelingen model.
Two situations in which one of the
endpoints was not reported for
some studies were then considered:
• Missing at random: 50% of
observations for OR2 are censored
at random
• Non-ignorable missing: The 50%
greater values in OR2 are censored
For the situations with missing data, two variants
of the bivariate random-effects model were
investigated:
• A two-stage approach in a global Bayesian model
using univariate model for studies with one endpoint
and bivariate model for studies with two endpoints
• Bayesian bivariate model with prior imputation
of the variance of second endpoint for studies with
one endpoint only, based on the correlation between
variances for the two endpoints.
The models were tested with two types of prior for
the correlation between outcomes:
• Non-informative prior
• Informative prior for the within-study correlation
and no-informative for the between-study
correlation
The logarithms of missing odds-ratios could be
imputed in the estimation process as long as
prior imputation of the variance of those values
was performed. Empirical correlations between
simulated log ORs, however, were substantially
lower than specified values (e.g. 0.26 for a specified
correlation of 1).
Using the van Houwelingen model, results of the
bivariate models were similar to results of univariate
models in situations without missing data. In
situations with missing data and non-informative
prior, using bivariate model has little impact on
results, whatever the correlation between endpoints.
The use of informative priors improved significantly
the results in presence of missing data. When an
informative prior was used for the within-study
correlation, estimates obtained using bivariate
models were closer to the “true” values of the ORs.
Conclusion
Bivariate meta-analysis can add strength to
univariate meta-analysis when outcomes are
strongly correlated, or if the number of studies is
large, and heterogeneity limited. It should always,
however, be contrasted with the results of two
independent univariate meta-analyses as these
have shown to perform better in some cases.
Bayesian bivariate meta-analysis also offers
potential to improve treatment effect estimates
when information is collected for two correlated
endpoints, but missing for one of them in some
studies.
REFERENCES
[1] Riley RD, Abrams KR, Sutton AJ, et al. Bivariate randomeffects meta-analysis and the estimation of betweenstudy correlation. BMC Med Res Methodol 2007;7:3.
[2] van Houwelingen HC, Arends LR, Stijnen T. Advanced
methods in meta-analysis: multivariate approach and
IC
meta-regression. Stat Med 2002;21:589-624. n
2015-2016 Board of Directors
call for nominations
...........................................................
ISPOR is a member-driven organization. Your participation is essential.
The activities of the organization are a response to member need. As a member-driven
organization, its governance is determined by the membership.
For the 2014-2015 ISPOR Board of Directors members (including terms of office), see:
http://www.ispor.org/board/index.asp.
As an ISPOR member, you are encouraged to submit a nomination to serve on the
2015-2016 ISPOR Board of Directors.
Self nominations are permitted.
Please email your recommendation, along with curriculum vitae, to: [email protected].
The deadline for submitting nominations is Friday, January 2, 2015.
16 Volume 20 Number 5 SEPTEMBER/OCTOBER 2014 ISPOR CONNECTIONS