Sonya J.
Snedecor
Pharmerit
International
MULTIVARIATE METAANALYSIS: USE AND
APPLICATIONS
Workshop W8, Tuesday, June 5, 2014
ISPOR 19 th International Meeting, Montreal, QC, Canada
Joseph C.
Cappelleri
Pfizer Inc
Yin (Wendy)
Wan
Pharmerit
International
John W.
Stevens
University of
Shef field
LEARNING OBJECTIVES
 Understand the concept of MVMA and
why it can be useful
 Understand data requirements for
MVMA
 Understand limitations associated with
MVMA
2
INTRODUCTION TO
MULTIVARIATE METAANALYSIS (MVMA)
Sonya J.
Snedecor
Pharmerit
International
3
META-ANALYSIS:
THE STATISTICAL AGGREGATION OF DATA
FROM MULTIPLE SOURCES
 Confirmatory data analysis
 Estimate an average effect even when
studies have conflicting results
 Make predictions of effects in new studies
 Potential for greater ability to extrapolate
to general population
 Can examine sources of heterogeneity
among study results
 Considered to be high in the evidence
hierarchy
4
FREQUENTLY MULTIPLE OUTCOMES
ARE OF INTEREST
 Multiple outcomes or treatment effects
within studies
 E.g., systolic and diastolic blood pressure
 Multiple time points
 All outcomes measured within the same
patients are (usually) correlated
 Yet, we meta-analyze each endpoint
separately in separate (univariate) metaanalyses
5
UNIVARIATE META-ANALYSIS (UMA)
INDIVIDUAL ASSESSMENTS OF DIFFERENT
OUTCOMES
Mean treatment
effect of
Outcome 1
Mean treatment
effect of
Outcome 2
6
MULTIVARIATE META -ANALYSIS (MVMA)
MULTIPLE OUTCOMES ASSESSED
CONCURRENTLY*
Mean treatment
effect of
Outcome 1
Mean treatment
effect of
Outcome 2
*Fine print: additional data required
7
ADVANTAGES OF MVMA
Outcome
2
Outcome
1
 Utilizes information of correlated
outcomes
 Borrows information about missing
outcomes from other studies
 Describes the multivariate
relationship between endpoints
Joint estimate of
treatment effects
 Obtains pooled estimates with better
statistical properties
 Generates joint confidence regions
 Can model, test and make predictions
based on the joint association of
endpoints
 Estimates some function of the pooled
8
endpoints
THERE ARE SEVERAL REASONS
MVMA IS NOT MORE COMMON
Tradition
We do it this way, because that’s the
way it’s done…
Increased complexity of
multivariate approach
Parsimony is my friend…
Need for specialized statistical If it can’t be done in SAS, it can’t be
done…
software or knowledge
Lack of appreciation for the
consequences of ignoring
correlations between
endpoints
This workshop can help with that…
9
A CLOSER LOOK AT
MULTIVARIATE METAANALYSIS
Joseph C.
Cappelleri
Pfizer Inc
10
WHY DO CORRELATIONS MATTER?
Observed values
span entire range for
each variable
Observed values
span entire 2D space
Variable 2
Correlation : a mutual relationship or connection
between two or more variables
 Conceptually, this means that the plausible range
of values of one outcome is dependent on the value
of the other
Variable 1
11
WHY DO CORRELATIONS MATTER?
Correlation : a mutual relationship or connection
between two or more variables
 Conceptually, this means that the plausible range
of values of one outcome is dependent on the value
of the other
When correlated,
each variable spans
the entire range
Observed values are
related and clustered
Correlated
Variable 2
Uncorrelated
Variable 1
12
CORRELATION EXAMPLE
DISTRIBUTION OF BMI AND CHOLESTEROL
LEVELS IN THE UNITED STATES (1988 -1994)
Pr(BMI=31) = 7%
When BMI and TC
are independent,
equal probability of
having 120 mg/dL
and 240 mg/dL
Pr(120) = 15%
Pr(240) = 15%
Pr(TC=240) = 27%
Assume BMI=31
At given BMI,
probability
distribution of TC is
changed
Schwartz and Woloshin 1999 Eff Clin Pract (1st 3 figures)
13
CORRELATION TYPE 1:
WITHIN-STUDY CORRELATIONS
 Indicates the association between the
endpoints within a study
 Arises from each individual in a study
contributing data towards each endpoint (or
time point)
 Within-study correlations between endpoints
are seldom reported in trials
14
CORRELATION TYPE 2:
BET WEEN-STUDY CORRELATIONS

Indicates how the true underlying endpoint summar y values
are related across studies

Is induced by differences across studies such as age,
changes in study characteristics, and threshold levels in
diagnostic studies

In diagnostic studies, for example, sensitivity and specificity
are usually negatively correlated due to the use of different
thresholds (and even for the same threshold)

Can accurately describe the true bivariate relationship
between the true log odds in a treatment group and the true
log odds in a control group (baseline risk)
animation

Effect modification with baseline risk as proxy for severity of
illness
15
SURGICAL AND NON-SURGICAL
TREATMENTS FOR
PERIODONTAL DISEASE
Study 1
Study 2
Between-study
variance is much
larger than withinstudy variance
Study 3
Study 4
Study 5
UVMA
BVMA
-0.75
-0.55
-0.35
-0.15
Attachment level
0.05
0.25
0.45
Probing depth
Riley 2000 J R Statist Soc; Berkey et al. 1995 J Dent Res; Berkey et al. 1998 Statist Med
0.65
Small withinstudy variances
result in similar
UMVA and BVMA
estimates
16
SCHOLASTIC APTITUDE TEST
SCORES BETWEEN COACHED AND
UNCOACHED STUDENTS
Study 1
…
Large between-study
variance AND withinstudy variance
Study 7
UVMA
BVMA
-2
-1.5
-1
-0.5
Math scores
0
0.5
1
Verbal scores
1.5
Large variances
2
result
in differences
between UMVA and
BVMA
Riley 2000 J R Statist Soc; Gleser and Olkin 1994 in The Handbook of Res Synth
17
CORRELATED OUTCOMES
EXAMPLE OF DIAGNOSTIC TESTS
Sensitivity and specificity are measures of the accuracy the diagnostic
test



Sensitivity: Pr(test = + | disease = +)
Specificity: Pr(test = – | disease = –)
For ever y test, there is one pair of sensitivity and specificity values


Within-study correlation not important because they are measured in different
subgroups (specificity in disease “ –” patients and sensitivity in disease “+”
patients)
Between-study correlation is important: sensitivity and specificity are
negatively correlated

Other sources of heterogeneity between studies may also be present



Thresholds to define “+” and “ –” test results (explicit)
Observers, laboratories, or equipment (implicit)
To pool data from multiple studies



One approach: combine data using standard methods for proportions
However, ignoring the correlation of the outcomes between studies is inappropriate
18
CORRELATED OUTCOMES
EXAMPLE OF DIAGNOSTIC TESTS

BVMA of sensitivity and specificity estimates…





Between-study variations (heterogeneity) separately
95% confidence intervals for each outcome
Confidence ellipse around the means of sensitivity and specificity
Also, the predictive ellipse for sensitivity and specificity for
individual studies
BVMA can also…



Obtain functions of sensitivity and specificity such as the
diagnostic odds ratio
Depict summary receiver operating characteristic (ROC) curve
Examine covariates with potentially separate effects on sensitivity
and specificity (e.g., for two or more diagnostic technologies)
19
EXAMPLE: DIAGNOSIS OF LYMPH
NODE METASTASES
Specificity of LAG
is significantly
lower than that of
CT (p=0.0002) and
MRI (p=0.0001)
LAG has the
highest sensitivity
Imaging test
Sensitivity
Specificity
Diagnostic OR*
Lymphangiography (LAG)
0.67 (0.57, 0.76)
0.80 (0.73, 0.85)
8.13 (5.16, 12.82)
Computed tomography
(CT)
0.49 (0.37, 0.61)
0.92 (0.88, 0.95)
11.34 (6.66, 19.30)
Magnetic resonance
imaging (MRI)
0.56 (0.41, 0.70)
0.94 (0.90, 0.97)
21.42 (10.81, 42.45)
DOR suggesting
MRI is much better
than others can be
misleading
*Diagnostic OR = Ratio of odds of true “+” relative to odds of false “+”
Reitsma et al. 2005 J Clin Epi
20
BIVARIATE SUMMARY ESTIMATES
AND 95% CONFIDENCE ELLIPSES
For each modality,
the region
containing the
likely combination
of the mean values
Confidence ellipses clearly
show the differences in
sensitivity and specificity of
LAG compared with CT and
MRI.
Reitsma et al. 2005 J Clin Epi
21
USING CORRELATIONS IN MVMA

It allows a joint synthesis of the multiple endpoints

It accounts for any correlation between endpoints that may
exist both within studies and between studies



Useful for sample estimates of treatment effect
Correlations allow a borrowing of strength across endpoints

Generally leads to corrected pooled estimates and standard errors

Standard errors tend to be lower relative to those from URMA

Especially when at least some endpoints are missing at random
across studies

Narrower joint regions or distributions of effect
MVMA rever ts to separate UVMA of each endpoint when all
correlations are zero and one endpoint does not borrow
strength from another endpoint
22
PRACTICAL EXAMPLES:
THE BIVARIATE CASE
Yin (Wendy)
Wan
Pharmerit
International
MVMA SOFTWARE

SAS


WINBUGS


Proc mixed was used in MVMA over a decade ago
Markov chain Monte Carlo (MCMC) method
STATA

mvmmeta, with options:
 wscorr(expression) – sensitivity analysis over a range of correlations
 wscorr(riley) – fit the Alternative Method as suggested by Riley et al.

R package: mvmeta
( ht t p: / / c ra n . r - pro j e c t . o rg / ) , mmeta( L uo , et al . 2 01 4 . J
St at So f t ware ), metaSEM( C heu ng , 2 01 3 ).
24
SAS EXAMPLE
DIAGNOSTIC TEST
proc mixed data=bi_meta method=reml cl;
class study_id modality;
model logit = dis*modality non_dis*modality /
noint cl df=1000, 1000, 1000,
1000, 1000, 1000;
Obtain estimates of mean
sensitivity and specificity
for each treatment
random dis non_dis /
subject=study_id type=un;
Random define between study covariance matrix
repeated / group=rec;
Repeat defines within-study covariance matrix
parms / parmsdata=cov
hold=4 to 91;
Dataset cov provides the within-study variance
contrast ‘CT_sens vs LAG_sens’
dis*modality 1 -1 0 /df=1000;
Use contrast statement for testing hypotheses for
differences in sensitivities and specificities
run;
25
Reitsma et al. 2005 J Clin Epi
SAS EXAMPLE
DIAGNOSTIC TEST
From the proc mixed model, we want to get
- means of sensitivity and specificity
- Variance-covariance matrix which combines the
between-study and within study variance-covariance
Means of log sensitivity and specificity
Variance-covariance matrix
26
SAS EXAMPLE
PERIODONTAL DISEASE
Estimate the means of the
outcomes
Set up the between-trial
variance-covariance matrix
Set up the within-trial
variance-covariance matrix
• Starting values for between-trial
variances and covariance
• Variances within each trial
• Covariances within each trial
27
Van Houwelingen et al. 2002 Stat Med
SAS EXAMPLE
PERIODONTAL DISEASE
Means of outcomes 1 and 2
Variance-covariance matrix
28
WINBUGS EXAMPLE
TREATMENTS FOR PERIODONTAL DISEASE
KNOWN COVARIANCES
model{ #Random-effects
for (i in 1:noStudies){
weight1[i]<-1/var1[i]
rho[i]<-cov12[i]/sqrt(var1[i]*var2[i]) #Within-study correlation
rho[i]<-cov12[i]/sqrt(var1[i]*var2[i])
#Within-study
y[i,1]~dnorm(m[i,1],weight1[i])
correlation
y[i,2]~dnorm(m2_prime[i],weight2[i])
m2_prime[i]<-m[i,2]+rho[i]*
#Mean of y2 | observed y1
sqrt(var2[i])/sqrt(var1[i])*(y[i,1]-m[i,1])
Specify within-study
var2_prime[i]<-var2[i]*(1-pow(rho[i],2))#Var of y2 | observed y1
correlation
as a function of
weight2[i]<-1/var2_prime[i]
the known covariance
}
}
for (i in 1:noStudies) {
m[i,1]~dnorm(mean[1],prec[1])
m[i,2]~dnorm(m.re[i],prec.vre)
m.re[i]<-mean[2]+rho.tau*sqrt(tau.sq[2])/sqrt(tau.sq[1])*
(m[i,1]-mean[1]) #Mean of effect 2 | effect 1
}
v.re<-tau.sq[2]*(1-pow(rho.tau,2)) #Var of effect 2 | effect 1
Prior specifications…
Mavridis and Salanti 2013 Stat Methods Med Res
29
WINBUGS EXAMPLE
TREATMENTS FOR PERIODONTAL DISEASE
KNOWN COVARIANCES
model{ #Random-effects
for (i in 1:noStudies){
weight1[i]<-1/var1[i]
rho[i]<-cov12[i]/sqrt(var1[i]*var2[i]) #Within-study correlation
y[i,1]~dnorm(m[i,1],weight1[i])
y[i,2]~dnorm(m2_prime[i],weight2[i])
y[i,1]~dnorm(m[i,1],weight1[i])
m2_prime[i]<-m[i,2]+rho[i]*
#Mean of y2 | observed y1
sqrt(var2[i])/sqrt(var1[i])*(y[i,1]-m[i,1])
y[i,2]~dnorm(m2_prime[i],weight2[i])
var2_prime[i]<-var2[i]*(1-pow(rho[i],2))#Var
observed y1
y1
m2_prime[i]<-m[i,2]+rho[i]*
#Mean of of
y2 y2
| |observed
weight2[i]<-1/var2_prime[i]
sqrt(var2[i])/sqrt(var1[i])*(y[i,1]-m[i,1])
}
}
for (i in 1:noStudies) {
m[i,1]~dnorm(mean[1],prec[1])
Specify likelihood of 1st
m[i,2]~dnorm(m.re[i],prec.vre)
outcome, then conditional
m.re[i]<-mean[2]+rho.tau*sqrt(tau.sq[2])/sqrt(tau.sq[1])*
likelihood
of 2nd outcome
(m[i,1]-mean[1])
#Mean of effect 2 | effect 1
}
v.re<-tau.sq[2]*(1-pow(rho.tau,2)) #Var of effect 2 | effect 1
Prior specifications…
Mavridis and Salanti 2013 Stat Methods Med Res
30
WINBUGS EXAMPLE
TREATMENTS FOR PERIODONTAL DISEASE
KNOWN COVARIANCES
model{ #Random-effects
for (i in 1:noStudies){
weight1[i]<-1/var1[i]
rho[i]<-cov12[i]/sqrt(var1[i]*var2[i]) #Within-study correlation
Specify (conditional)
y[i,1]~dnorm(m[i,1],weight1[i])
y[i,2]~dnorm(m2_prime[i],weight2[i]) random effects means
m2_prime[i]<-m[i,2]+rho[i]*
#Mean of y2 | observed y1
sqrt(var2[i])/sqrt(var1[i])*(y[i,1]-m[i,1])
var2_prime[i]<-var2[i]*(1-pow(rho[i],2))#Var of y2 | observed y1
weight2[i]<-1/var2_prime[i]
for (i in 1:noStudies) {
}
m[i,1]~dnorm(mean[1],prec[1])
}
for m[i,2]~dnorm(m.re[i],prec.vre)
(i in 1:noStudies) {
m.re[i]<m[i,1]~dnorm(mean[1],prec[1])
mean[2]+rho.tau*sqrt(tau.sq[2])/sqrt(tau.sq[1])*
m[i,2]~dnorm(m.re[i],prec.vre)
(m[i,1]-mean[1]) #Mean of effect 2 | effect 1
m.re[i]<-mean[2]+rho.tau*sqrt(tau.sq[2])/sqrt(tau.sq[1])*
}
(m[i,1]-mean[1]) #Mean of effect 2 | effect 1
}
v.re<-tau.sq[2]*(1-pow(rho.tau,2)) #Var of effect 2 | effect 1
Prior specifications…
Mavridis and Salanti 2013 Stat Methods Med Res
31
WINBUGS EXAMPLE
TREATMENTS FOR PERIODONTAL DISEASE
UNKNOWN COVARIANCE
model{ #Random-effects
for (i in 1:noStudies){
weight1[i]<-1/var1[i]
rho[i]~dbeta(1,1)
#Uniformprior
prior
[0,1]
rho[i]~dbeta(1,1) #Uniform
on on
[0,1]
y[i,1]~dnorm(m[i,1],weight1[i])
y[i,2]~dnorm(m2_prime[i],weight2[i])
m2_prime[i]<-m[i,2]+rho[i]*
#Mean of y2 | observed y1
sqrt(var2[i])/sqrt(var1[i])*(y[i,1]-m[i,1])
var2_prime[i]<-var2[i]*(1-pow(rho[i],2))#Var
| study
observed y1
Unknown rhoof
for y2
each
weight2[i]<-1/var2_prime[i]
are given prior distributions
}
}
for (i in 1:noStudies) {
m[i,1]~dnorm(mean[1],prec[1])
m[i,2]~dnorm(m.re[i],prec.vre)
m.re[i]<-mean[2]+rho.tau*sqrt(tau.sq[2])/sqrt(tau.sq[1])*
(m[i,1]-mean[1]) #Mean of effect 2 | effect 1
}
v.re<-tau.sq[2]*(1-pow(rho.tau,2)) #Var of effect 2 | effect 1
Prior specifications…
Mavridis and Salanti 2013 Stat Methods Med Res
32
PRACTICAL SUMMARY
SAS Proc Mixed
WinBUGS
Convenient
Requires knowledge of distribution
functions (e.g. prior, likelihood, etc.)
Flexibility in covariance structure
NA
Requires within-study correlations to
be known
Does not require known within-study
correlation estimates
Provides estimates of population
means and between-trial covariance
matrix
Provides posterior distributions for all
parameters
Larger standard deviations with few
studies
Different prior specifications can
lead to different results, especially
when there are only few studies
33
MVMA LIMITATIONS
AND OTHER
CONSIDERATIONS
Or the harbinger of gloom
John W.
Stevens
ScHARR,
University of
Sheffield
Source: http://britcomgreats.blogspot.co.uk/2012/11/one-foot-in-grave.html
34
TWO PERSPECTIVES
One view:
“In the realm of research synthesis . . . the consequences of accounting for
(modeling) dependence or ignoring it are not well understood” (Becker et
al., 2004).
Another view:
“Psychologist: one who thinks no univariate data are Normally distributed
but that all multivariate data are.” Senn, S
“Multivariate analysis will bring you many dimensions but you will not enjoy
them.” Senn, S
“Multivariate analysis: such pretty pictures.” Senn, S
35
BACK TO BASICS (1)
 Univariate data can be analysed exactly according
to an appropriate likelihood function for each
treatment arm

E.g. a Binomial distribution for the number of observed
events out of the number of patient randomised
 Multivariate meta-analyses generally resort to
analysing study -specific treatment effects

E.g. the sample log odds ratio depending on the estimate of
the variance of the log odds ratio
 Study-specific treatment effects are then assumed to
be (multivariate) normally distributed
36
BACK TO BASICS (2)
 The performance associated with study specific treatment effects will depend on:
 Sample size (i.e. Central Limit Theorem)
 How close the probability of an event is to 0 or 1
 The presence of zero values
37
WHAT IS THE RESEARCH QUESTION
AND WHAT ARE THE RESULTS FOR?
 Comparative effectiveness
 In this case the main question may be:
“Is there evidence of a treatment effect and can the
treatment work in patients not included in the study?”
 Parameter estimates may be only slightly changed
unless there is substantial missing data
 The effort involved in a MVMA may not be
worthwhile
38
WHAT IS THE RESEARCH QUESTION
AND WHAT ARE THE RESULTS FOR?
 Health technology assessment

In this case the objective may be to characterise the joint
distribution for absolute treatment effects such as:
 The mean systolic and diastolic blood pressure on each treatment
arm
 The proportion of patients in each ACR and EULAR category on
each treatment arm
 The mean time to disease progression and death on each
treatment arm

Allowing for correlation between parameters can
substantially reduce uncertainty in output parameters
(Briggs et al., 2006)

Characterising the joint uncertainty about input parameters
and propagating the uncertainty through to the output may
affect ICERs and expected net benefit, particularly in non linear models
39
PROBLEMS WITH WITHIN-STUDY
CORRELATION

Estimating within-study correlations would not be an issue if
individual patient -level data were available for each study


In practice, this is unrealistic and we generally deal with published
data for some or all studies
When within-study correlations are unavailable solutions
include:

Impute within-study correlations from available studies (ignores
uncertainty)

Perform sensitivity analyses over a range of assume values for the
within-study correlations (computer intensive)

Use a Bayesian approach (i.e. multiple imputation) based on an
informative prior distribution for the within -study correlations using
external data or expert opinion (coherent and transparent )
40
OTHER LIMITATIONS

Publication bias


Ordered categorical data


If the objective is to correlate primary and secondary outcomes
and secondary outcomes are not missing at random then this may
bias both primary and secondary effects
We can correlate treatment effects from multiple ordered
categorical data outcomes but it is not clear (to me) how to use the
estimates from a MVMA to generate absolute treatment effects
Time-to-event

We can correlate (log) hazard ratios for multiple outcomes such as
OS and PFS assuming that hazards are proportional, else absolute
effects such as mean PFS and OS will be biased (as in the
univariate case)
41
SUMMARY

“Except when within-study variation is ver y small relative to
between–study variation, the within -study correlation plays a
crucial role in the amount or borrowing of strength.” Riley
RD, 2009

The question being addressed affects the appropriateness of
the scale of measurement (as with univariate analyses)

An impor tant feature of multivariate meta -analyses is
dealing with within-study correlation when they are unknown

Diagnostic test accuracy is a special case of a multivariate
meta-analysis in which sensitivity and specificity are
correlated between studies but there is no within -study
correlation.
42
REFERENCES

B e r k ey e t a l . 1 9 9 5 . M u l t i p l e - o u t c o m e s m e t a - a n a l y s i s o f t r e a t m e n t s f o r p e r i o d o n t a l d i s e a s e .
J o u r n a l o f D e n t a l Re s e a r c h 74 : 1 0 3 0 - 1 0 3 9

B e r k ey e t a l . 1 9 9 8 . M e t a - a n a l y s i s o f m u l t i p l e o u t c o m e s b y r e g r e s s i o n w i t h r a n d o m e f f e c t s .
S t a t i s t i c s i n M e d i c i n e 17: 2 5 37 - 2 5 5 0

C h e u n g , M i k e W - L . 2 01 3 . m e t a S E M : A n R P a c k a g e f o r M e t a - A n a l y s i s u s i n g S t r u c t u r a l
Equation Modeling.

G l e s e r a n d O l k i n . 1 9 9 4 S to c h a s t i c a l l y d e p e n d e n t e f f e c t s i z e s . I n T h e H a n d b o o k o f R e s e a r c h
Syntheses (eds Cooper and Hedges)

J a c k s o n D e t a l . M u l t i va r i a te m e t a - a n a l y s i s . P o te n t i a l a n d p r o m i s e . S t a t i s t i c s i n M e d i c i n e
2 01 1 3 0 : 24 81 - 24 9 8

L u o e t a l . 2 01 4 . m m e t a : A n R P a c k a g e f o r M u l t i va r i a te M e t a - A n a l y s i s . J o u r n a l O f S t a t i s t i c a l
Software 56(11): 1-26.

M av r i d i s a n d S a l a n t i . 2 01 3 A p r a c t i c a l i n t r o d u c t i o n to m u l t i va r i a te m e t a - a n a l y s i s . S t a t
M e t h o d s M e d Re s 2 2 : 1 3 8 - 1 5 8

Re i t s m a J B e t a l . 2 0 0 5 . B i va r i a te a n a l y s i s o f s e n s i t i v i t y a n d s p e c i f i c i t y p r o d u c e s i n f o r m a t i o n
s u m m a r y m e a s u r e s i n d i a g n o s t i c r ev i e w s . J o u r n a l o f C l i n i c a l E p i d e m i o l o g y 5 8 : 9 8 2 - 9 9 0 .

R i l ey R D e t a l . 2 0 07. B i va r i a te r a n d o m - e f f e c t s m e t a - a n a l y s i s a n d t h e e s t i m a t i o n o f b e t w e e n s t u d y c o r r e l a t i o n . B M C M e d i c a l Re s e a r c h M e t h o d o l o g y 7: 3

R i l ey R D . 2 0 0 9 . M u l t i va r i a te m e t a - a n a l y s i s : t h e e f f e c t o f i g n o r i n g w i t h i n - s t u d y c o r r e l a t i o n .
J o u r n a l o f t h e Roy a l S t a t i s t i c a l S o c i e t y A 17 2 ( 4 ) : 7 8 9 - 81 1 .

Va n H o u w e l i n g e n H C . , 2 0 0 2 . A d va n c e d m e t h o d s i n m e t a ‐ a n a l y s i s : m u l t i va r i a te a p p r o a c h a n d
43
m e t a ‐ r e g r e s s i o n S t a t i s t i c s i n M e d i c i n e 21 : 5 8 9 - 6 24