27/02/2012
Systematic Reviews
Statistical tests for heterogeneity
Professor Declan Devane
Chair of Midwifery
National University of Ireland Galway
Email: [email protected]
Assessing statistical heterogeneity
•  If there is substantial heterogeneity among studies in
a systematic review, it might be inappropriate to do a
meta-analysis.
•  How do we know if there is ‘substantial’
heterogeneity?
1.  Visual inspection of a forest plot of studies included in the
review;
2.  Assessment of results of tests for statistical heterogeneity.
1
27/02/2012
Statistical tests for heterogeneity
•  Cochran Q (Chi-square, X2)
•  I2
•  Tau2
Cochran Q (Chi-square, x2)
•  It assesses whether observed differences in results are compa7ble with chance alone. •  If the Cochran X2 (Cochran Q) is sta7s7cally significant (usually p<0.1) there is definite heterogeneity. –  The level of significance for X2 (Cochrane Q) is oGen set at 0.1 due to the low power of the test to detect heterogeneity •  Rule of thumb: if X2 is bigger than df (which is 1 less than number of studies in forest plot) then heterogeneity is present 2
27/02/2012
Study or Subgroup
Bowsher 1997
Kieburtz 1998
Lampl 2002
Leijon 1989
Max 1988
Pilowsky 1982
Shlay 1998
Turkington 1980
Vrethem 1997
Watson 1982
Amitriptyline
Placebo
Events Total Events Total Weight
32
38
22
34 14.6%
23
34
9
41 12.4%
17
20
16
19 14.6%
10
15
1
15
4.7%
16
34
4
25
9.7%
4
12
3
12
7.7%
31
71
28
65 14.0%
19
19
0
20
2.7%
24
33
8
33 12.2%
16
24
2
24
7.2%
Risk Ratio
M-H, Random, 95% CI
1.30 [0.98, 1.73]
3.08 [1.65, 5.74]
1.01 [0.77, 1.32]
10.00 [1.46, 68.69]
2.94 [1.12, 7.73]
1.33 [0.38, 4.72]
1.01 [0.69, 1.49]
40.95 [2.65, 633.88]
3.00 [1.58, 5.68]
8.00 [2.06, 31.07]
Total (95% CI)
300
288 100.0%
Total events
192
93
Heterogeneity: Tau² = 0.43; Chi² = 59.65, df = 9 (P < 0.00001); I² = 85%
Test for overall effect: Z = 3.14 (P = 0.002)
Risk Ratio
M-H, Random, 95% CI
2.23 [1.35, 3.69]
0.01
0.1
1
10
100
Favours Placebo Favours Amitriptyline
Cochran Q (Chi-square)
•  The X2 (Cochran Q) test has low power to detect heterogeneity if there are few studies in the meta-­‐analysis and may, conversely, give a highly significant result if it comprises many large studies, even when the heterogeneity is unlikely to affect the conclusions. 3
27/02/2012
I2
•  An Index, I2, which does not depend upon the number of studies, the type of outcome data or the choice of treatment effect (e.g. rela7ve risk) can be used to quan7fy the impact of heterogeneity and assess inconsistency. •  I2 = 100x(Q – df)/Q I2
•  I2 represents the percentage of the total varia7on across studies due to heterogeneity; it takes values from 0% to 100%, with the value of 0% indica7ng no observed heterogeneity. If there is evidence of sta7s7cal heterogeneity, we should proceed cau7ously, inves7gate the reasons for its presence and modify our approach accordingly, perhaps by dividing the studies into subgroups of those with similar characteris7cs. 4
27/02/2012
Study or Subgroup
Bowsher 1997
Kieburtz 1998
Lampl 2002
Leijon 1989
Max 1988
Pilowsky 1982
Shlay 1998
Turkington 1980
Vrethem 1997
Watson 1982
Amitriptyline
Placebo
Events Total Events Total Weight
32
38
22
34 14.6%
23
34
9
41 12.4%
17
20
16
19 14.6%
10
15
1
15
4.7%
16
34
4
25
9.7%
4
12
3
12
7.7%
31
71
28
65 14.0%
19
19
0
20
2.7%
24
33
8
33 12.2%
16
24
2
24
7.2%
Risk Ratio
M-H, Random, 95% CI
1.30 [0.98, 1.73]
3.08 [1.65, 5.74]
1.01 [0.77, 1.32]
10.00 [1.46, 68.69]
2.94 [1.12, 7.73]
1.33 [0.38, 4.72]
1.01 [0.69, 1.49]
40.95 [2.65, 633.88]
3.00 [1.58, 5.68]
8.00 [2.06, 31.07]
Total (95% CI)
300
288 100.0%
Total events
192
93
Heterogeneity: Tau² = 0.43; Chi² = 59.65, df = 9 (P < 0.00001); I² = 85%
Test for overall effect: Z = 3.14 (P = 0.002)
Risk Ratio
M-H, Random, 95% CI
2.23 [1.35, 3.69]
0.01
0.1
1
10
100
Favours Placebo Favours Amitriptyline
I2
•  Thresholds for the interpretation of I2 can be
misleading, since the importance of
inconsistency depends on several factors. A
rough guide to interpretation is as follows:
–  0% to 40%: might not be important;
–  30% to 60%: may represent moderate
heterogeneity;
–  50% to 90%: may represent substantial
heterogeneity;
–  75% to 100%: considerable heterogeneity*.
5
27/02/2012
Tau2
•  aA estimate of the between-study variance in a
random-effects meta-analysis is given as (known as
tau-squared (T2 or Tau2)).
•  If > 1, suggests presence of substantial statistical
heterogeneity.
Study or Subgroup
Bowsher 1997
Kieburtz 1998
Lampl 2002
Leijon 1989
Max 1988
Pilowsky 1982
Shlay 1998
Turkington 1980
Vrethem 1997
Watson 1982
Amitriptyline
Placebo
Events Total Events Total Weight
32
38
22
34 14.6%
23
34
9
41 12.4%
17
20
16
19 14.6%
10
15
1
15
4.7%
16
34
4
25
9.7%
4
12
3
12
7.7%
31
71
28
65 14.0%
19
19
0
20
2.7%
24
33
8
33 12.2%
16
24
2
24
7.2%
Risk Ratio
M-H, Random, 95% CI
1.30 [0.98, 1.73]
3.08 [1.65, 5.74]
1.01 [0.77, 1.32]
10.00 [1.46, 68.69]
2.94 [1.12, 7.73]
1.33 [0.38, 4.72]
1.01 [0.69, 1.49]
40.95 [2.65, 633.88]
3.00 [1.58, 5.68]
8.00 [2.06, 31.07]
Total (95% CI)
300
288 100.0%
Total events
192
93
Heterogeneity: Tau² = 0.43; Chi² = 59.65, df = 9 (P < 0.00001); I² = 85%
Test for overall effect: Z = 3.14 (P = 0.002)
Risk Ratio
M-H, Random, 95% CI
2.23 [1.35, 3.69]
0.01
0.1
1
10
100
Favours Placebo Favours Amitriptyline
6
27/02/2012
7
27/02/2012
8
27/02/2012
Thank you
Professor Declan Devane
Chair of Midwifery
National University of Ireland Galway
Email: [email protected]
9