Meta Analysis A family of (statistical) procedures used to combine or compare results of multiple studies Analysis of Research • Primary : original analysis of data • Secondary : reanalysis of old data • Meta : analysis of analyses (Systematic Review) “ Vote Counting Vote Counting ” ” An Example (Treat A vs Treat B): • 100 studies n=30 each • if medium effect size (will discuss) • then 33% (p<.05) 67% (NS) • conclusion: not significant (x wrong!) • hence, no “vote counting” Hypothetical Example A B C D Con n mean 41 11 29 225 104 9 11 23 Exp n mean t 41 17 2.72 ** 33 175 1.95 98 12 2.03 * 11 31 1.56 Note : very different means (diff instruments?) Calculation Steps t df 1sided p A 2.72 80 B 1.95 60 C 2.03 200 D 1.56 20 S 4.36 Z 2 log e p .004 2.65 .972 1.91 .022 2.02 .067 1.50 11.04 0.06 7.65 5.40 4.26 24.15 Combined Test Fisher : chisq = 2 S log e p df = 2 n where p : 1sided pvalue in each test n : number of tests Results Fisher : chisq = 24.15 df = 2 x 4 = 8 p = 0.0022 Stouffer : Z = 4.26 / sqrt (4) = 2.13 p = 0.0332 Winer : Z = 4.36 / sqrt [(80/78) + …+(20/18)] = 2.13 p = 0.0332 Questions • Weighting ? e.g. sample size ? • Measure of significance => pvalue • Measure of magnitude => ???? i.e, degree null hypothesis rejected ??? The Hypothetical Example Con Exp ES = M 1 M 2 sd (M 2 M 1) ¸ sd A 11 17 10 0.60 B 225 175 100 0.50 C 9 12 7 0.43 D 23 31 12 0.75 ES : effect size (GLM) ave = 0.32 Interpretation of ES • ES = 0.43 implies (for 2 groups) treatment mean is 0.43 SD units above control mean • from normal table, 0.43 SD = 67%tile • i.e., an average person in treatment group is better than 67% people in control group Types of “ Effect Size Effect Size ” ” ES (specific) : standardized difference ES (Generic) : • correlation coefficient • odds ratio / relative risk • difference in proportion Different “types” in the same metaanalysis ? Steps in Meta Analysis Analysis • Get results of each study into common metric – effect size (continuous data) – correlation coefficient (continuous) – odds ratio, relative risk (categorical) • Combine results to form summary statistics • Test overall significance • Compare whether individual results are homogeneous (software demo) Using SPSS (odds ratio) Study 1 I C Outcome Yes 10 20 No 30 40 Total 100 100 Study 2 I C Outcome Yes 25 44 No 175 156 Total 200 200 Using SPSS (data set up) up) Study 1 1 1 1 2 2 2 2 Row 1 1 2 2 1 1 2 2 Col 1 2 1 2 1 2 1 2 Data 10 20 30 40 25 44 175 156 Output * BreslowDay test of homogeneity * MantelHaenszel common odds ratio Question !! Study X Y 1 .. .. 1 .. .. 1 .. .. 1 .. .. 2 .. .. 2 .. .. 2 2 .. .. .. .. How to get common correlation coefficient using SPSS when raw data available ? 160 140 120 100 80 60 Y 40 20 20 X 40 60 80 100 120 140 Partial correlation a Coefficients Unstandardized Standardized Coefficients Coefficients Model B Std. Error Beta t 1 (Constant) 44.447 16.714 2.659 X .824 .135 .742 6.097 S 26.013 12.313 .257 2.113 a.Dependent Variable: Y Correlations Sig. Zeroorder Partial Part .017 .000 .997 .828 .100 .050 .993 .456 .035 Fixed vs Random Effects Models • Fixed, which assumes all studies – no heterogeneity (can be tested using Cochran test) (confidence interval artificially narrower if not) – estimating a single true underlying effect size • Random – assumes different underlying effect sizes – more conservative (because 2 variations) Difficulties • Rounded pvalue – e.g., p<0.05; NS; “**”, … • Calculating effect size (ES) from graph • ES calculated using different methods – E.g., t in 2 studies; chisq in 4 studies • Precise sample size not available – e.g. missing not reported for each variable Limitations of Meta Limitations of Meta Analysis • Published research biased in favour of significant findings => biased metaanalysis results • Difficult to have research with exactly the same measuring techniques, definitions of variables, … Publication Bias – Funnel Plot 6 5 Inverse SE 4 3 2 1 0 3 2 1 0 Log Odds Ratio 1 2 3 “ Classic Classic ” ” Publication Bias 6 5 Inverse SE 4 3 2 1 0 3.0 2.0 1.0 0.0 Log OR 1.0 2.0 3.0 Publication Bias – ““ Adjustment Adjustment ” • analyse only large studies • include unpublished studies – risky, quality of study questionable – Cook (1993) : 46/150 MA used unpublished results – 47% editors : OK to include – 30% : should not be included • prospective registration of trials • change in publication process Sensitivity Analysis Impact of followings on Effect Size estimate • changing inclusion criteria • include and exclude unpublished studies • exclude small studies especially if publication bias suspected • quality : include only those judged “pass” by at least 2 raters Study Quality • How to assess study quality ? • = = Critical appraisal ! • Quality Scoring System • Checklist • Workshop in Critical Appraisal • 20 24 January 2003 • Preview : next 4 slides Problems ( Riet Riet et al, 1990) • chronic pain & acupuncture (effective ?) • could do metaanalysis : 51 papers • check methodology (next 3 slides) • maximum possible score : 100 • only 11 papers scored > 50 • highest score : 62 • poor quality (methodology) ! • metaanalysis, if done, result reliable ?? Methodological Criteria I • Comparability of Prognosis (35) – Homogeneity – Prestratification – Randomization – Compare relevant baseline variables shown – n >= 50 per group – loss to followup <= 20 3 3 12 2 10 5 Methodological Criteria II • Adequate Intervention (30) – avoidance of DNIC – adeq description of acupun procedures – quality of clinician(s) / therapist(s) – existing treatment in control group 2 10 15 3 • Data Presentation (5) – Readers can do inferential stat 5 Methodological Criteria III • Adequate Effect Measurement (30) – patients blinded – evaluator blinded – followup >= 3 months – pain as an outcome measure – use of medication – ADL – Remarks on sideeffects 10 5 5 3 2 3 2 References Articles : • Gotzsche, Hammarquist & Burr; BMJ; 1998 • Schafer; Measu & Eval in Couns & Develop; 1999 • References from these 2 articles Books • Glass, McGaw & Smith; 1981 • Wolf; 1986 • Cook et al; 1994
© Copyright 2026 Paperzz