Exercise 2-Effect Size Coding
1
DIRECTIONS: COMPLETE THIS EXERCISE ON YOUR
COMPUTER AND UPLOAD YOUR ANSWERS TO CANVAS. YOU
SHOULD USE BORENSTEIN CHAPTER 4 AND THE FORMULAS
THAT I’VE GIVEN YOU TO ANSWER QUESTIONS 1-6. NOTE
THAT QUESTION 7 ASKS ABOUT CODING A PARTICULAR
STUDY FROM YOUR MODEL META-ANALYSIS.
NAME:
ACKNOWLEDGE COLLABORATORS:
Cohen’s d
2
 Example from Kim & Quinn(2013): Schacter & Jo (2005)

Discovered through electronic search
Question (1) Compute Swithin and d. Note that the parentheses
below are the standard deviations.
3
 Example: Schacter & Jo (2005)
 𝑆𝑤𝑖𝑡ℎ𝑖𝑛 =
 If only the overall sample size is reported for an
experiment, assume treatment/control sample sizes
are equal
 d=
Question (2) Compute 𝑉𝑑 and 𝑆𝐸𝑑
4
 Example: Schacter & Jo (2005)
 Vd =
 𝑆𝐸𝑑 =
Question (3) Compute J and Hedge’s g
5
 Example: Schacter & Jo (2005)
 𝐽=
 𝑔=
Covariate-adjusted Cohen’s d
6
 Sometimes a study might report a covariate-adjusted
effect instead of unadjusted descriptive statistics
𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑
 𝑑𝑎𝑑𝑗 =
𝑌1
𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑
− 𝑌2
𝑆𝑤𝑖𝑡ℎ𝑖𝑛𝑎𝑑𝑗
,
where
 𝑆𝑤𝑖𝑡ℎ𝑖𝑛𝑎𝑑𝑗 =
𝑆𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑
1−𝑅 2
,
where 𝑆𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑 is the covariate adjusted standard deviation
(Borenstein, 2009)
Covariate-adjusted Cohen’s d
7
 Alternatively, you can estimate 𝑑𝑎𝑑𝑗 with the
treatment coefficient in a regression table if the
outcome used in the regression has been
standardized
 In a randomized trial, covariates are added to
improve the precision of the treatment effect
estimate, not to reduce bias. In an RCT, d and 𝑑𝑎𝑑𝑗
should be similar (though the standard errors will
not be).
SE for Covariate-adjusted Cohen’s d
8
 Remember that the SE for an ES must account for
sampling variability not only in the means, but also
in the pooled sd.
𝑛1 + 𝑛2 1 − 𝑅2
𝑑2
𝑉𝑑𝑎𝑑𝑗 =
+
𝑛1 𝑛2
2(𝑛1 + 𝑛2 )
 If you are taking 𝑑𝑎𝑑𝑗 from a regression coefficient,
the SE on that coefficient will underestimate 𝑉𝑑 𝑎𝑑𝑗
because it does not account for sampling variability
in the sd.
Question (4) Compute SE for Covariate-adjusted
Cohen’s d from Kim (2006)
9
 Example from Kim (2006)

“discovered” through electronic search
 𝑆𝐸𝑑𝑎𝑑𝑗 =
Question (5) Compute Covariate-adjusted
Hedge’s g from Kim (2006)
10
 Use the same Hedge’s g formula as before, with an
adjustment to J for the loss of additional degrees of
freedom by the inclusion of covariates
where 𝑑𝑓 = 𝑛1 + 𝑛2 − 2 − 𝑞,
where q is the number of covariates in the model.
 Hedge’s g =
Recovering an ES using the t-statistic
11
 If a study reports just a t-statistic without
means/standard deviations, the ES can be estimated
using the following formula:
 𝐸𝑆 = 𝑡
𝑛1 +𝑛2
𝑛1 𝑛2
Question (6) Compute an effect size using the tstatistic from Allington et al. (2010)
12
 Example from Kim & Quinn (2013): Allington et al. (2010)

Discovered through electronic search
“Our first comparison tested the hypothesis that the FCAT
performances of the treatment students would exceed those of the
control group. A t-test found statistically significant differences (t =
2.434, df = 1,328, p = .015) in the performance of the treatment
and control students on the FCAT administered after three
consecutive summer book distributions.” (p. 420)
𝐸𝑆 (𝑢𝑠𝑖𝑛𝑔 𝑡ℎ𝑒 𝑡 − 𝑠𝑡𝑎𝑡𝑖𝑠𝑡𝑖𝑐 𝑎𝑏𝑜𝑣𝑒) = show the computations you
made:
Question (7) Using 1 of your primary studies from your model metaanalysis, explain how you coded the effect size.
13
 For the final question, choose one of the primary studies




included in your model meta-analysis. Provide enough detail
so that the second rater could replicate the effect size coding.
1. Compute an effect size from your primary study. How,
exactly, did you compute the effect size. In detail, tell me the
relevant text or numbers (and page number) you used to
generate the effect size.
2. What questions arose as you did this exercise?
3. How long did it take you to code one effect size? Please note
that everyone will have a very different study to code. I tried
coding most all of the effect sizes and am familiar with most of
the issues that will arise, but I want you to get a sense of why
some studies are easy to code (while others are difficult).
In class next week, I will ask you to share your answers to 1-3
above.
Type your Answers to Q7 here
(add more slides if you need space)
14
References
15
 Allington, R. L., McGill-Franzen, A., Camilli, G., Williams, L., Graff, J., Zeig, J., &





Nowak, R. (2010). Addressing summer reading setback among economically
disadvantaged elementary students. Reading Psychology, 31, 411–427.
doi:10.1080/02702711.2010.505165
Borenstein, M., Hedges, L. V., Higgins, J. P. T., Rothstein, H. R. Introduction to
Meta-Analysis. West Sussex, UK: John Wiley & Sons, Lt.
Jacob, B. A., & Lefgren, L. (2004). Remedial education and student achievement: A
regression-discontinuity analysis. Review of Economics and Statistics, 68, 226–
244. doi:10.1162/003465304323023778
Kim, J. S. (2006). Effects of a voluntary summer reading intervention on reading
achievement: Results from a randomized field trial. Educational Evaluation and
Policy Analysis, 28, 335–355. doi:10.3102/01623737028004335
Kim, J.S., & Quinn, D.M. (2013). The effects of summer reading on low-income
children’s literacy achievement from kindergarten to grade 8: A meta-analysis of
classroom and home interventions. Review of Educational Research , 83(3) 386431. DOI: 10.3102/0034654313483906
Schacter, J., & Jo, B. (2005). Learning when school is not in session: A reading
summer day-camp intervention to improve the achievement of exiting first-grade
students who are economically disadvantaged. Journal of Research in Reading, 28,
158–169. doi:10.1111/j.1467-9817.2005.00260.x