EDF 6486
Research Methods in Education: Experimental Design & Analysis
Homework Due March 26, 2012
Solutions
Hinkle, et al, Chapter 19
1. A behavioral psychologist is interested in whether goal-setting activities can be
applied successfully in public school classrooms. The psychologist randomly selects 39
middle-school students and randomly assigns them to three different groups that are
assigned different levels of goal difficulty. The dependent variable is performance on a
verbal-fluency task. Before the study, each student is given a general-ability test, whose
scores are used as the covariate. For the data below, use α = .05 in this data analysis.
Verbal General
fluency ability
Group
(Y)
(X)
1
42
16
1
40
16
n = 10
1
48
19
ΣY = 457
1
55
24
Y = 45.7
1
45
20
ΣY2 = 21113
1
51
22
ΣXY = 9110
1
47
20
SSY = 281.60
1
38
23
r = .50
1
46
20
1
45
18
2
51
25
n = 10
2
50
18
ΣY = 512
2
45
17
Y  51.2
2
52
19
ΣY2 = 26496
2
47
17
ΣXY = 10541
2
50
20
SSY = 281.60
2
62
23
r = .67
2
60
24
2
48
21
2
47
20
3
46
20
3
47
21
3
43
19
n = 10
3
54
26
ΣY = 479
3
49
21
Y  47.9
3
44
25
ΣY2 = 23105
3
53
24
ΣXY = 10559
3
52
23
SSY = 160.90
3
42
18
r = .70
3
49
22
ΣX = 198
X  19.8
ΣX2 = 3986
SSX = 65.60
ΣX = 204
X  20.4
ΣX2 = 4234
SSX = 72.40
ΣX = 219
X  21.9
ΣX2 = 4857
SSX = 60.90
As the authors point out, the computations for an analysis of covariance are onerous and
time consuming. So, we will use SPSS to answer this question.
a. Complete an ANOVA on both the covariate (general ability) and the dependent
variable (verbal fluency).
The source table shown below gives the ANOVA for general ability.
Tests of Between-Subjects Effects
Dependent Variable: General ability
Type III Sum
of Squares
df
Mean Square
23.400(a)
2
11.700
12854.700
1
12854.700
23.400
2
11.700
198.900
27
7.367
13077.000
30
222.300
29
a R Squared = .105 (Adjusted R Squared = .039)
Source
Corrected Model
Intercept
method
Error
Total
Corrected Total
F
1.588
1744.982
1.588
Sig.
.223
.000
.223
Partial Eta
Squared
.105
.985
.105
Note that the null hypothesis that the mean general ability scores for students taught
with the different teaching methods are all equal is not rejected (p = .223). This is
what we want to see. It tells us that the covariate is not affected by the treatment
and this is one of the assumptions of ANCOVA.
Next, the source table for verbal fluency.
Tests of Between-Subjects Effects
Dependent Variable: Verbal fluency
Type III Sum
of Squares
df
Mean Square
153.267(a)
2
76.633
69890.133
1
69890.133
153.267
2
76.633
670.600
27
24.837
70714.000
30
823.867
29
a R Squared = .186 (Adjusted R Squared = .126)
Source
Corrected Model
Intercept
method
Error
Total
Corrected Total
F
3.085
2813.948
3.085
Sig.
.062
.000
.062
Partial Eta
Squared
.186
.990
.186
Note that the null hypothesis that the mean verbal fluency score for students taught
with the different teaching methods are all equal is not rejected (p = .062). The
teaching method used does not seem to have an effect on students’ verbal fluency
scores.
b. Test the assumption of homogeneity of regression.
We can assume that there is homogeneity of variance if there is no interaction between
the covariate and the independent variable. So, we will now test the null hypothesis that
no such interaction exists. The source table for this ANOVA is given below.
Tests of Between-Subjects Effects
Dependent Variable: Verbal fluency
Type III Sum
of Squares
df
Mean Square
416.510(a)
5
83.302
278.050
1
278.050
1.353
2
.676
253.610
1
253.610
5.314
2
2.657
407.356
24
16.973
70714.000
30
823.867
29
a R Squared = .506 (Adjusted R Squared = .403)
Source
Corrected Model
Intercept
method
X
method * X
Error
Total
Corrected Total
F
4.908
16.382
.040
14.942
.157
Sig.
.003
.000
.961
.001
.856
Partial Eta
Squared
.506
.406
.003
.384
.013
The null hypothesis that there is an interaction between the independent variable
(teaching method) and the covariate (general ability) is not rejected (p = .856). This tells
us that the regression lines are parallel (homogeneous) and that this assumption of the
analysis of covariance is met.
c. Complete the ANCOVA.
The source table below is obtained from the analysis of covariance.
Tests of Between-Subjects Effects
Dependent Variable: Verbal fluency
Type III Sum
of Squares
df
Mean Square
411.197(a)
3
137.066
278.752
1
278.752
257.930
1
257.930
159.802
2
79.901
412.670
26
15.872
70714.000
30
823.867
29
a R Squared = .499 (Adjusted R Squared = .441)
Source
Corrected Model
Intercept
X
method
Error
Total
Corrected Total
F
8.636
17.563
16.251
5.034
Sig.
.000
.000
.000
.014
Partial Eta
Squared
.499
.403
.385
.279
The null hypothesis that the means of verbal fluency test adjusted for the general
ability test scores are the same for each of the three teaching methods is rejected (p =
.014). Further, we see that 27.9% of the variance of these adjusted verbal fluency
scores is accounted for by the teaching method used. This is a respectable effect size.
Note that the null hypothesis that the mean of the covariate (general ability test scores)
is the same across the three methods of teaching is also rejected (p < .001). This is a
good thing. It wouldn’t make any sense to use general ability to adjust the dependent
variable (verbal fluency) if the groups did not differ on this variable.
d. Compute the adjusted means. The table below gives the adjusted means. SPSS
calls them “estimated marginal means.”
Teaching method
Dependent Variable: Verbal fluency
95% Confidence Interval
Teaching method
1
2
3
Mean
Std. Error Lower Bound Upper Bound
46.725(a)
1.285
44.083
49.367
51.542(a)
1.263
48.946
54.137
46.533(a)
1.305
43.852
49.215
a Covariates appearing in the model are evaluated at the following values: General ability = 20.70.
e. Conduct the Tukey post hoc test on these adjusted means.
SPSS will not conduct Tukey post hoc tests on adjusted means. However, we can get
these comparisons pairwise comparisons in the OPTIONS window of UNIVARIATE.
The results are presented below. Keep in mind that we must adjust our level of
significance since we are doing multiple comparisons on the same data. Using the
Bonferroni technique we find that α’ = α/k = .05/3 = .017. We will, therefore, conduct
these comparisons using the .017 level of significance. The results are shown below.
There is a significant difference between the adjusted mean scores on the verbal fluency
test of students taught with methods 1 and 2. Examining the adjusted means we see that
students using method 2 scored higher than those using method 1.
There is no significant difference between the adjusted mean verbal fluency scores of
students using methods 1 and 3.
There is a significant difference between the adjusted mean scores on the verbal fluency
test of students taught with methods 2 and 3. Based on the adjusted means we can say
that students using method 2 scored higher than those using method 3.
Green et al, p. 221
Sam is interested in the relationship between college professors’ academic
discipline and their actual ability to fix a car, holding constant mechanical aptitude. Five
professors were randomly selected from mechanical engineering, psychology, and
philosophy departments at a major university. Each professor completed a mechanical
aptitude scale. Scores on this measure have a mean of 100 and a standard deviation of
15. The professors were then rated on how well they performed four automotive
maintenance tasks: changing oil, changing the points and plugs, adjusting the carbeuretor,
and setting the timing on a 1985 Pontiac. Ratings were based on the degree of success in
completing a task and the amount of time needed to complete it. Lower scores reflect
more efficiency at completing the automotive maintenance tasks.
The data view of the data file is shown below.
1. Transform the scores on the four mechanical task ratings by z-scoring them and then
summing the z scores so that Sam has a single measure of professors’ mechanical
performance efficiency.
We can convert the four mechanical task ratings using the DESCRIPTIVES
procedure under the DESCRIPTIVE STATISTICS submenu in SPSS. First, click
on the ANALYZE menu on the top of the data view screen and then click on the
DESCRIPTIVE STATISTICS submenu as shown below.
Now click on the DESCRIPTIVES submenu to obtain the following dialog box.
Now, move the four mechanical task rating variable into the Variable(s) box on the
right hand side using the right arrow. To obtain the z scores click on the Save
standardized values as variables box on the bottom left of the dialog. You should
see the dialog box looking like the one on the next page.
Now, click on the OK button. You can ignore the output on the output screen.
What we really want is the z scores that can be seen on the data screen as shown
below.
Now we can sum the z scores of the four tasks using the COMPUTE facility under
the TRANSFORM menu in SPSS. Click on the TRANSFORM menu at the top of
the data view screen and then click on COMPUTE to get the dialog box shown on
the next page.
We will call the
variable with the
sum of the four task
z scores totalz and
so we will type it in
the Target variable
box. In the
Numeric
Expression box we
will type sum ( and
move each of the
four task z scores,
separating them
with commas. The
dialog box should
look like the one
shown below.
Now, click on the OK button and see the new variable on the data screen as shown
on the next page.
2. Evaluate whether the relationship between mechanical aptitude and mechanical
performance efficiency is the same of all three types of professors (homogeneity-ofslopes assumption). What should Sam conclude about the homogeneity-of slopes
assumption? Report the appropriate statistics from the output to justify your
conclusion.
We can test the assumption of homogeneity-of-slopes using its definition that tells us
that there should be no interaction between the covariate (the score on the mechanical
aptitude test) and the independent variable (the professors’ academic discipline). This
requires doing a special ANCOVA where we specify a model different from the total
factorial default that SPSS usually uses. To do this, we will click on the ANALYSIS
menu and then click on the GENERAL LINEAR MODEL and choose
UNIVARIATE, which will give the dialog box shown on the next page.
Now, move the variable
totalz into the Dependent
Variable box, the
variable group into the
Fix Factor(s) box, and
the covariate, machapt,
into the Covariate(s)
box.
Next, click the Model
button on the upper right
hand corner of the dialog
box and obtain the new
dialog box shown below.
Now, click on
the Custom
button on top
of the page.
The Factors &
Covariates box
will darken.
Highlight
group and
move it into the
Model box.
Next, do the
same for
mechapt
variable. Finally, use the CONTROL key on your keyboard to highlight both
variables at the same time in the Factors & Covariates box and move them into the
Model box. The completed dialog box should look like the one on the next page.
Now click on the Continue button to get the main dialog box and click on the OK
button. You should get the dialog box shown below.
Tests of Between-Subjects Effects
Dependent Variable: totalz
Type III Sum
of Squares
df
Mean Square
132.085(a)
5
26.417
34.860
1
34.860
2.249
2
1.125
34.689
1
34.689
2.136
2
1.068
14.287
9
1.587
146.373
15
146.373
14
a R Squared = .902 (Adjusted R Squared = .848)
Source
Corrected Model
Intercept
group
mechapt
group * mechapt
Error
Total
Corrected Total
F
16.641
21.960
.708
21.852
.673
Sig.
.000
.001
.518
.001
.534
Note that the null hypothesis that there is no interaction between the independent
variable and the covariate cannot be rejected (p = .534). Hence, Sam should conclude
that his data meet the homogeneity-of-slopes assumption and that it is appropriate for
him to continue this analysis.
3. Conduct the standard ANCOVA on these data.
The ANCOVA is conducted using the UNIVARIATE procedure under the GENERAL
LINEAR MODEL submenu of the ANALYZE menu on the top of the data display
screen. Move the variable totalz into the Dependent Variable box, the variable group
into the Fix Factor(s) box, and the covariate, machapt, into the Covariate(s) box. You
might check the Models dialog box to make certain that the Full factorial model button
is selected. Next click on the Options button in the main UNIVARIATE dialog box to
be obtain the dialog box shown on the next page.
Highlight the variable
group and in the Factors
and Factor Interactions
box and use the right
arrow to move it into the
Display Means for box.
This tells the system to
display the adjusted
means. Now check off
the Descriptive
statistics, Estimates of
effect size, and Observed
power boxes under
Display in order to
obtain these statistics.
The dialog box should
look like the one shown
below.
Finally, click on the
Continue button to get
back to the main dialog
box. Now click the OK
button in the main
dialog box to obtain the
results.
Tests of Between-Subjects Effects
Dependent Variable: totalz
Type III Sum
Source
of Squares
Corrected Model
129.949b
Intercept
35.929
mechapt
36.169
group
19.803
Error
16.423
Total
146.373
Corrected Total
146.373
a. Computed using alpha = .05
df
3
1
1
2
11
15
14
Mean Square
43.316
35.929
36.169
9.901
1.493
F
29.012
24.064
24.225
6.632
Sig.
.000
.000
.000
.013
Partial Eta
Squared
.888
.686
.688
.547
Noncent.
Parameter
87.036
24.064
24.225
13.263
Observed
Powera
1.000
.994
.994
.814
b. R Squared = .888 (Adjus ted R Squared = .857)
The null hypothesis that the mean score on the mechanical tasks adjusted for the score
on the mechanical aptitude test is the same for all three groups of professors is rejected
at the α = .05 level of significance (p = .013).
The effect size associated with the effect due to professor type is .547. This tells us
that 54.7% of the variance of the adjusted mechanical task scores is due to professor
type. This is a rather large effect size.
The F value associated with the covariate is 24.225. This tells us that the mean value
of this covariate is not the same in all three groups of professors. This gives evidence
to the notion that this is an appropriate covariate to use in this case.
The so called “Estimated Marginal Means” given by SPSS are the adjusted means.
They are shown for each group of professors in the table below.
Professor type
Dependent Variable: totalz
95% Confidence Interval
Professor type
Mechanical engineering
Psychology
Philosophy
Mean
Std. Error Lower Bound Upper Bound
-1.875(a)
.619
-3.236
-.513
.916(a)
.551
-.298
2.129
.959(a)
.656
-.485
2.403
a Covariates appearing in the model are evaluated at the following values: Mechanical aptitude test
score = 99.47.
Finally, we will use lmatrices to do post hoc tests keeping in mind that we must use a
Bonferroni procedure to determine the appropriate level of significance for these tests.
We will use α’ = .05/3 = .017 since there are three groups of professors. The syntax
shown on the next page will give us these tests.
Running this syntax gives us the output shown below.
Custom Hypothesis Tests #1
Contrast Results (K Matrix)
Contrast
L1
a
Contrast Es timate
Hypothesized Value
Difference (Estimate - Hypothesized)
Std. Error
Sig.
95% Confidence Interval
for Difference
Lower Bound
Upper Bound
a. Based on the user-s pecified contrast coefficients (L') matrix:
Engineers vs Psychologists
Depende
nt
Variable
totalz
-2.790
0
-2.790
.803
.005
-4.557
-1.023
Te st Results
Dependent Variable: totalz
Sum of
Source
Squares
df
Contrast
18.032
1
Error
16.423
11
Mean Square
18.032
1.493
F
12.077
Sig.
.005
There is a significant difference between the adjusted mean task scores of Mechanical
Engineer professors and Psychology professors (p = .005). We can see from the adjusted
means that Mechanical Engineering professors had a lower score (were more efficient)
than Psychology professors.
Custom Hypothesis Tests #2
Contrast Results (K Matrix)
Contrast
L1
a
Depende
nt
Variable
totalz
-2.833
0
Contrast Es timate
Hypothesized Value
Difference (Estimate - Hypothesized)
Std. Error
Sig.
95% Confidence Interval
for Difference
-2.833
1.012
.017
-5.060
-.607
Lower Bound
Upper Bound
a. Based on the user-s pecified contrast coefficients (L') matrix:
Engineers vs Philos ophers
Te st Results
Dependent Variable: totalz
Sum of
Source
Squares
df
Contrast
11.709
1
Error
16.423
11
Mean Square
11.709
1.493
F
7.842
Sig.
.017
The null hypothesis that there is no difference in the mean mechanical tasks scores of
Mechanical Engineering professors and Philosophy professors is not rejected (p = .017).
Custom Hypothesis Tests #3
Contrast Results (K Matrix)
Contrast
L1
a
Depende
nt
Variable
totalz
-.043
0
Contrast Es timate
Hypothesized Value
Difference (Estimate - Hypothesized)
Std. Error
Sig.
95% Confidence Interval
for Difference
-.043
.887
.962
-1.996
1.909
Lower Bound
Upper Bound
a. Based on the user-s pecified contrast coefficients (L') matrix:
Ps ychologist vs Philosophers
Te st Results
Dependent Variable: totalz
Sum of
Source
Squares
df
Contrast
.004
1
Error
16.423
11
Mean Square
.004
1.493
F
.002
Sig.
.962
The null hypothesis that there is no difference in the adjusted mean scores on the
mechanical tasks between Psychology and Philosophy professors is not rejected (p =
.962).