Topic
Pages
Simple linear regression (SLR) - heights example
1-9
Simple linear regression
10
Correlation
12-20
Probability Model for SLR
21
Multiple linear regression
33
Matrix formulation for MLR
36-40
Variable selection in MLR
41-52
Partial and seq. sums of squares
47-50
ANOVA as regression
69
General linear model
72
ANCOVA
82-88
Lack-of-fit
89-92
One-way ANOVA,contrasts
93-102
Multiple comparisons
103-109
Expected mean squares, part I
110-111
Sample size computation
112-116
Orthogonal polynomial contrasts
117-121
Multi-factorial expts
122
2 × 2 expts
122-133
More than 2 levels
134
More than 2 factors
141
Unbalanced example
149
Blocking
156-165
Latin squares
166-175
One-way random effects
176-189
Mixed models
190-235
Split-plot expts
236-255
Rough Table of Contents for notes that accompany the lectures
1
Year
Yield
Rainfall
Year
Yield
Rainfall
Year
Yield
Rainfall
Year
Yield
Rainfall
1890
24.5
9.6
1900
34
10.8
1910
34.9
9.3
1920
38.3
11.6
1891
33.7
12.9
1901
19.4
7.8
1911
30.1
7.7
1921
35.2
12.1
1892
27.9
9.9
1902
36
16.2
1912
36.9
11
1922
35.5
8
1893
27.5
8.7
1903
30.2
14.1
1913
26.8
6.9
1923
36.7
10.7
1894
21.7
6.8
1904
32.4
10.6
1914
30.5
9.5
1924
26.8
13.9
1895
31.9
12.5
1905
36.4
10
1915
33.3
16.5
1925
38
11.3
1896
36.8
13
1906
36.9
11.5
1916
29.7
9.3
1926
31.7
11.6
1897 1898
29.9 30.2
10.1 10.1
1907 1908
31.5 30.5
13.6 12.1
1917 1918
35
29.9
9.4
8.7
1927
32.6
10.4
y1, . . . , y38 and x1, . . . , x38.
1899
32
10.1
1909
32.3
12
1919
35.2
9.5
Yields y (in bushels/acre) on corn raised in six midwestern states
from 1890 to 1927 recorded with rainfall x (inches/yr).
An example: The association between corn yield and rainfall
ST 512: Exptl Stats for Biol. Sciences II
Week 1 Simple linear regression
Reading: Rao, Ch. 10
Osborne, ST512
2
10
•
••
•
12
• •
•
•
•
X: inches of rain
•
•
14
•
•
16
•
•
−1 ≤ rxy ≤ 1
• correlation coefficients always between -1 and 1:
• rxy is a measure of the linear assn. between x and y in a dataset.
Some properties of rxy
sxy is called the sample covariance of x and y:
P
(xi − x̄)(yi − ȳ)
sxy =
.
n−1
rxy
P
(xi − x̄)(yi − ȳ)/(n − 1)
sxy
p
= P
=
P
2
2
s
(xi − x̄) /(n − 1) ∗ (yi − ȳ) /(n − 1)
x sy
• How can we use this association to predict future yield, if we have
an idea about what the rainfall will be?
• How can we use this association to estimate average yield, given
a certain level of rainfall?
• To what degree is the variability in yield described or explained
by its association with rainfall?
• If |rxy | = 1, then x and y are said to be perfectly correlated.
• The bigger |rxy |, the stronger the linear association
• The closer rxy is to -1, the stonger the negative linear association
(y tends to be smaller than avg. when x bigger than avg.)
•
•
••
•
• How can we measure the strength of the linear association?
8
•
•
is defined by
(x1, y1), (x2, y2), . . . , (xn, yn)
• The closer rxy is to 1, the stonger the positive linear association
between x and y
•
•
•
•••
•
3
Correlation
Definition: The sample correlation coefficient rxy of the paired data
Osborne, ST512
• How can we describe the association between yield and rainfall?
Does it appear linear?
Some questions:
•
•
•
•
••
•• •
Corn yields from 1890-1927
A scatterplot provides some indication of an association between y
and x. In particular, yields seem to increase with rainfall.
Osborne, ST512
Y:Bushels per acre
35
30
25
20
s2x = 5.13 sx = 2.27
sxy
3.98
3.98
= 0.40
=√
=
sx sy
9.87
5.13 × 19.0
4
Here, E(·) denotes mathematical expectation. This concept underlies all of the statistical modelling to be discussed in this course.
The population correlation coefficient ρ
Just as x̄ provides empirical information about a population mean
µx, rxy can be used for inference about the population correlation
coefficient ρxy . This parameter refers to the correlation among x
and y in the population of interest:
(X − µX ) (Y − µY ))
= ρ.
ρXY = E
σX
σy
rxy =
ȳ = 31.9, s2y = 19.0 sy = 4.44
P
147.3
(xi − x̄)(yi − ȳ)
=
= 3.98
sxy =
n−1
38 − 1
Applying the formula for rxy , we get
x̄ = 10.8,
Summary statistics for corn yields data:
Osborne, ST512
5
can be rearranged to yield an approximate 100(1 − α)% confidence
interval for ψ:
zα/2
1+R
1
log
±√
.
2
1−R
n−3
An approximate 100(1−α)% confidence interval for ρ can be obtained
by inverting the Fisher transformation:
1+ρ
1
.
ψ = log
2
1−ρ
The probability statement
√
1+R
1+ρ
1
− log
< zα/2)
1−α = Pr(z1−α/2 < n − 3 log
2
1−R
1−ρ
A test statistic useful for inference about ρ is
1+R
1√
1+ρ
Z(ρ) = ( n − 3)(log
− log
).
2
1−R
1−ρ
Asymptotically, Z has the standard normal (N (0, 1)) distribution so
that it can be used to derive methods of inference for ρ (testing and
confidence intervals).
Under H0 : ρ = 0,
1√
1+R ·
( n − 3) log
∼ N (0, 1).
2
1−R
A large-sample test of H0 with level α then rejects H0 whenever
1√
1+R
n − 3 log
> zα/2
2
1−R
or
1√
1+R
< z1−α/2
n − 3 log
2
1−R
where zα satisfies α = Pr(Z > zα ) with Z ∼ N (0, 1).
Osborne, ST512
2ψ
proc corr data=butterfat FISHER;
var butterfat temp;
run;
2. Suppose that two variables X and Y have correlation ρ = 0.6.
What is the probability that a random sample of n = 30 observations from this bivariate population will yield a sample correlation
coefficient of 0.7 or higher?
Hint for # 1.:
1. Examine the butterfat and temperature data plotted on the next
page. Is there evidence of linear association? The sample correlation coefficient is r = −0.45 based on randomly sampled days.
Carry out an appropriate test. Obtain a 95% confidence interval for the population correlation coefficient describing the linear
association between % butterfat and temperature.
Exercise:
(There is a one-to-one correspondence between testing and interval
estimation here, so that H0 : ρ = 0 would be rejected at α = 0.05.)
For the corn yields data, r = 0.4 and n = 38, and a 95% interval is
given by
(0.09 < ρ < 0.64).
ρ=
e −1
e2ψ + 1
Evaluating ρ at the limits for ψ leads to the interval
!
√
√
1+r −2z/ n−3
1+r 2z/ n−3
e
−
1
e
−
1
1−r
√
√
.
, 1−r
1+r −2z/ n−3
1+r 2z/ n−3
e
+
1
+1
1−r
1−r e
Note that ρ and ψ are related by
6
10
20
AGE
40
50
60
45
temperature
55
Percentage of fat in milk
30
Resolution run (5k), 2004
70
65
80
Some example scatterplots (r1 = 0.04 and r2 = −0.45)
Osborne, ST512
16
14
Osborne, ST512
pace
pctfat
12
10
8
6
5.0
4.8
4.6
4.4
7
56 58 60 62 64
56 58 60 62 64 66
y
y
4. r = −0.6
3. r = 0.1
2. r = 0.7
1. r = 0.3
•
•
28
• ••
•
•
•
•
•
30
•
26
•
• ••
• ••
24
•
•
•
•
•
28
••
•
•
32
•
•
x
x
•
30
•
•
32
•
•
34
••
•
••
• •
•
36
•
•
34
•
•
•
•
38
•
36
•
y
y
8
•
26
26
•
•
•
•
•
28
•••
• ••• • ••
28
• •
• •
•
•
x
30
x
30
•
•
••
32
•
32
•
••
•
• •
••
•• •
•• •
•
•
•
•
•
•
34
34
•
•
An exercise/activity:
Label the four plots below with the four sample correlation coefficients:
Osborne, ST512
58 60 62 64
55 60 65 70 75 80
9
• Among 50 countries examined in a dietary study, high positive
correlation among fat intake and cancer (see figure, next page).
This example is taken from from Statistics by Freedman, Pisani
and Purves.
• Lurking variable - a third variable that is responsible for a correlation between two others. (A.k.a. confounding factor.) An
example would be to assess the association between say the reading skills of children and other measurements taken on them, such
as shoesize. There may be a statistically significant association
between shoe size and reading skills, but that doesn’t imply that
one causes the other. Rather, both are positively associated with
a third variable, age.
• In a city, if you were to observe the amount of damage and the
number of fire engines for enough recent fires, you would likely
see a positive and significant correlation among these variables.
Obviously, it would be erroneous to conclude that fire engines
cause damage.
• A study finds a high positive correlation between coffee drinking
and coronary heart disease. Newspaper reports say the fragrant
essence of the roasted beans of Coffea arabica are a menace to
public health.
Correlation does not imply causation
Famous examples of spurious correlations:
Osborne, ST512
(p. 152, Statistics by Friedman, Pisani, Purves and Adhikari)
In countries where people eat lots of fat like the United
States rates of breast cancer and colon cancer are high.
This correlation is often used to argue that fat in the diet
causes cancer. How good is the evidence?
Discussion. If fat in the diet causes cancer, then the
points in the diagram should slope up, other things being
equal. So the diagram is some evidence for the theory.
But the evidence is quite weak, because other things aren’t
equal. For example, the countries with lots of fat in the
diet also have lots of sugar. A plot of colon cancer rates
against sugar consumption would look just like figure 8,
and nobody thinks that sugar causes colon cancer. As it
turns out, fat and sugar are relatively expensive. In rich
countries, people can afford to eat fat and sugar rather
than starchier grain products. Some aspects of the diet in
these countries, or other factors in the life-style, probably
do cause certain kinds of cancer and protect against other
kinds. So far, epidemiologists can identify only a few of
these factors with any real confidence. Fat is not among
them.
Osborne, ST512
10
Osborne, ST512
11
β
Ei
|{z}
| 0 +{zβ1x}i +
random error
deterministic
component
component
12
µ(x) = β0xβ1 .
Nonlinear regression: other models for E(Y |X = x) such as
β0, β1 and σ 2 are modelled as fixed, unknown parameters which can
be estimated from the data using simple linear regression.
• error variance σ 2
• slope term β1, average change in E(Y ) per unit increase in x
• intercept term β0 = E(Y |X = 0) (where µ(x) crosses y-axis.)
• independent variable or predictor variable X (right side)
• response or dependent variable Y (left side of regression equation)
Definitions:
2. Var(Y |X = x) = σ
2
1. µ(x) = E(Y |X = x) = β0 + β1x
where E1, . . . , En are independent and identically distributed normal
random variables with mean 0 and variance σ 2.
iid
(Write Ei ∼ N (0, σ 2).)
Note that this implies
Yi =
A probabilistic model for Y conditional on X = x:
A linear model for regression
Observe n independent pairs (y1, x1), (y2, x2), . . . , (yn, xn)
Osborne, ST512
1
n X
1
n
n
2 X
X
yi − (β̂0 + β̂1 xi ) =
(yi − ŷi )2 =
e2i
ŷi = β̂0 + β̂1xi.
13
An unbiased estimate of σ 2 is given by
SS[E]
.
σ̂ 2 = M S[E] =
n−2
Elementary calculus can show that β̂0 and β̂1 which minimize the
sum of squared residuals SS[E] are given by
P
(xj − x̄)(yj − ȳ)
P
β̂1 =
(xj − x̄)2
Sxy
=
(OL notation, p. 542)
Sxx
sxy
= 2 (sample covariance ÷ sample variance )
sx
sy
= rxy
sx
β̂0 = ȳ − β̂1x̄
ei = yi − ŷi
• residual for the ith observation:
• Predicted value of response Yi given X = xi:
Definitions:
is minimized. These are “least squares” (LS) estimates.
SS[E] =
Choose β̂0 and β̂1 so that
1
Fitting a linear model
• Choose “best” values for β0, β1.
Osborne, ST512
=
β̂0 =
=
=
= 19.0, sxy = 3.98, rxy = 0.40
19.0
5.1
0.776( bushels per acre ÷ inches per year)
ȳ − β̂1x̄
31.9 − 0.776(10.8)
23.5 bushels per acre
= (0.40)
sxy
s2x
3.98
=
5.13
sy
= rxy
sx r
β̂1 =
=
5.1, s2y
Note that
P
1. n1 (yi − ŷi) = 0
P
2. n1 (yi − ŷi)2 is minimized
ŷ = 23.5 + (0.776)x.
yielding the least squares line of
so that
s2x
x̄ = 10.8 inches ȳ = 31.9 bushels per acre
For the corn yield data, recall that
is called the linear regression of y on x.
(It is also called the least squares line.)
y = β̂0 + β̂1x
Definition: The line satisfying the equation
Osborne, ST512
14
15
i=1
X
X
(yi − ŷi)2
e2i
= SS[T OT ] − SS[R]
=
SS[E] =
=
i=1
2
β̂1 Sxx
(xi − x̄)2
(β̂0 + β̂1xi − (β̂0 + β̂1x̄))2
(ŷi − ȳ)2
n
X
i=1
n
X
= β̂12
=
SS[R] =
n
X
i=1
The sums of squares are defined by
n
X
SS[T OT ] =
(yi − ȳ)2
Source Sum of squares df Mean Square
F-Ratio
Regression
SS[R]
1
M S[R]
M S[R]/M S[E]
Error
SS[E]
n−2
M S[E]
Total
SS[T OT ]
n−1
The ANOVA table from simple linear regression
Observed variability in the response Y is measured by the total sum
of squares SS[T OT ] and can be partitioned into independent components: the sum of squares due to regression, SS[R] and the sum
of squares due to error, SS[E].
Osborne, ST512
16
vs
H1 : β1 6= 0
SE(β̂1)
.
M S[R]
T =F =
M S[E]
2
• The ratio of SS[R] to SS[T OT ] is called the coefficient of determination, or sometimes, simply “r-square”. It represents the
proportion of variation observed in the response variable y which
can be “explained” by its linear association with x. In simple lin2
ear regression, “r-square” is in fact equal to rxy
. (But this isn’t
the case in multiple regression.) It is also equal to the squared
correlation between yi and ŷi. (This is the case in multiple regression.)
• The mean square for error, M S[E], is an unbiased estimator for
σ 2, the common variance of the response variable Y conditional
on an observed independent variable X. (E(M S[E]) = σ 2 =
Var(Y |X)). (Here, conditional means for those elements in the
population with independent variable X = x.) As such, it can be
used to construct confidence intervals for β0 and β1. It is based
on n − 2 degrees of freedom.
T =
β̂1
at level α. The critical value for F is the upper α percentile from
the F distribution with 1 numerator and n − 2 denominator
degrees of freedom. This F test is equivalent to a T test based
on the statistic we’re about to discuss:
H0 : β1 = 0
• The F ratio can be used to test for “significance of regression” or
to test the null hypothesis that the slope parameter, β1 is zero:
Osborne, ST512
Confidence intervals for β0, β1
17
100(1 − α)% confidence intervals for β1 and β0 are given by
s
M S[E]
.
β̂1 ± t(n − 2, α/2)
Sxx
s
x̄2
1
+
β̂0 ± t(n − 2, α/2) M S[E]
.
n Sxx
√
Take
and substitute M S[E] for σ 2 to get estimated standard
errors:
s
M S[E]
c β̂1) =
SE(
Sxx
s
1
x̄2
c β̂0) =
M S[E]
SE(
+P
(xi − x̄)2
n
σ2
Var(β̂1|x1, . . . , xn) = P
(xi − x̄)2
x̄2
1
2
+P
Var(β̂0|x1, . . . , xn) = σ
n
(xi − x̄)2
• for normal data, Ȳ ⊥ β̂1 which leads to
E(β̂1|x1, . . . , xn) = β1 and E(β̂0|x1, . . . , xn) = β0
• unbiasedness:
Important results for sampling distribution of (β̂0, β̂1) (given x1, . . . , xn)
Osborne, ST512
18
β̂1 − slope0
c β̂1)
SE(
Exercise: derive these variances.
This yields a confidence interval of the form
s
1 (x0 − x̄)2
+
β̂0 + β̂1x0 ± t(n − 2, α/2) M S[E]
n
Sxx
The conditional mean E(Y |X = x0) can be estimated by evaluating
the regression function µ(x0) at the estimates β̂0, β̂1. The conditional
variance of the expression isn’t too difficult:
1 (x0 − x̄)2
2
+
Var(β̂0 + β̂1x0|X = x0) = σ
n
Sxx
Confidence interval for E(Y |X = x0)
T =
Any hypothetical slope, like H0 : β1 = slope0 may be tested using
the T -statistic below with df = n − 2:
Osborne, ST512
19
•
•
•
•
8
•
•
•
•
••
•
•
•
•
•
•
10
•
•
•
•
•
•
•
•
•
•
•
12
•
•
corn.rain
•
•
•
•
Corn yields with LS line
•
14
•
•
16
•
•
For 95% CIs, use t(26, 0.025) = 2.028. For β1, note that
Sxx = (n − 1)s2x = 5.13(38 − 1) = 189.8 so that a c.i. is given by
r
16.4
0.776 ± (2.028)
189.8
or
0.776 ± (2.028)(0.294) or 0.776 ± 0.596
For β̂0,
s
1
(10.8)2
23.5 ± (2.028) 16.4
+
38
189.8
or
23.5 ± (2.028)(3.24) or 23.5 ± 6.57
The α = 0.05 critical value is F (1, 36, 0.05) = 4.11 Therefore, there
is a significant, positive linear association between yield and rainfall.
The yield on corn by rainfall example
Source Sum of squares df Mean Square F-Ratio
Regression
114
1
114
6.95
Error
591
36
16.4
Total
705
37
Osborne, ST512
corn.yield
35
30
25
20
Prediction
20
or
34.4 ± 8.5
or
(25.9, 42.9)
(14 − 10.8)2
1
23.5 + 0.776(14) ± 2.028 16.4 1 + +
38
189.8
s
Example: Suppose that one year rainfall is x0 = 14 inches, but that
yield Y0 from the six states hasn’t been measured. Obtain a 95%
prediction interval for Y0, using the model.
In order to form a 95% prediction interval take
s
1 (x0 − x̄)2
Yˆ0 ± t(n − 2, α/2) M S[E] 1 + +
.
n
Sxx
We’ve seen that the predicted value of Y based on the linear regression is given by Yˆ0 = β̂0 + β̂1x0.
• the randomness of a future value Y .
• the sampling distribution of β̂0 and β̂1
Often, prediction of the response variable Y for a given value, say x0,
of the independent variable is of interest. In order to make statements
about future values of Y , we need to take into account
Osborne, ST512
34.4 ± 2.32
or
(32.1, 36.7)
21
Exercise taken from Dickey’s notes:
An industrial quality control expert takes 200 hourly measurements
on an industrial furnace which is under control and finds that a 95%
confidence interval for the mean temperature is (500.35, 531.36). As a
result he tells management that the process should be declared out of
control whenever hourly measurements fall outside this interval and,
of course, is later fired for incompetence. (Why and what should he
have done?)
What is the difference?
or
A 95% confidence interval for E(Y |X = 14) is given by
s
(14 − 10.8)2
1
23.5 + 0.776(14) ± 2.028 16.4
+
38
189.8
The 95% prediction interval is (25.9, 42.9).
Osborne, ST512
22
It is not safe to extrapolate the results of a linear regression for the
purposes of prediction beyond the range of observed independent
variables.
In the space of 176 years the Lower Mississippi has shortened
itself 252 miles. That is an average of a trifle more than one
mile and a third per year. Therefore any calm person, who is
not blind or idiotic, can see that in 742 years from now, the
lower Mississippi will be one mile and three quarters long,
and Cairo, Ill., and New Orleans will have joined their streets
together.
A note of caution:
Mark Twain, in Life on the Mississippi:
Osborne, ST512
2
3
4
65
65
64
4.58 4.67 4.60
12
13
14
56
62
37
4.82 4.65 4.66
5
61
4.83
15
37
4.95
6
7
55
39
4.55 5.14
16
17
45
57
4.60 4.68
8
41
4.71
18
58
4.65
9
10
46
59
4.69 4.65
19
20
60
55
4.6 4.46
23
7
3
13
10
14
11
6 14
5 15
15
15
yj
xj
yj
xj
4 8 7
7 10 4
9
9
14
11
20
16 19.8 18.4 17.1 15.5 14.7 17.1
88.6 71.6 93.3 84.3 80.6 75.2 69.7 82
15 17.2 16
17 14.4
79.6 82.6 80.6 83.5 76.3
• xj Temperature (o F )
• Striped ground cricket.
• yj Chirps per second
3. Cricket Data
xj
yj
15.4 16.2
69.4 83.3
• The same scoring is used to quantify duck behavioral traits.
• A 0 corresponds to a purely mallard phenotype and a 15 corresponds to a purely
pintail phenotype.
• Mallard and Pintail ducks were crossed yielding n = 11 second generation males
with attributes as given in the table
• yj : Behavioral index
• xj : Plumage index
2. Hybrid duck data:
Date
1
xj
64
yj
4.65
Date 11
xj
56
yj
4.36
• xj : temperature on date j
• n = 20 successive days
• yj : average “percent butterfat” for 10 cows on date j
1. Butterfat by temperature in cows:
Osborne, ST512
64
65
66
67
68
parent
69
70
71
72
73
61.7 62.2 63.2 64.2 65.2 66.2 67.2 68.2 69.2 70.2 71.2 72.2 73.2 73.7
73.0 0
0
0
0
0
0
0
0
0
0
0
1
3
0
72.5 0
0
0
0
0
0
0
1
2
1
2
7
2
4
71.5 0
0
0
0
1
3
4
3
5
10
4
9
2
2
70.5 1
0
1
0
1
1
3
12
18
14
7
4
3
3
69.5 0
0
1
16
4
17
27
20
33
25
20
11
4
5
68.5 1
0
7
11
16
25
31
34
48
21
18
4
3
0
67.5 0
3
5
14
15
36
38
28
38
19
11
4
0
0
66.5 0
3
3
5
2
17
17
14
13
4
0
0
0
0
65.5 1
0
9
5
7
11
11
7
7
5
2
1
0
0
64.5 1
1
4
4
1
5
5
0
2
0
0
0
0
0
64.0 1
0
2
4
1
2
2
1
1
0
0
0
0
0
/* ------------------------------------------------------------------| Stigler , History of Statistics pg. 285 gives Galton’s famous data |
| on heights of sons (columns,Y) and average parents’ height (rows,X) |
| scaled to represent a male height (essentially sons’ heights versus |
| fathers’ heights).
Taken from Dickey’s website.
|
\ -------------------------------------------------------------------*/
61
62
63
64
65
66
67
68
69
70
71
72
73
son
74
A classical dataset:
The association between height of adults and their parents
Osborne, ST512
24
25
for i = 1, . . . , n(n = 928).
Question: Suppose we ignore midparent height x. Consider estimating the mean E(Y ). Propose a model. Propose a method for
obtaining a confidence interval for the mean height of the sons in
the population from which these data were randomly sampled.
Use summary statistics on page 6 to complete this naive analysis.
2. Var(Y |X = x) = σ 2 (Three unknown parameters β0, β1, σ 2
quantify the whole population of interest.)
1. µ(x) = E(Y |X = x) = β0 + β1x
This implies
iid
(Write Ei ∼ N (0, σ 2).)
• normally distributed random variables with mean 0 and error
variance σ 2.
• identically and
• independent,
where E1, . . . , En are
Yi = β0 + β1xi + Ei
Consider a statistical model for these data, randomly sampled from
some population of interest. In particular, choose a model which
accounts for the apparent linear dependence of the mean height of
sons on midparent height X. Let Y1, . . . , Yn denote the sons’ heights.
Given X = xi,
Osborne, ST512
12. Under the model, what is the true average height of sons with
midparent height x = 68?
11. Is this the true average height in the whole population of sons
whose midparent height is x = 68?
10. What is the estimated average height of sons whose midparent
height is x = 68?
9. How much of the observed variation in the heights of sons (the
y-axis) is explained by this “best” line?
8. What is the line that best fits these data, using the criterion that
smallest sum of squared residuals is “best?”
7. What is a region of plausible values for β1 suggested by the data?
6. How much does β̂1 vary about β1 from sample to sample? (Provide an estimate of the standard error, as well as an expression
indicating how it was computed.)
5. Is β̂1 = β1?
4. True/false: (a) β̂1 is a statistic (b) β̂1 is a parameter (c) β̂1 is
unknown.
3. What is the observed value of β̂1?
1. What is the meaning, in words, of β1?
22. Is E1, . . . , E928 ∼ N (0, σ 2) a reasonable assumption?
iid
21. What are plausible values for ρ suggested by the data?
20. Define µY , σY , µX , σX , ρ. Parameters or statistics?
19. Let Y denote the height of a male randomly sampled from this
population and X his midparent height. Is it true that the population correlation coefficient ρ satisfies
Y − µY
X − µX
ρ = E[
×
]?
σY
σX
18. What quantity can you use to describe or characterize the linear
association between height and midparent height in the whole
population? Is this a parameter or a statistic?
17. Is the observed linear association between son’s height and midparent height strong? Report a test statistic.
16. Is the estimated standard error of µ̂(72) bigger, smaller, or the
same as that for µ̂(68)?
15. What is the estimated standard error of the estimated average for
sons with midparent height x = 68, µ̂(68) = β̂0 + 68β̂1? Provide
an expression for this standard error.
14. What is the estimated standard deviation among the population
of sons whose parents have midparent height x = 72? Bigger,
smaller, or the same as that for x = 68? Is your answer obviously
supported or refuted by inspection of the scatterplot?
27
2. True/false: (a) β1 is a statistic (b) β1 is a parameter (c) β1 is
unknown.
Osborne, ST512
13. What is the estimated standard deviation among the population
of sons whose parents have midparent height x = 68? Would you
call this standard deviation a “standard error?”
26
Many questions to answer using regression analysis:
Osborne, ST512
proc print;
title "questions regarding prediction, estimation when x=68, x=72";
run;
data questions; set out1;
if son=.;
data big;
set big questions;
run;
proc reg;
model son=parent/clb;
output out=out1 residual=r p=yhat ucl=pihigh lcl=pilow uclm=cihigh lclm=cilow
stdp=stdmean;
data questions;
/* these values used for prediction or estimation at x=68,x=72 */
input parent son;
cards;
68 .
72 .
;
run;
proc means; var son parent;
data big; set galton; drop j count;
do j=1 to count;output; end;
proc print data=big(obs=20);
data Galton;
array cdata(14);
if _n_ = 1 then input cdata1-cdata14 @ ;
retain cdata1-cdata14; drop cdata1-cdata14 i;
input parent @;
do i = 1 to 14; input count @ ; son=cdata(i);
output; end;
cards;
61.7 62.2 63.2 64.2 65.2 66.2 67.2 68.2 69.2 70.2 71.2 72.2 73.2 73.7
73.0 0
0
0
0
0
0
0
0
0
0
0
1
3
0
72.5 0
0
0
0
0
0
0
1
2
1
2
7
2
4
71.5 0
0
0
0
1
3
4
3
5
10
4
9
2
2
70.5 1
0
1
0
1
1
3
12
18
14
7
4
3
3
69.5 0
0
1
16
4
17
27
20
33
25
20
11
4
5
68.5 1
0
7
11
16
25
31
34
48
21
18
4
3
0
67.5 0
3
5
14
15
36
38
28
38
19
11
4
0
0
66.5 0
3
3
5
2
17
17
14
13
4
0
0
0
0
65.5 1
0
9
5
7
11
11
7
7
5
2
1
0
0
64.5 1
1
4
4
1
5
5
0
2
0
0
0
0
0
64.0 1
0
2
4
1
2
2
1
1
0
0
0
0
0
;
proc print data=Galton(obs=100);
run;
options ls=75 nodate;
Osborne, ST512
28
29
1
Variable
N
Mean
Std Dev
Minimum
Maximum
-------------------------------------------------------------------------son
928
68.0884698
2.5179414
61.7000000
73.7000000
parent
928
68.3081897
1.7873334
64.0000000
73.0000000
--------------------------------------------------------------------------
The MEANS Procedure
/*
proc univariate data=out1 normal plot;
title "residual analysis";
var r;
run;
*/
The SAS System
proc gplot;
plot son*parent;
run;
quit;
symbol1 i=rl
value=dot;
*goptions dev=pslepsf colors=(black);
proc print;run;
data fisherz;
n=928;
r=sqrt(0.2105);
rratio=(1+r)/(1-r);
z=probit(0.975);
expon=probit(0.975)/sqrt(n-3);
rlow=(rratio*exp(-2*expon)-1)/(rratio*exp(-2*expon)+1);
rhigh=(rratio*exp(2*expon)-1)/(rratio*exp(2*expon)+1);
run;
Osborne, ST512
2.81088
0.04114
Standard
Error
18.42510
0.56556
29.45796
0.72702
95% Confidence Limits
8.52
15.71
<.0001
<.0001
n
928
1
68
72
1
2
Obs
parent
Obs
r
yhat
2.69551
z
67.7429
70.1434
cilow
1.95996
0.07457
0.16871
stdmean
rratio
67.8893
70.4745
0.45880
.
.
son
0.064443
expon
68.0356
70.8056
cihigh
0.40645
rlow
63.4936
66.0688
pilow
0.50815
rhigh
72.2849
74.8801
pihigh
30
.
.
r
3
<.0001
Pr > F
Pr > |t|
0.2105
0.2096
t Value
R-Square
Adj R-Sq
246.84
F Value
questions regarding prediction, estimation when x=68, x=72
1
1
Intercept
parent
Mean
Square
1236.93401
5.01109
Parameter Estimates
23.94153
0.64629
DF
1
1
Intercept
parent
Parameter
Estimate
Variable
DF
Variable
2.23855
68.08847
3.28770
1236.93401
4640.27261
5877.20663
Parameter Estimates
1
926
927
DF
Sum of
Squares
Analysis of Variance
The REG Procedure
Dependent Variable: son
Root MSE
Dependent Mean
Coeff Var
Model
Error
Corrected Total
Source
Osborne, ST512
31
12. µ(68) = β0 + 68β1.
11. Not sure, as µ(68) = β0 + 68β1 is unknown.
10. µ̂(68) = β̂0 + 68β̂1 = 67.9 (from output also.)
9. r2 = 21%
8. y = 23.9 + 0.65x
7. Add and subtract about 2 SE to get (0.57, 0.73)
5. β1 is the slope of the population mean, β̂1 is the slope from the
SLR of the observed data. β̂1 = β1 is unlikely.
p
c β̂1) = M S[E]/Sxx = 0.04 (from output.)
6. SE(
4. β̂1 = 0.65 is an observed value of a statistic.
3. β̂1 = 0.65 son inches/midparent inch (from output.)
2. β1 is an unknown parameter.
1. Change in average son’s height (inches) per one inch increase in
midparent height (in the whole population.)
Answers to questions from simple linear regression
analysis of Galton’s height data
Osborne, ST512
32
22. Residuals reasonably symmetric, no heavy tails.
or 0.41 < ρ < 0.51.
21. Using the complicated expression in Rao and in notes, the confidence interval is
!
"
√
√
1+r −2z/ n−3
1+r 2z/ n−3
e
−
1
e
−
1
1−r
√
√
, 1−r
1+r −2z/ n−3
1+r 2z/ n−3
e
+
1
+1
1−r
1−r e
20. These parameters describe the bivariate population of son and
midparent heights.
19. True.
18. ρ is a population correlation coefficient.
X a (928 × 2) design matrix.
c
c
16. SE(µ̂(72))
> SE(µ̂(68))
√
17. r = 0.21 = 0.46, moderate, positive.
15. SE(β̂0 + 68β̂1) = 0.07. Expressions given by
s
(68 − x̄)2
1
c
P
M S[E]
+
SE(µ̂(68)) =
n
(xi − x̄)2
p
(1, 68)′M S[E](X ′X)−1(1, 68)
=
13.
p
M S[E] = 2.24. Not a SE.
p
14. M S[E] = 2.24. (Assume homoscedasticity.)
Osborne, ST512
33
or
or
Y1
Y2
...
Y8
=
=
=
=
β0 + β1X11 + β2X12 + E1
β0 + β1X21 + β2X22 + E2
...
β0 + β1X81 + β2X82 + E8
Yi = β0 + β1Xi1 + β2Xi2 + Ei
Y = β0 + β1IQ + β2TIME + error
Consider a regression model for the GRADE of subject i, Yi, in which
the mean of Yi is a linear function of two predictor variables Xi1 = IQ
and Xi2 = Study TIME for subjects i = 1, . . . , 8:
IQ Study TIME GRADE
105
10
75
110
12
79
120
6
68
116
13
85
122
16
91
130
8
79
114
20
98
102
15
76
Topic: Multiple linear regression (MLR)
Reading Rao, Ch. 11
Multiple linear regression - an example
A random sample of students taking the same exam:
Osborne, ST512
Symbol used is ’*’.
IQ
STUDY |
|
|
|
|
20 +
*(98)
|
19 +
|
18 +
|
17 +
|
16 +
*(91)
|
15 +
*(76)
|
14 +
|
13 +
*(85)
|
12 +
*(79)
|
11 +
|
10 +
*(75)
|
9 +
|
8 +
*(79)
|
7 +
|
6 +
*(68)
|
--+---------+---------+---------+---------+---------+---------+100
105
110
115
120
125
130
Plot of STUDY*IQ.
GRADE AND STUDY TIME EXAMPLE FROM ST 512 NOTES
Osborne, ST512
34
35
σ̂ 2 =
SS[E]
n−p−1
What is the uncertainty associated with these parameter estimates?
β̂0 = 0.74, β̂1 = 0.47, β̂2 = 2.1
For this example, with p = 2 and n = 8,
– β0 : average response when x1 = x2 = . . . = xp = 0
– βi is called a partial slope for xi. Represents mean change in y
per unit increase in xi with all other independent variables
held fixed.
• β0, β1, . . . , βp are p + 1 unknown regression parameters:
• σ 2 is unknown error variance parameter.
Interpretations of regression parameters:
i=1
Least squares estimates of regression parameters minimize SS[E]:
n
X
(yi − β0 − β1xi1 − · · · − βpxip)2
SS[E] =
• As in SLR, E1, . . . , En ∼ N (0, σ 2),
iid
Yi = β0 + β1xi1 + β2xi2 + · · · + βpxip + Ei.
• For i = 1, . . . , n, MLR model for Yi :
• response variable for ith subject denoted by Yi
• Observed values of p independent variables for ith subject from
sample denoted by xi1, xi2, . . . , xip
A multiple linear regression (MLR) model w/ p independent variables Let p independent variables be denoted by x1, . . . , xp.
Osborne, ST512
Matrix formulation of MLR 36
=
=
=
=
β0 + β1x11 + β2x12 + · · · + βpx1p + E1
β0 + β1x21 + β2x22 + · · · + βpx2p + E2
...
β0 + β1xn1 + β2xn2 + · · · + βpxnp + En
For individual i,
Yi = xi·β + Ei
Here, the error vector E is assumed to follow a multivariate normal
distribution with variance-covariance matrix σ 2In
• E denotes an error vector (n × 1)
• β denotes a vector of regression parameters ((p + 1) × 1)
• X denotes a design matrix (n × (p + 1))
• Y denotes a response vector (n × 1)
where
Y = Xβ + E
This system of n equations can be expressed using matrices:
Y1
Y2
...
Yn
The MLR model for Y1, . . . , Yn is given by
xi· = (1, xi1, xi2, xi3, . . . , xip).
Let a (1 × (p + 1)) vector for p observed independent variables for
individual i be defined by
Osborne, ST512
37
X β̂
X(X ′X)−1X ′y
Hy
Y − Ŷ
Y − X β̂
(I − H)Y
• e is the vector of residuals
• H = X(X ′X)−1X is called the hat matrix
• Ŷ is called the vector of fitted or predicted values
Ŷ =
=
=
e =
=
=
Some more simplified expressions:
• X is assumed to be of full rank
the regression parameter vector β̂
• (X ′X)−1 may be verbalized as “x prime x inverse”
b is the estimated variance-covariance matrix for the estimate of
•Σ
(What are the dimensions of each of these quantities?)
β̂ = (X ′X)−1X ′Y
Var(β̂) = σ 2(X ′X)−1
= Σ
c β̂) = M S[E](X ′X)−1
Var(
b
= Σ
c ′β̂) = a′Σa
b
Var(a
Some simplified expressions: (a is a known p × 1 vector)
Osborne, ST512

8
919 100
X ′X =  919 106165 11400 
100 11400 1394


28.90 −0.23 −0.22
(X ′X)−1 =  −0.23 0.0018 0.0011 
−0.22 0.0011 0.0076


0.74
(X ′X)−1X ′Y =  0.47  =?
2.10

38
SS[E] = e′e = (Y − Ŷ )′(Y − Ŷ ) = 45.8, e′e/df = 9.15 =?


264.45 −2.07 −2.05
b = M S[E](X ′X)−1 =  −2.07 0.017 0.010 
Σ
−2.05 0.010 0.070
and
For the IQ, Study TIME example, with p = 2 independent variables
and n = 8 observations, consider
X, Y, (X ′X)−1, (X ′X)−1X; Y, X(X ′X)−1X ′Y :


1 105 10
 1 110 12 


 1 120 6 




 1 116 13 
X=

 1 122 16 


 1 130 8 


 1 114 20 
1 102 15
Osborne, ST512
Some questions - use preceding page
39
6. θ̂ ± t(0.025, 5)SE(θ̂) or 83.6 ± 2.57(1.14) or (80.7, 86.6)
4. Unknown population mean: θ = β0 + β1(113) + β1(14)
Estimate : θ̂ = (1, 113, 14) ∗ β̂ = 83.6
c β̂)(1, 113, 14)′
5. Var((1, 113, 14) ∗ β̂) = (1, 113, 14)Var(
√
′
b 113, 14) = 1.3 or SE(θ̂) = 1.3 = 1.14
or (1, 113, 14)Σ(1,
3. H0 : β1 = 0, T-statistic: t =
√ (β̂1 − 0)/SE(β̂1)
Observed value is t = .47/ .017 = .47/.13 = 3.6 > 2.57,
(“β̂1 differs significantly from 0.”)
1. β̂1 = 0.47 (second element of (X ′X)−1X ′Y , exam points per IQ
point for students studying the same amount)
√
b
2. 0.017 = 0.13 (square root of middle element of Σ)
Some answers
6. Report a 95% confidence interval
5. Report a standard error
4. Estimate the mean grade among the population of ALL students
with IQ = 113 who study T IM E = 14 hours.
3. Is β1 = 0 plausible, while controlling for possible linear associations between Test Score and Study time? (t(0.025, 5) = 2.57)
2. What is the standard error of β̂1?
1. What is the estimate for β1? Interpretation?
Osborne, ST512
Dependent
Variable
75.0000
76.0000
.
8
9
DF
1
1
1
Obs
1
Variable
Intercept
IQ
STUDY
Variable
Intercept
IQ
STUDY
STUDY
100
11400
1394
8399
Std Error
Mean Predict
1.9325
(abbreviated)
80.5426
1.9287
83.6431
1.1414
Predicted
Value
71.4447
75.5847
80.7092
GRADE
651
74881
8399
53617
Pr > F
0.0014
85.5005
86.5771
-4.5426
.
Residual
3.5553
STUDY
-2.051710248
0.010265884
0.0697933458
Pr > |t|
0.9656
0.0149
0.0005
F Value
32.57
GRADE
0.7365546771
0.473083715
2.1034362851
45.759884688
95% CL Mean
66.4770
76.4124
IQ
-2.069103589
0.016894712
0.010265884
Output Statistics
Intercept
264.47864999
-2.069103589
-2.051710248
Covariance of Estimates
t Value
0.05
3.64
7.96
Mean
Square
298.05756
9.15198
Standard
Error
16.26280
0.12998
0.26418
Sum of
Squares
596.11512
45.75988
641.87500
Parameter
Estimate
0.73655
0.47308
2.10344
DF
2
5
7
STUDY
-0.224182192
0.0011217122
0.0076260404
2.1034362851
Analysis of Variance
IQ
-0.226082693
0.0018460178
0.0011217122
0.473083715
X’X Inverse, Parameter Estimates, and SSE
IQ
919
106165
11400
74881
Model Crossproducts X’X X’Y Y’Y
Intercept
8
919
100
651
Intercept
28.898526711
-0.226082693
-0.224182192
0.7365546771
Source
Model
Error
Corrected Total
Variable
Intercept
IQ
STUDY
GRADE
Variable
Intercept
IQ
STUDY
GRADE
The SAS System
The REG Procedure
DATA GRADES; INPUT IQ STUDY GRADE @@; CARDS;
105 10 75 110 12 79 120 6 68 116 13 85 122 16 91 130 8 79 114 20 98 102 15 76
DATA EXTRA; INPUT IQ STUDY GRADE; CARDS;
113 14 .
DATA BOTH; SET GRADES EXTRA;
PROC REG; MODEL GRADE = IQ STUDY/P CLM XPX INV COVB;
Osborne, ST512
40
41
p - number of regression parameters in full model
q - number of regression parameters in reduced model
p − q - number of regression parameters being tested.
Recall:
X
SS[R] =
(Ybi − Ȳ )2
X
SS[E] =
(Ybi − Yi)2
X
SS[T ot] =
(Yi − Ȳ )2
• Model 1 nested in Model 5
• Model 4 nested in Model 1
• Model 5 nested in Model 4
A nested in B −→ A called reduced, B called full.
• Model 1 nested in Model 4
• Model 2 nested in Model 4
• Model 3 nested in Model 4
A is nested in B means model A can be obtained by restricting (e.g.
setting to 0) parameter values in model B.
True or false:
7. µ(x1, x2, x3) = E(Y |x1, x2, x3) = β0 + β2x2 + β3x3
6. µ(x1, x2, x3) = E(Y |x1, x2, x3) = β0 + β1x1 + β2x2
5. µ(x1, x2, x3) = E(Y |x1, x2, x3) = β0 + β1x1 + β3x3
4. µ(x1, x2, x3) = E(Y |x1, x2, x3) = β0 + β1x1 + β2x2 + β3x3
3. µ(x1, x2, x3) = E(Y |x1, x2, x3) = β0 + β3x3
2. µ(x1, x2, x3) = E(Y |x1, x2, x3) = β0 + β2x2
1. µ(x1, x2, x3) = E(Y |x1, x2, x3) = β0 + β1x1
Model Selection
x1, x2, x3 denote p independent variables. Consider several models:
Osborne, ST512
42
• Model 5 in model 4: ?
• Model 1 in model 5: R(β3|β1)
• Model 3 in model 4 ?
• Model 2 in model 4 ?
• Model 1 in model 4: R(β2, β3|β1)
Extra SS terms for comparing some of the nested models on preceding page:
Theory gives that if H0 holds (model A is appropriate) F behaves
according to the F distribution with p − q numerator and n − p − 1
denominator degrees of freedom.
(ok to supress β0 in these extra SS terms.)
R(βq+1, βq+2, . . . , βp|β0, β1, β2, . . . , βq ) = SS[R]f − SS[R]r .
Difference in the numerator called an extra regression sum of squares:
Let F =
(SS[E]r − SS[E]f )/(p − q) M S[H0]
=
M S[E]f
M S[E]
(r and f abbreviate reduced and full, respectively.)
H0 : βq+1 = βq+2 = · · · = βp = 0 (model A ok)
H1 : βq+1, βq+2, · · · , βp not all 0 (need model B)
Model Selection - concepts
In comparing two models, suppose
β1, . . . , βq in reduced model (A)
β1, . . . , βq , βq+1, . . . , βp in full model (B).
Comparison of models A and B amounts to testing
Osborne, ST512
An example: How to measure body fat?
43
x2
43.1
49.8
51.9
54.3
42.2
53.9
58.5
52.1
49.9
53.5
56.6
56.7
46.5
44.2
42.7
54.4
55.3
58.6
48.2
51.0
x3
29.1
28.2
37.0
31.1
30.9
23.7
27.6
30.6
23.2
24.8
30.0
28.3
23.0
28.6
21.3
30.1
25.7
24.6
27.1
27.5
y
11.9
22.8
18.7
20.1
12.9
21.7
27.1
25.4
21.3
19.3
25.4
27.2
11.7
17.8
12.8
23.9
22.6
25.4
14.8
21.1
Symbol
y
x1
x2
x3
Summary statistics:
x1
19.5
24.7
30.7
29.8
19.1
25.6
31.4
27.9
22.1
25.5
31.1
30.4
18.7
19.7
14.6
29.5
27.7
30.2
22.7
25.2
Variable
Body fat
Triceps
Thigh Circ.
Midarm Circ.
mean st. dev.
20.2
5.1
25.3
5.0
51.2
5.2
27.6
3.6
For each of n = 20 healthy individuals, the following measurements
were made: bodyfat percentage yi, triceps skinfold thickness, x1,
thigh circumference x2, midarm circumference x3
Osborne, ST512
15
20
y
25
15
20
x1
25
30
45
50
x2
0.84327
<.0001
0.87809
<.0001
0.14244
0.5491
x1
x2
x3
0.45778
0.0424
0.92384
<.0001
1.00000
0.84327
<.0001
x1
0.08467
0.7227
1.00000
0.92384
<.0001
0.87809
<.0001
x2
x3
1.00000
0.08467
0.7227
0.45778
0.0424
0.14244
0.5491
55
25
x3
30
35
44
Multicollinearity: linear associations among the independent variables; causes problems such as inflated sampling variances for β̂.
Marginal associations between y and x1 and between y and x2 are
highly significant, providing evidence of a strong r ≈ 0.85 linear
association between average bodyfat and triceps skinfold and between
average bodyfat and thigh circumference.
1.00000
y
Pearson Correlation Coefficients, N = 20
Prob > |r| under H0: Rho=0
y
Osborne, ST512
30
25
20
15
35
30
25
25
20
15
55
50
45
DF
1
1
DF
1
1
Variable
Intercept
x3
DF
1
1
DF
1
1
1
1
Variable
Intercept
x2
Variable
Intercept
x1
Variable
Intercept
x1
x2
x3
Standard
Error
5.65741
0.11002
Standard
Error
9.09593
0.32663
Parameter
Estimate
14.68678
0.19943
Standard
Error
3.31923
0.12878
Standard
Error
99.78240
3.01551
2.58202
1.59550
Parameter
Estimate
-23.63449
0.85655
Parameter
Estimate
-1.49610
0.85719
Parameter
Estimate
117.08469
4.33409
-2.85685
-2.18606
Parameter Estimates
The SAS System
The REG Procedure
Yields the following (abbreviated) output
proc reg data=bodyfat;
model y=x1 x2 x3;
model y=x1;
model y=x2;
model y=x3;
*model y=x1 x2 x3/xpx i covb corrb;
data bodyfat;
input x1 x2 x3 y;
cards;
19.5 43.1 29.1 11.9
24.7 49.8 28.2 22.8
(data abbreviated)
22.7 48.2 27.1 14.8
25.2 51.0 27.5 21.1
;
Osborne, ST512
t Value
1.61
0.61
t Value
-4.18
7.79
t Value
-0.45
6.66
t Value
1.17
1.44
-1.11
-1.37
Pr > |t|
0.1238
0.5491
Pr > |t|
0.0006
<.0001
Pr > |t|
0.6576
<.0001
Pr > |t|
0.2578
0.1699
0.2849
0.1896
1
45
Model Selection - examples
46
Adding x2, x3 leads to a model that is too complex for it’s own good.
proc reg data=bodyfat;
model y=x1 x2 x3;
test x2=0,x3=0;
run;
To get this F -ratio in SAS, try
After accounting for variation in bodyfat explained by linear dependence on triceps, there is still some linear association between mean
bodyfat and at least one of x2, x3 (thigh,midarm).
Q: Conclusion from this comparison of nested models?
(396.9 − 352.3)/2 22.3
=
= 3.64
F =
6.15
6.15
How many df ? The 95th percentile is F (0.05, , ) = 3.63.
after accounting for x1.
H0 : β2 = β3 = 0 vs H1 : β2, β3 not both 0
or the null hypothesis
Model A : µ(x1, x2, x3) = β0 + β1x1
Model B : µ(x1, x2, x3) = β0 + β1x1 + β2x2 + β3x3
In the bodyfat data, consider comparing the simple model that Y
depends only on x1 (triceps) and not on x2 (thigh) or x3 (midarm)
after accounting for x1 versus the full model that it depends on all
three.
Osborne, ST512
47
Intercept
x1
x2
x3
Parameter
117.0846948
4.3340920
-2.8568479
-2.1860603
Estimate
1
1
1
DF
99.78240295
3.01551136
2.58201527
1.59549900
1.17
1.44
-1.11
-1.37
t Value
12.70489278
7.52927788
11.54590217
Mean Square
352.2697968
33.1689128
11.5459022
Mean Square
Standard
Error
12.70489278
7.52927788
11.54590217
Type III SS
352.2697968
33.1689128
11.5459022
Type I SS
0.1699
0.2849
0.1896
Pr > F
<.0001
0.0337
0.1896
Pr > F
<.0001
Pr > F
0.2578
0.1699
0.2849
0.1896
Pr > |t|
2.07
1.22
1.88
F Value
57.28
5.39
1.88
F Value
21.52
F Value
Note agreement between p−values from Type III F tests and p−values
from t tests from parameter estimates from MLR.
x1
x2
x3
Source
1
1
1
Source
x1
x2
x3
DF
Model
Error
Corrected Total
132.3282039
6.1503055
3
16
19
Source
396.9846118
98.4048882
495.3895000
The GLM Procedure
Sum of
DF
Squares
Mean Square
PROC GLM can replace PROC REG to get the SUMS OF SQUARES
for use in model selection as in the following output:
Osborne, ST512
=
=
=
=
=
352.3
33.2
11.5
12.7
7.5
48
Exercise: specify the comparison of nested models that corresponds
to each of these F -ratios.
F =
7.5/1
11.5/1
12.7/1
= 2.07, F =
= 1.22, F =
= 1.88.
6.15
6.15
6.15
(Partial) effects significant ? (Use F (0.95, 1, 16) = 4.49.)
Type III F -ratios from bodyfat data for x1, x2, x3, respectively:
association between y and xj
Type III test for βj - test of partial
after
accounting
for
all
other
x
i
R(β1|β0)
R(β2|β0, β1)
R(β3|β0, β1, β2)
R(β1|β0, β2, β3)
R(β2|β0, β1, β3)
In the output on the preceding page,
Type I sums of squares - sequential (order of selection matters)
Type III sums of squares - partial
Type II sums of squares - partial (change in SSE due to adding
term A to model with all other terms
not ‘containing’ A)
Osborne, ST512
49
F (0.05, 1, 17) = 4.45 : nested model 1 is rejected in favor of model 6:
there is evidence (p = 0.037) of association between y and x2 after
accounting for dependence on x1.
SS[E]f = (SS[T ot]−SS[R]f ) and SS[R]f = SS[R]r +R(β2|β0, β1)
Note that
For 1. use R(β2|β0, β1) in the F ratio:
R(β2|β0, β1)
F =
M S[E]6
33.2
=
(SS[T ot] − R(β1, β2|β0)/(20 − 2 − 1)
33.2
=
(495.4 − 352.3 − 33.2)/(20 − 2 − 1)
33.2
=
109.9/17
= 5.1
2. Compare models 2 and 6
1. Compare models 1 and 6
Some model comparison examples
In all three of these tests, M S[E] computed from full model (#4).
3. Type I test for x3 from PROC GLM same as type III test for x3.
2. Type I SS for x2 from PROC GLM appropriate for
test of association between y and x2 after accounting for x1.
1. Type I SS for x1 from PROC GLM appropriate for SLR of y on
x1 .
In GLM output, which models are the type I tests comparing?
Osborne, ST512
50
DF
1
1
1
=
=
=
Pr > |t|
0.0348
0.4737
0.0369
Mean
Square
192.71935
6.46769
F Value
29.80
Type I SS
8156.76050
352.26980
33.16891
R(β1|β0, β2)/(∆df )
M S[E]f
(SS[R]f − SS[R]r )/1
6.5
352.3 + 33.2 − 382.0
6.5
3.4
≈ 0.5
6.5
Standard
Error t Value
8.36064
-2.29
0.30344
0.73
0.29119
2.26
F =
Parameter
Estimate
-19.17425
0.22235
0.65942
DF
2
17
19
Sum of
Squares
385.43871
109.95079
495.38950
The REG Procedure
Note that all of these comparisons of nested models are easy to carry
out using the TEST statement in PROC REG.
• Take model with x2. (Has higher r2 too.)
Pr > F
<.0001
Type II SS
34.01785
3.47289
33.16891
• x1 gives you nothing when you add it to model with x2
• x2 gives you a little when you add it to model with x1
Conclusions?
Variable
Intercept
x1
x2
Source
Model
Error
Corrected Total
proc reg;
model y=x1 x2/ss1 ss2;
run;
To compare models 2 and 6, we need SS[R]r = R(β2|β0) = 382.0
which cannot be gleaned from preceding output. You could also get
2
it from ryx
× SS[T ot] or from running something like
2
Osborne, ST512
51
DF
1
1
1
t Value
0.05
3.64
7.96
Mean
Square
298.05756
9.15198
Standard
Error
16.26280
0.12998
0.26418
Sum of
Squares
596.11512
45.75988
641.87500
Parameter
Estimate
0.73655
0.47308
2.10344
DF
2
5
7
Pr > F
0.0014
Pr > |t|
0.9656
0.0149
0.0005
F Value
32.57
Now the test for dependence on IQ is significant p = 0.0149. Why?
Variable
Intercept
IQ
study
Source
Model
Error
Corrected Total
Analysis of Variance
The REG Procedure
It appears that IQ has nothing to do with grade, but we did not look
at study time. Looking at the multiple regression we get
ANOVA (Grade on IQ)
SOURCE DF
SS
MS
F p-value
IQ
1 15.9393 15.9393 0.153 0.71
Error
6 625.935 104.32
3. Regress GRADE on TIME IQ TI where TI = TIME*IQ.
2. Regress GRADE on IQ and TIME.
1. Regress GRADE on IQ.
Another example, revisiting test scores and study times
Consider this sequence of analyses:
Osborne, ST512
1
1
1
1
Intercept
IQ
study
IQ_study
72.20608
-0.13117
-4.11107
0.05307
Parameter
Estimate
3
4
7
DF
54.07278
0.45530
4.52430
0.03858
Standard
Error
1.34
-0.29
-0.91
1.38
t Value
203.60344
7.76617
Mean
Square
0.2527
0.7876
0.4149
0.2410
Pr > |t|
Parameter Estimates
610.81033
31.06467
641.87500
Sum of
Squares
52975
15.93930
580.17582
14.69521
Type I SS
26.22
F Value
13.84832
0.64459
6.41230
14.69521
Type II SS
0.0043
Pr > F
1
52
Thus we expect an extra hour of study to increase the grade by 1.20
points for someone with IQ = 100 and by 2.26 points for someone
with IQ = 120 if we use this interaction model. Since the interaction
is not significant, we may want to go back to the simpler “main
effects” model. (This example taken from Dickey’s ST512 notes.)
Ĝ = (72.21 − 15.7) + (−4.11 + 6.37)S.
and if IQ 120 we get
Ĝ = (72.21 − 13.1) + (−4.11 + 5.31)S
Now if IQ = 100 we get
Ĝ = 72.21 − 0.13 ∗ I − 4.11 ∗ S + 0.0531(I ∗ S)
Discussion of the interaction model. We call the product I*S =
IQ*STUDY an ”interaction” term. Our model is
DF
Variable
Model
Error
Corrected Total
Source
The SAS System
The REG Procedure
Analysis of Variance
The interaction model
Osborne, ST512
53
Q: Why is Var(β̂0) bigger in M2 than in M1?
Q: M S[E]M6 < M S[E]M1 and M S[E]M6 < M S[E]M2 but the partial slopes have larger standard errors in M6. Why?
Design matrices




1 19.5
1 19.5 43.1
 1 24.7 49.8 
 1 24.7 




. . 
 .. ..

...
XM 6 =  . .
 XM 1 =  .. .. 




 1 22.7 
 1 22.7 48.2 
1 25.2
1 25.2 51.0
(similarly for XM 2).


? 506.1 1023.4
(X ′X)M 6 =  13386.3 26358.7 
52888.0
? ?
1.39 −0.053
′
′
−1
(X X)M 1 =
(X X)M 1 =
?
0.002
? ?
5.08 −0.098
′
′
−1
(X X)M 2 =
(X X)M 2 =
?
0.0019


10.8 0.29 −0.35

=
(X ′X)−1
0.014 −0.012 
M6
0.013
M1 : µ(x1, x2, x3) = β0 + β1x1
M2 : µ(x1, x2, x3) = β0 + β2x2
M6 : µ(x1, x2, x3) = β0 + β1x1 + β2x2
Some questions about design matrices
Recall three models under consideration for the bodyfat data
Osborne, ST512
52
10
10
81
28
39
41
42
1
2
3
4
157
158
159
160
age
race
16.6833
16.9500
17.1333
17.4000
46.8833
53.6000
53.6167
54.3167
sex
M
M
M
M
F
F
F
M
14
•
•
10
•
•
•
•
•
•
•
•
•
•
•
20
• ••
•
••
•
• •
•
•
•
•
••
••
•
•
•
•
•
••
•
•
30
•
x: age
40
50
• •
•
••
•
•
•
••
••
••
• ••• •
•
•• •
•
•••
•
•• •
••
• •
• •
•
• •
•
• •
•
•• •• •
• •
• •
•
•
•
•
•
•
•
•
•
• •
•
•
• •
•• •
•
•
•
•
•
•
•
•
•
60
•
• •
•
•
70
Resolution Run (5K), Raleigh 1/1/2002
•
Symbol Variable mean st. dev. variance
y
Pace
9.1
2.2
5.0
Age
35.1
14.7
216.5
x
15.1000
17.2667
17.2667
17.5000
5.38333
5.46667
5.51667
5.61667
pace
Recall the Resolution Run 5k race data
Obs
Summary statistics (n = 160):
(abbreviated)
Osborne, ST512
y: pace
12
10
8
6
54
55
with F (0.05, 1, 157) = 3.90. Since 26.2 >> 3.9, the linear model is
implausible when compared to the quadratic model.
Q1: Does β1 have the same interpretation in both models?
Q2: How can we compare the two models?
A2: Using F -ratios to compare nested models (see output next page).
R(β2|β0, β1)
F =
M S[E]f ull
(SS[R]f ull − SS[R]red)/1
=
M S[E]f ull
(113.6 − 1.1)/1
=
4.3
(SS[E]red − SS[E]f ull )/1
=
M S[E]f ull
(787.0 − 674.4)/1
=
4.3
= 26.2 !
2
β̂1
=
SE
Yi = β0 + β1xi + Ei for i = 1, . . . , 160
Compare this model with the (previously discarded) SLR model
• σ 2 is the unknown error variance of paces given age x.
• β = (β0, β1, β2)′ is a vector of unknown regression parameters
where Ei ∼ N (0, σ 2).
iid
Yi = β0 + β1xi + β2x2i + Ei for i = 1, . . . , 160
Quadratic model for pace (Y ) as a function of age (x):
Osborne, ST512
DF
1
158
159
DF
1
1
DF
2
157
159
Variable
Intercept
age
age2
DF
1
1
1
Parameter
Estimate
11.78503
-0.19699
0.00294
Standard
Error
0.70216
0.04113
0.00057380
2.07265
9.12063
Sum of
Squares
113.64500
674.44972
788.09472
t Value
16.78
-4.79
5.12
R-Square
Adj R-Sq
0.1442
0.1333
Pr > F
<.0001
Type I SS
13310
1.09650
112.54850
F Value
13.23
Pr > |t|
<.0001
<.0001
<.0001
Mean
Square
56.82250
4.29586
Analysis of Variance
Model: MODEL2
Pr > F
0.6396
Pr > |t|
<.0001
0.6396
0.0014
-0.0049
F Value
0.22
t Value
19.51
0.47
R-Square
Adj R-Sq
Mean
Square
1.09650
4.98100
Standard
Error
0.45724
0.01203
2.23182
9.12063
Sum of
Squares
1.09650
786.99821
788.09472
Parameter
Estimate
8.92271
0.00564
Root MSE
Dependent Mean
Source
Model
Error
Corrected Total
Variable
Intercept
age
Root MSE
Dependent Mean
Source
Model
Error
Corrected Total
Analysis of Variance
/* age2 defined in data step as age*age */
PROC REG;
MODEL pace=age;
MODEL pace=age age2/ss1; /* ss1 generates sequential sums of squares */
/* only the 2nd model statement really necessary */
RUN;
The SAS System
1
The REG Procedure
Model: MODEL1
56
Osborne, ST512
or
40
AGE
60
µ̂(x) = 11.785 − 0.197x + 0.00294x2
20
Resolution Run (5k), 1/1/2004
µ̂(age) = 11.785 − 0.197age + 0.00294age2.
Fitted model is
pace
Osborne, ST512
16
14
12
10
8
6
80
57
58
8
10
12
14
Girth
16
18
20
65
70
Height
75
80
85
10
20
30
A scatterplot matrix
40
50
Volume
60
70
µ(x1, x2) = E(Y |x1, x2) = β0 + β1x1 + β2x2.
For trees with x1, x2 the model for mean volume is
where errors are assumed iid normal w/ constant variance σ 2.
Yi = β0 + β1xi1 + β2xi2 + Ei for i = 1, . . . , n
Given x1 and x2, a MLR model for these data is given by
• Yi: volume of timber, in cubic feet.
• xi2: “height” (in feet)
• xi1: “girth”, tree diameter in inches
Inference for response Y given predictor xi·
A random sample of n = 31 trees is drawn from a population of
trees. On each tree, indexed by i, three variables are measured:
Osborne, ST512
85
80
75
70
65
20
18
16
14
12
10
8
70
60
50
40
30
20
10
59
Variable
Intercept
Girth
Height
Variable
Intercept
Girth
Height
R-Square
Adj R-Sq
Sum of
Squares
7684.16251
421.92136
8106.08387
Standard
Error
8.63823
0.26426
0.13015
Girth
0.4321713812
0.0698357838
-0.017860301
Covariance of Estimates
Parameter
Estimate
-57.98766
4.70816
0.33925
Parameter Estimates
3.88183
30.17097
12.86612
DF
2
28
30
Intercept
74.6189461
0.4321713812
-1.050768886
DF
1
1
1
Root MSE
Dependent Mean
Coeff Var
Source
Model
Error
Corrected Total
Pr > |t|
<.0001
<.0001
0.0145
F Value
254.97
Height
-1.050768886
-0.017860301
0.0169393298
t Value
-6.71
17.82
2.61
0.9480
0.9442
Mean
Square
3842.08126
15.06862
Pr > F
<.0001
• Obtain a 95% prediction interval of y0, the volume from an individual tree sampled from this population of 80 footers, with girth
15 inches.
SAS generates β̂ and Vd
ar(β̂) = M SE ∗ (X ′X)−1
• Estimate the mean volume among these trees, along with a standard error and 95% confidence interval.
Some questions involving linear combinations of regression coefficients
Consider all trees with girth x01 = 15 in and height x02 = 80 f t .
Osborne, ST512
E(a′W ) = a′µW
Var(a′W ) = a′ΣW a.
60
http://www.stat.ncsu.edu/people/dickey/courses/st512/crsnotes/notes 1.htm
−1
100
32
15
Girth
80
Height
.
Volume
p
sepred
39.7748
0.84918
p
95% Prediction limits? Use ±t(.025, 28) .72 + M S(E).
treenumber
Obs
Substitution of β̂ and Σ̂ = M SE(X X) gives the estimates:


−58.0
µ̂(x0) = (1, 15, 80)  4.71 
0.34
= 39.8
 

1
74.62 0.43 −1.05
c




))
=
(1,
15,
80)
Var(µ̂(x
15
0.43
0.070
−0.018
0
80
−1.05 −0.018 0.017
= 0.72
√
c
.72 = 0.849
SE(µ̂(x
0 )) =
which can be obtained using PROC REG and the missing y trick:
′
E(x′0β̂) = x′0β
Var(x′0β̂) = x′0Σ̂x0
Inference for the mean response in MLR
Consider all trees with Girth 15 and Height 80 To estimate mean
volume among these trees, along with an estimated standard error,
take x′0 = (1, 15, 80) and consider µ̂(x0) = x′0β̂.
(See
Moments of linear combinations of random vectors (Appendix B)
Let W denote a p × 1 random vector with mean µW and covariance
matrix ΣW . Suppose a is a p × 1 (fixed) vector of coefficients. Then
Osborne, ST512
61
2
ry1·2,...,p
=
R(β1|β0, β2, . . . , βp)
.
SS[T ot] − R(β2, β3, . . . , βp|β0)
Note also (see Figure 11.7) that
2
The partial coeff. of determination is ry1·2,3,···
,p .
ry1·2,3,··· ,p = correlation between ey·2,3,...,p and e1·2,3,...,p.
Call these sets of residuals ey·2,3,...,p and e1·2,3,...,p respectively. The
partial correlation between y and x1 after accounting for the linear
association between y and x2, x3, . . . , xp is defined as
Y = β0 + β2x2 + · · · + βpxp + E
X1 = β0 + β2x2 + · · · + βpxp + E
is defined as the correlation coefficient between the residuals computed from the two regressions below:
E(Y |x1, x2, . . . , xp) = β0 + β1x1 + β2x2 + · · · + βpxp
Partial correlations
The partial correlation coefficient for x1 in the MLR
Osborne, ST512
43.1
49.8
51.9
54.3
42.2
x2
11.9
22.8
18.7
20.1
12.9
y
15.2190
19.6764
24.8195
24.0481
14.8762
py_1
ey_1
e2_1
0.04571
1.05061
20.0494
e1_2
1.34939
0.60956
4.74782
1.72013
1.74728
ey_2
13.2827 -1.38267
19.0215 3.77847
20.8203 -2.12028
22.8760 -2.77599
12.5118 0.38822
py_2
62
DF
1
1
1
Variable
Intercept
x1
x2
.
0.03062
0.23176
Squared
Partial
Corr Type II
Note: partial correlations obtained in SAS using PCORR2 option:
Q: Write the coefficients of determination in terms of extra sums of
squares, using R(·|·) notation.
Q: Anything wrong with throwing both x1 and x2 in the final model?
Q: If you had to choose one variable or the other from x1 and x2,
which would it be?
2
2
= 0.03062 and ry2·1
= 0.23176.
ry1·2
The partial correlation coefficient between y and x1 after accounting
for x2 is ry1·2 = 0.17 and the partial for x2 after accounting for x1 is
ry2·1 = 0.48. The partial coefficients of determination are
19.5
24.7
30.7
29.8
19.1
x1
-3.31903 -2.48145
3.12360 -0.78756
-6.11952 -4.46384
-3.94805 -1.19739
-1.97616 -2.99637
(abbreviated)
20 25.2 51.0 21.1 20.1050 0.99500 -0.06892
1
2
3
4
5
Obs
bodyfat data
Bodyfat data, compare models 1,2 and 6 (ignore x3.)
Osborne, ST512
63
x1
----+------+------+------+------+------+------+------+------+------+---5 +
+
|
1
|
|
2
|
|
|
y |
1
1
|
|
|
|
1
1
|
|
1
1
1
|
0 +
1
+
|
1
|
|
1 1
1
|
|
1
|
|
1
|
|
1
1
|
|
|
|
1
|
-5 +
+
----+------+------+------+------+------+------+------+------+------+----4
-3
-2
-1
0
1
2
3
4
5
The SAS System
The REG Procedure
Partial Regression Residual Plot
• using the PARTIAL command in the MODEL statement of PROC
REG
• in SAS/INSIGHT by clicking
• Analyze • Fit XY • Output tab • Plots: (Partial Leverage).
is called a partial regression plot for x1 or a partial leverage plot of
x1 in the MLR. They can be generated
X1 = β0 + β2x2 + · · · + β3xp + E
versus the residuals from the regression
Y = β0 + β2x2 + · · · + β3xp + E
Partial regression plots
A plot of the residuals from the regression
Osborne, ST512
x2
64
A3: They can illuminate possible outliers.
A2: They can convey info about nonlinear associations between y
and xi after accounting for linear associations with other variables.
A1: They convey info. about linear associations between y and a candidate independent variable xi after accounting for linear associations
between y and other independent variables x1, x2, . . . , xi−1, xi+1, . . . , xp
Q: What can these plots tell us?
-10
-5
0
y
5
Partial Regression Residual Plot
-------+------+------+------+------+------+------+------+------+------|
|
+
+
|
1
|
|
1
1
1
|
|
1
1 1
|
|
1
1 1
|
+
1
1
1
+
|
1
|
|
1
|
|
1
1
1
|
|
1
|
+
+
|
1
|
|
|
|
|
|
|
+
+
-------+------+------+------+------+------+------+------+------+-------5
-4
-3
-2
-1
0
1
2
3
Osborne, ST512
65
• The sorted residuals can be plotted against the normal inverse
of the empirical CDF of the residuals in a normal plot to assess
the normal distributional assumption. A nonlinear association in
such a q-q plot indicates nonnormality.
• Residuals can be plotted against predicted values to look for inhomogeneity of variance (heteroscedasticity).
• Residuals can be plotted against independent variables to check
for model inadequacy.
Residual diagnostics
5. Estimate θ, the peak age to run a 5k in the fastest time. Is θ
a linear function of regression parameters? Can you obtain an
unbiased estimate of the standard error of θ?
4. At what rate is µ(x) changing with age? Estimate the appropriate function.
3. Explain the difference between the two intervals in questions 1
and 2.
2. Obtain a 95% prediction interval for your time if you are about
to run the race.
1. Suppose you are a local 32 yr-old male runner. Regarding these
data as randomly sampled from the population of all local runners, fit a quadratic regression function and use it to obtain an
estimate of the mean 5k pace in your cohort of all 32 yr-old male
runners. Report a standard error and 95% confidence interval.
Some exercises (hint: use matrix algebra or SAS).
Osborne, ST512
66
pi =
Rank of e(i)
.
N +1
These can be used to obtain the correspoding theoretical quantiles
via
q(i) = z(1 − p(i)).
The empirical cumulative probability associated with e(i) is
Rank of e(i) = i.
As an illustration, we’ll obtain the eij and the q(i) for the data in a
table. To do this, we’ll need the ranks of the residuals
3. Plot the (ordered) residuals on the vertical axis versus the (ordered) theoretical quantiles on the horizontal axis.
2. For each i = 1, . . . , n compute the expected quantile from
i
q(i) = z(1 −
).
n+1
e(1) ≤ e(2) ≤ · · · ≤ e(n).
1. Obtain the observed quantiles by ordering the residuals:
Osborne, ST512
age
28
39
41
42
40
23
40
42
28
18
32
51
46
17
39
6
52
10
10
81
Obs
1
2
3
4
5
6
7
8
9
10
11
152
153
154
155
156
157
158
159
160
13.0000
13.0000
13.0000
14.4667
14.4667
15.1000
17.2667
17.2667
17.5000
5.3833
5.4667
5.5167
5.6167
5.9333
6.0000
6.0167
6.0667
6.0833
6.1833
6.4333
pace
10.9401
10.4356
10.2778
7.7671
11.4534
11.0579
10.9473
10.9473
14.7178
7.5837
7.7671
7.8735
7.9351
7.8175
7.7250
7.8175
7.9351
7.5837
8.0067
7.5718
yhat
-2.20040
-2.30046
-2.35681
-2.31841
-1.88416
-1.72501
-1.80083
-1.86841
-1.50040
-1.82335
-1.13847
...
2.05990
2.56445
2.72222
6.69954
3.01324
4.04215
6.31937
6.31937
2.78223
resid
140.0
147.0
150.0
160.0
155.0
157.0
158.5
158.5
152.0
14.0
10.0
6.0
9.0
18.0
25.0
23.0
19.0
27.0
22.0
42.0
rankresid
ods listing close;
proc reg data=running;
model pace=sexf age age2;
output out=resids p=yhat r=resid;
run;
proc rank data=resids out=resids2;
ranks rankresid;
var resid;
run;
data resids2;
set resids2;
ecdf=rankresid/(160+1);
*160 runners;
q=probit(ecdf);
run;
ods listing ;
proc print data=resids2 ;
var age pace yhat resid rankresid ecdf q;
run;
proc gplot data=resids;
plot resid*q;
run;
The SAS System
Osborne, ST512
0.86957
0.91304
0.93168
0.99379
0.96273
0.97516
0.98447
0.98447
0.94410
0.08696
0.06211
0.03727
0.05590
0.11180
0.15528
0.14286
0.11801
0.16770
0.13665
0.26087
ecdf
1
1.12434
1.35974
1.48840
2.49991
1.78332
1.96263
2.15636
2.15636
1.59015
-1.35974
-1.53728
-1.78332
-1.59015
-1.21700
-1.01405
-1.06757
-1.18498
-0.96329
-1.09551
-0.64067
q
67
y
10 20 30
••
•
••
•• • •••••
• ••••
5 10 15 20
x
••
c
•
• ••• ••
• •
•
•
•
•
• •
••
5 10 15 20
x
• •
•
4. Model fits
3. Nonnormality
2. Nonlinearity
5
•
• •
•
•
• • •
•
••
•
•
0
•
•
•
••• • •
•
• •
c
••
•
•
15
20
predicted
•
•
•
b
••
•
10 20 30
predicted
••
10
•
• ••
•••••• • • •
a
•
•
•
40
•
•
•
•• • •
•
•••••
••
•
•
•
•
••
d
•
•• •
• •
• •
-2 -1 0 1 2
-2 -1 0 1 2
Quantiles of Standard NormalQuantiles of Standard Normal
••
••
•••
• ••••••
c
••
•••••
• ••
• • • • ••••••
-2 -1 0 1 2
-2 -1 0 1 2
Quantiles of Standard NormalQuantiles of Standard Normal
•• •
••••••••
• ••
a
•
• •
•• • • •• •
d
10 15 20
x
• •• • ••• • • •
b
• •
• • • • •
• • •
5 10 15 20
x
•
1. Heteroscedasticity
y
0 20 40 60
40
ordered residuals
-10 10
a
residual
-2 0
2
y
20
0
y
20
10
ordered residuals
-10 0 10
residual
-10
0
10
residual
-10
10
ordered residuals
-5 5 15
ordered residuals
-2 0 2
residual
-5 5 15
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
15 20
predicted
•
d
•
10 15 20
predicted
•
25
•
•
25
• •
•
•
•••• •• • •
10
•
5
••
b
An exercise: Match up letters a,b,c,d with the model violation
Osborne, ST512
68
69
Binding Percentage Sample mean
29.6 24.3 28.5 32
28.6
27.3 32.6 30.8 34.8
31.4
5.8 6.2 11 8.3
7.8
21.6 17.4 18.3 19
19.1
29.2 32.8 25 24.2
27.8
1
2
3
4
binding fraction boxplot
5
Q: Are the population means for these 5 treatments plausibly equal?
Q: Do these (sample) treatment means differ significantly?
A completely randomized design (CRD) was used.
Antibiotic
Penicillin G
Tetracyclin
Streptomycin
Erythromycin
Chloramphenicol
ANOVA revisited:
Following data come from study investigating binding fraction for
several antibiotics using n = 20 bovine serum samples:
ST 512
Topic: The general linear model
Reading Ch. 12, Ch. 16
Osborne, ST512
35
30
25
20
15
10
5
70
Conclusion? (Use F (0.05, 4, 15) = 3.06)
Parameter estimates µ̂, τ̂1, τ̂2, τ̂3, τ̂4, τ̂5? (ȳ++ = 22.935)?
Standard errors of parameter estimates?
Sum of Mean
Source
d.f. squares Square F
Treatments 4
1481
370 41
Error
15
136
9
Total
19 1617
To test H0 : τ1 = τ2 = . . . = τ5 = 0, we just carry out one-way
ANOVA:
• σ 2 - (population) variance of bf for a given antibiotic
• τi - difference between (population) mean for treatment i and µ
• µ - overall population mean (avg of 5 treatment population means)
for i = 1, . . . , 5 and j = 1, . . . , 4, where Eij are i.i.d. N (0, σ 2) errors.
Unknown parameters
Yij = µ + τi + Eij
Modelling the binding fraction expt
One model parameterizes antibiotic effects as differences from mean:
Osborne, ST512
71
SS[T OT ] =
SS[E] =
SS[T ] =
XX
XX
XX
(yij − ȳ++)2
(yij − ȳi+)2
(ȳi+ − ȳ++)2
SS[E]
(N −t)
Mean
Square
F
SS[T ]
M S[T ]
M S[T ] = (t−1) F = M
S[E]
N − t SS[E] M S[E] =
N − 1 SS[T OT ]
Sum of
squares
SS[T ]
• model is overparameterized (6 parameters, 5 means)
• standard errors for parameter estimates (β̂) can’t be obtained
• problem: (X ′X)−1 does not exist
where a design matrix X of 1’s and 0’s of dimension (20 × 6) could
be specified. Note that β0 = µ and βj = τj .
Yi = β0 + β1xi1 + β2xi2 + β3xi3 + β4xi4 + β5xi5 + Ei i = 1, . . . , 20
The MLR model is
The linear model µij = E(Yij ) = µ + τi could be fit using MLR with
5 indicator variables x1, . . . , x5 for the 5 antibiotics. Let
(
1 if treatment j
xij =
0 else
where
Error
Total
Source
d.f.
Treatments t − 1
Table for balanced one-way ANOVA
Yij denotes j th observation receiving level i of treatment factor with
t levels, for a total of N observations.
Osborne, ST512
72


















X=
















1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
0
0
0
0



































where Ei ∼ N (0, σ 2). The X matrix looks like
iid
Yi = β0 + β1xi1 + β2xi2 + β3xi3 + β4xi4 + Ei i = 1, . . . , 20
Here, reparameterizing using 5 − 1 indicator variables leads to a
general linear model. Define x1, x2, x3, x4 as before. Then the MLR
model is
A general linear model
Models which parameterize the effects of classification factors this
way are general linear models. One-way ANOVA and linear regression models are general linear models. The linearity pertains to the
parameters, not the explanatory variables.
Osborne, ST512
73

(Compare with page 1 69.)
µ̂(1, 0, 0, 0)
µ̂(0, 1, 0, 0)
µ̂(0, 0, 1, 0)
µ̂(0, 0, 0, 1)
µ̂(0, 0, 0, 0)
=
=
=
=
=
β̂0 + β̂1
β̂0 + β̂2
β̂0 + β̂3
β̂0 + β̂4
β̂0
=
=
=
=
=
28.6
31.4
7.8
19.1
27.8
Estimates for the five treatment means obtained by substitution of
β̂ into µ(x1, x2, x3, x4) = β0 + β1x1 + β2x2 + β3x3 + β4x4

27.8

0.8 




β̂ = (X ′X)−1X ′Y = 
3.6 


 −20.0 
−8.7
For the one-way ANOVA,
• continuous covariates (as opposed to indicators) can be added
and it is still a general linear model
• (X ′X)−1 exists
Remarks:
Osborne, ST512
2.3 −2.3 −2.3 −2.3 −2.3
4.5 2.3 2.3 2.3 


4.5 2.3 2.3 

4.5 2.3 
4.5

74
µ̂(1, 0, 0, 0)
µ̂(0, 1, 0, 0)
µ̂(0, 0, 1, 0)
µ̂(0, 0, 0, 1)
µ̂(0, 0, 0, 0)
=
=
=
=
=
β̂0 + β̂1
β̂0 + β̂2
β̂0 + β̂3
β̂0 + β̂4
β̂0
=
=
=
=
=
a′β̂
b′β̂
c′β̂
d′β̂
β̂0
Recall from one-way ANOVA that
s
r
M
S[E]
9.1 √
c i+) =
SE(ȳ
= 2.3 = 1.5
=
ni
4
c β̂0) = Var(
c β̂0 + β̂j )
a′Σ̂a = b′Σ̂b = c′Σ̂c = d′Σ̂d = Σ̂11 = 2.3 = Var(
√
so the estimated SE for any sample treatment mean is 2.3 = 1.5.
and
Then
a′ = (1, 1, 0, 0, 0), b′ = (1, 0, 1, 0, 0), c′ = (1, 0, 0, 1, 0), d′ = (1, 0, 0, 0, 1).
Let a, b, c, d be defined by
Σ̂ = M S[E](X ′X)−1




=


For standard errors, use Σ̂:
Osborne, ST512
75
µ(x1, x2, x3)
µ(x1, x2, x3)
µ(x1, x2, x3)
µ(x1, x2, x3)
µ(x1, x2, x3)
µ(x1, x2, x3)
=
=
=
=
=
=
20
40
AGE
60
Resolution Run (5k), 1/1/2004
Men
Women
80
x4
x5
β0
β0 + β3x3
β0 + β1x1
β0 + β1x1 + β2x2
β0 + β1x1 + β2x2 + β3x3
β0 + β1x1 + β2x2 + β3x3 + β4 x
1 x3 +β5 x
2 x3
|{z}
|{z}
A general linear model for 5k times of men AND women
(Resolution Run, Jan. 1, 2004, Centennial Campus)
Quadratic model µ(x) = β0 +β1x+β1x2 was used for the association
between mean pace for male runners and age x. Consider modelling
female race times as well. How could the model be extended to
incorporate sex differences? Let x2 = x2 and let an indicator variable
x3 be defined by
(
1 female
x3 =
0 male
Some candidate models:
Osborne, ST512
pace
16
14
12
10
8
6
DF
1
Variable
Intercept
sexf
DF
1
1
Root MSE
Source
Model
Error
Corrected Total
Variable
Intercept
Root MSE
Source
Model
Error
Corrected Total
R-Square
0.0000
t Value
40.76
6.61
Pr > F
<.0001
Pr > |t|
<.0001
<.0001
F Value
43.70
0.2167
Pr > F
.
Pr > |t|
<.0001
F Value
.
t Value
51.82
Mean
Square
170.74137
3.90730
Standard
Error
0.20280
0.31819
1.97669
Sum of
Squares
170.74137
617.35335
788.09472
Model: MODEL2
Parameter
Estimate
8.26614
2.10335
DF
1
158
159
R-Square
Mean
Square
.
4.95657
Standard
Error
0.17601
2.22634
Sum of
Squares
0
788.09472
788.09472
Parameter
Estimate
9.12063
DF
0
159
159
data race5k;
set race5k;
sexf=(sex="F");
age2=age*age; agef=age*sexf; age2f=age2*sexf;
run;
proc reg data=one ;
model pace=;
model pace=sexf; /* equivalent to two-sample t-test */
model pace=age age2;
model pace=sexf age age2;
model pace=sexf age age2 agef age2f;
test agef=0, age2f=0;
run;
The REG Procedure
Model: MODEL1
Osborne, ST512
76
DF
1
1
1
DF
1
1
1
1
R-Square
0.1442
t Value
15.85
7.44
-4.81
5.67
Pr > F
<.0001
for women
for men
Pr > |t|
<.0001
<.0001
<.0001
<.0001
F Value
30.33
0.3684
Pr > F
<.0001
Pr > |t|
<.0001
<.0001
<.0001
F Value
13.23
t Value
16.78
-4.79
5.12
Mean
Square
96.78284
3.19068
Standard
Error
0.64228
0.29535
0.03562
0.00049481
1.78625
Sum of
Squares
290.34851
497.74621
788.09472
Model: MODEL5
Parameter
Estimate
10.18317
2.19792
-0.17146
0.00281
DF
3
156
159
R-Square
Mean
Square
56.82250
4.29586
Standard
Error
0.70216
0.04113
0.00057380
2.07265
Sum of
Squares
113.64500
674.44972
788.09472
Parameter
Estimate
11.78503
-0.19699
0.00294
DF
2
157
159
Fitted model is


β0 + β1x + β2x2



 = 10.18 − 0.17x + 0.0028x2
µ(x) =

β0 + β1x + β2x2 + β3



 = 10.18 + 2.20 − 0.17x + 0.0028x2
Variable
Intercept
sexf
age
age2
Root MSE
Source
Model
Error
Corrected Total
Variable
Intercept
age
age2
Root MSE
Source
Model
Error
Corrected Total
Model: MODEL4
(For MODEL3 output, see linear and quadratic fits from multiple regression notes)
Osborne, ST512
77
µ(x) =
0
DF
1
1
1
1
1
1
1
3
R-Square
Standard
Error
0.88641
1.23237
0.04842
0.00064628
0.07298
0.00103
1.79206
Parameter
Estimate
10.60848
1.25728
-0.19986
0.00321
0.06882
-0.00103
DF
5
154
159
0.3725
4
5

β0 + β3 + (β1 + β4)x + (β2 + β5)x2





10.61 + 1.25 + (−0.20 + 0.07)x + (0.0032 − 0.0010)x2




11.86 − 0.13x + 0.0022x2
2
Pr > F
<.0001
Pr > |t|
<.0001
0.3092
<.0001
<.0001
0.3471
0.3217
F Value
18.28
t Value
11.97
1.02
-4.13
4.96
0.94
-0.99
Mean
Square
58.70566
3.21147


β0 + β1x + β2x2 + β3(0) + β4(0) + β5(0)





10.61 − 0.20x + 0.0032x2



β + β x + β x2 + β (1) + β (x) + β (x2)
Variable
Intercept
sexf
age
age2
agef
age2f
Root MSE
Source
Model
Error
Corrected Total
Sum of
Squares
293.52828
494.56644
788.09472
Model: MODEL6
women
men
78
20
20
Females
Males
Females
Males
AGE
40
AGE
Model 6
40
Model 5
60
60
80
80
79
Which model is “better”? What do we mean by “better?” Is there
a test we can use to compare these models?
Osborne, ST512
16
14
Osborne, ST512
pace
pace
12
10
8
6
16
14
12
10
8
6
Comparison of models 5 and 6
80
2
154
DF
1.58988
3.21147
Mean
Square
Which model do we choose at this point?
Numerator
Denominator
Source
0.50
F Value
Test 1 Results for Dependent Variable pace
The REG Procedure
Model: MODEL6
0.6105
Pr > F
proc reg;
model pace=age age2 sexf agef age2f;
test agef=0, age2f=0;
run;
to get the following model selection F -ratio in the output:
In SAS, you could use
The observed F -ratio is not significant on df = 2, 154.
The F -ratio
R(β4, β5|β0, β1, β2, β3)/(5 − 3) 3.2/2
1.6
F =
=
= 0.5
=
M S[E]f
3.21
3.21
R(β4, β5|β0, β1, β2) = SS[R]f − SS[R]r = 293.5 − 290.3 = 3.0
Extra sum of squares:
reduced: µ(x1, x2, x3) = β0 + β1x1 + β2x2 + β3x3
full: µ(x1, x2, x3) = β0 + β1x1 + β2x2 + β3x3 + β4x1x3 + β5x2x3
Osborne, ST512
81
−β1
2β2
θW =
−(β1 + β4)
2(β2 + β5)
Q: What criterion can we use to assess the estimation?
Q: Which of these estimates are closest to the true peak(s)?
Q: Which of these estimates is “better”?
Things to ponder:
These are nonlinear functions of regression parameters. Note that
acceptance of any model but 6 implies equality of these peak ages.
(
30.5 different intercepts model (5)
θ̂W =
30.1 full model (6)
(
30.5 different intercepts model (5)
θ̂M =
31.1 full model (6)
θM =
More stuff
Estimate the “peak” running age for men and for women. Is it
different? θM and θW denote peak running ages for men and women
respectively. Using calculus on the model 6 regression,
Osborne, ST512
82
Source
Model
Error
Corrected Total
Dependent Variable: y
DF
3
16
19
Sum of
Squares
797.800000
2334.400000
3132.200000
The GLM Procedure
Mean Square
265.933333
145.900000
Q: Is there evidence of a vitamin supplement effect?
F Value
1.82
Pr > F
0.1836
Diet
y(g)
Diet
y
Diet
y
Diet
y
1
48
2
65
3
79
4
59
1
67
2
49
3
52
4
50
1
78
2
37
3
63
4
59
1
69
2
75
3
65
4
42
1
53
2
63
3
67
4
34
1 ȳ1+ = 63 2 ȳ2+ = 57.8 3 ȳ3+ = 65.2 4 ȳ4+ = 48.8
1 s1 = 12.3 2
s2 = 14.9
3
s3 = 9.7
4
s4 = 10.9
An nutrition example
A nutrition scientist conducted an experiment to evaluate the effects
of four vitamin supplements on the weight gain of laboratory animals.
The experiment was conducted in a completely randomized design
with N = 20 animals randomized to a = 4 supplement groups, each
with sample size n ≡ 5. The response variable of interest is weight
gain, but calorie intake z was measured concomitantly.
Analysis of covariance, ANCOVA
Covariates are predictive responses. Associations between covariates
z and the main response variable of interest y can be used to reduce
unexplained variation σ 2.
Osborne, ST512
y
48
67
78
69
53
z Diet y z Diet y z Diet y z
350 2 65 400 3 79 510 4 59 530
440 2 49 450 3 52 410 4 50 520
440 2 37 370 3 63 470 4 59 520
510 2 73 530 3 65 470 4 42 510
470 2 63 420 3 67 480 4 34 430
83
Exercise: specify the parametric mean weight gain for the first subject in each treatment group, conditional on their caloric intakes.
and errors Ei ∼ N (0, σ 2).
iid
where xij is an indicator variable for subject i receiving vitamin
supplement j:
(
1 subject i receives supplement j
xij =
0 else
Yi = β0 + β1xi1 + β2xi2 + β3xi3 + βz zi + Ei for i = 1, . . . , 20
Q: How and why could these new data be incorporated into analysis?
A: ANCOVA can be used to reduce unexplained variation.
Model, given zi,
Diet
1
1
1
1
1
But calorie intake z was measured concomitantly:
Osborne, ST512
DF
3
1
Source
diet
z
Type III SS
1537.071659
1153.880373
Type I SS
797.800000
1153.880373
Mean Square
512.357220
1153.880373
Mean Square
265.933333
1153.880373
F Value
6.51
14.66
F Value
3.38
14.66
Pr > F
0.0049
0.0016
Pr > F
0.0463
0.0016
Pr > F
0.0038
84
NOTE: the drop in
√
proc glm;
class diet;
model y=diet z;
means diet;
lsmeans diet/stderr;
run;
M SE (was σ̂ ≈ 12g is σ̂ ≈ 9g)
FYI: this model was fit with the following code:
To test for a diet effect: H0 : β1 = β2 = β3 = 0, use the type III
F -ratio, on 3 and 15 numerator and denominator degrees of freedom.
(Note that this is a comparison of nested models.)
Q: Conclusion?
DF
3
1
F Value
6.20
y Mean
58.70000
Mean Square
487.920093
78.701308
Root MSE
8.871376
Sum of
Squares
1951.680373
1180.519627
3132.200000
Coeff Var
15.11308
DF
4
15
19
Source
diet
z
R-Square
0.623102
Source
Model
Error
Corrected Total
Dependent Variable: y
The GLM Procedure
Class Level Information
Class
Levels
Values
diet
4
1 2 3 4
Proceeding with MLR analysis of this general linear model:
Osborne, ST512
5
5
5
5
N
63.0000000
57.8000000
65.2000000
48.8000000
12.2678441
14.8727940
9.6540147
10.8949530
-------------y------------Mean
Std Dev
442.000000
434.000000
468.000000
502.000000
58.9067059
61.0737259
36.3318042
40.8656335
-------------z------------Mean
Std Dev
85
ȳ1,a
ȳ2,a
ȳ3,a
ȳ4,a
=
=
=
=
β̂0 + β̂1 + β̂z (461.5)
β̂0 + β̂2 + β̂z (461.5)
β̂0 + β̂3 + β̂z (461.5)
β̂0 + β̂z (461.5)
Here, z̄ = (442 + 434 + 468 + 502)/4 = 461.5. The adjusted means
are then just
Adjusted means are estimated mean weight gains at a common reference value (sample mean, z̄) of the covariate, z.
2. weight gain depends on caloric intake
1. caloric intake may vary by diet (presumably by chance, not because of diet)
Unadjusted means do not make any adjustment for the facts that
These means y are computed without taking z into account, so they
are called unadjusted means.
1
2
3
4
Level of
diet
The GLM Procedure
Adjusted and unadjusted means
Recall the sample mean weight gains for the four diets
(generated by the means diet; statement in proc glm):
Osborne, ST512
86
B
B
B
B
B
22.41252629
6.19932022
6.35678835
5.80625371
.
0.04394140
-1.59
3.92
3.22
3.81
.
3.83
t Value
0.1324
0.0014
0.0058
0.0017
.
0.0016
Pr > |t|
=
=
=
=
Standard errors of ȳj,a
′
−35.7 + 24.3 + 0.17(461.5) = 66.3
−35.7 + 20.4 + 0.17(461.5) = 62.4
−35.7 + 22.1 + 0.17(461.5) = 64.1
−35.7 + +0.17(461.5) = 42.0
What is the standard error of c′β̂?
Consider ȳ2,a. What vector c is needed so that c β̂ = ȳ2,a?
ȳ1,a
ȳ2,a
ȳ3,a
ȳ4,a
Substitution of β̂ into the expressions for adjusted means yields
NOTE: The X’X matrix has been found to be singular, and a generalized
inverse was used to solve the normal equations. Terms whose
estimates are followed by the letter ’B’ are not uniquely estimable.
-35.66310108
24.29519136
20.44121688
22.12060844
0.00000000
0.16825319
Intercept
diet
diet
diet
diet
z
1
2
3
4
Estimate
Parameter
Standard
Error
To get SAS to report the estimated regression parameter vector β̂,
use the solution option in the model statement. The default parameterization is the one we’ve adopted here where β0 is the mean
of the last level of the classification treatment factor:
Osborne, ST512
87
66.2809372
62.4269627
64.1063543
41.9857458
y LSMEAN
4.0588750
4.1473443
3.9776677
4.3482563
Standard
Error
<.0001
<.0001
<.0001
<.0001
Pr > |t|
DF
3
16
19
Sum of
Squares
14095.00000
40760.00000
54855.00000
The GLM Procedure
Mean Square
4698.33333
2547.50000
F Value
1.84
Pr > F
0.1798
Q: Why are the standard errors for the adjusted means different?
Q: Which adjusted means require the most adjustment?
Q: If you are a lab animal and you want to gain weight, which diet(s)
would you choose?
Q: Among the diets, which we’ve concluded are different, what are
the differences? (Look at the means, have a guess.)
A: No evidence that treatment affects covariate.
Source
Model (diet)
Error
Corrected Total
Dependent Variable: z
Concerns:
Aside from the usual residual-based checks for model adequacy, does
treatment affect the covariate? To check this, one could carry out a
one-way ANOVA treating z as a response variable and check for a
diet effect on the mean of z:
1
2
3
4
diet
The GLM Procedure
Least Squares Means
To get SAS to produce the adjusted means and estimated standard
errors, use an lsmeans statement for the factor diet
Osborne, ST512
Osborne, ST512
80
70
y: grams
50
60
40
30
350
400
Diet 1
Diet 2
Diet 3
Diet 4
450
z: calories
500
Vitamin supplement ANCOVA
550
88
89
0
100
200
numpass
300
400
500
SLR model : µ(x) = β0 + β1x
one-factor ANOVA model : µij = µ + τi
Two models for mean plant height:
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
height(cm)
21
i : trt group x : Number of passes
yij : Height(cm)
1
0
20.7 15.9 17.8 17.6
2
25
12.9 13.4 12.7 9.0
75
11.8 12.6 11.4 12.1
3
4
200
7.6 9.5 9.9 9.0
500
7.8 9.0 8.5 6.7
5
Lack-of-fit of a SLR model (supplementary to textbook)
Hiking example: completely randomized experiment involving alpine
meadows in the White Mountains of New Hampshire. N = 20 lanes
of dimension 0.5m × 1.5m randomized to 5 trampling treatments:
Osborne, ST512
Levels
5
Standard
Error
0.80592
0.00331
R-Square
Adj R-Sq
DF
1
3
DF
0
3
Source
numpass
cnumpass
Type III SS
0.0000000
101.8666772
Mean Square
.
33.9555591
Mean Square
141.2953228
33.9555591
F Value
.
16.47
F Value
68.53
16.47
1
Pr > F
0.0004
Pr > F
.
<.0001
Pr > F
<.0001
<.0001
Pr > F
<.0001
Pr > |t|
<.0001
0.0004
F Value
29.48
y Mean
11.79500
Mean Square
60.7905000
2.0618333
Root MSE
1.435909
Type I SS
141.2953228
101.8666772
Coeff Var
12.17387
DF
4
15
19
0.5155
0.4886
F Value
19.15
t Value
17.51
-4.38
Mean
Square
141.29532
7.37745
Values
0 25 75 200 500
Sum of
Squares
243.1620000
30.9275000
274.0895000
height(cm)
Source
numpass
cnumpass
R-Square
0.887163
Source
Model
Error
Corrected Total
Parameter
Estimate
14.11334
-0.01449
2.71615
11.79500
Sum of
Squares
141.29532
132.79418
274.08950
The GLM Procedure
Class Level Information
DF
1
1
DF
1
18
19
Class
cnumpass
Label
Intercept
Dependent Variable: y
Variable
Intercept
numpass
Root MSE
Dependent Mean
Source
Model
Error
Corrected Total
The SAS System
The REG Procedure
Dependent Variable: y height(cm)
Analysis of Variance
proc glm data=one;
class cnumpass;
model y=numpass cnumpass;
run;
proc reg data=one;
model y=numpass;
run;
Osborne, ST512
90
SS[lack of fit]/(t − 1 − p)
M S[pure error]
91
SS[lack-of-fit] = SS[T rt] − SS[R]poly
= SS[E]poly − SS[E]f ull
= SS[E]poly − SS[pure error]
M S[pure error] = M S[E]f ull
34
101.867/3
≈
= 16.5.
30.93/15
2.1
(highly significant since F (0.01, 3, 15) = 5.42.)
=⇒ model misspecified: SLR model suffers from lack of fit.
Next step: either go with the one-factor ANOVA model or specify
some other model, such as quadratic.
F =
and the sum of squares for pure error is SS[E]f ull = 30.93 yielding
SS[lack of fit] = 243.163 − 141.295 = 101.867 on t − 1 − p = 3df
In a simple linear (p = 1) model for the meadows data,
and
where
F =
When the t treatments have an interval scale, the SLR model, and
all polynomials of degree p ≤ t − 2, are nested in one-factor ANOVA
model with t treatment means.
F -ratio for lack-of-fit
To test for lack-of-fit of a polynomial (reduced) model of degree p,
use extra sum-of-squares F-ratio on t − 1 − p and N − t df :
Osborne, ST512
92
– nested vs. crossed designs
– three-factor ANOVA
– a × b experiments
– 2 × 2 experiments
• Next packet
∗ multiple comparisons,
∗ expected mean squares
∗ power computations
– Specific topics:
– Introduction, notation, jargon:
Terms: factors, levels, treatments, treatment combinations,
main effects, interaction effects, crossed factors, nested factors, contrasts, orthogonal contrasts, expected mean squares,
multiplicity of comparisons, familywise or experimentwise error rates, power.
• This packet
ST 512
Topic: Completely randomized factorial designs
Reading Rao, Ch. 9,13
Osborne, ST512
c1 + c2 + · · · + ct =
1
t
X
cj = 0,
93
θ̂ = c1Ȳ1+ + c2Ȳ2+ + · · · + ctȲt+.
Result: The best estimator for a contrast of interest can be obtained
by substituting treatment group sample means ȳi+ for treatment
population means µi in the contrast θ:
Definition: Contrasts in with more than two nonzero coefficients are
called complex contrasts.
Definition: Contrasts in which only two of the coefficients are nonzero
are called simple or pairwise contrasts.
the linear combo is called a contrast .
If
Definition: The cis are the coefficients of the linear combination.
is called a linear combination of the treatment means.
θ = c 1 µ1 + c 2 µ2 + · · · + c t µt
with Eij ∼ N (0, σ 2),
a linear function of the group means of the form
iid
Yij = µi + Eij , i = 1, 2, . . . , t, and j = 1, 2, . . . , ni
Comparisons (contrasts) among means
Definition: In the one-way ANOVA layout:
Osborne, ST512
94
29.6
27.3
5.8
21.6
29.2
Binding
Sample Sample
Percentage
mean variance
24.3 28.5 32
28.6
10.4
32.6 30.8 34.8 31.4
10.1
6.2 11 8.3
7.8
5.7
17.4 18.3 19
19.1
3.3
32.8 25 24.2 27.8
15.9
Sum of Mean
d.f. squares Square F
4
1481
370 41
15
136
9.05
19 1617
Substitution of ȳ1+ and ȳ2+ yields θ̂ = 28.6 − 31.4 = −2.8.
Q: How good is this estimate?
Source
Treatments
Error
Total
Antibiotic
Penicillin G
Tetracyclin
Streptomycin
Erythromycin
Chloramphenicol
Recall the binding fraction data and ANOVA table:
θ̂ = µ̂1 − µ̂2 = Ȳ1+ − Ȳ2+
Using the result, point estimator of θ is
θ = µ1 − µ2 = (1)µ1 + (−1)µ2 + (0)µ3 + (0)µ4 + (0)µ5
Example For the binding fraction data, consider the pairwise contrast comparing penicillin (population) mean to Tetracyclin mean:
Osborne, ST512
Sampling distribution of θ̂
estimated by
95
est − null θ̂ − θ0 H0
=
∼ tN −t.
ˆ
ˆ θ̂)
SE
SE(
P
q
P c2
ci ȳi+ ± t(N − t, α/2) M S[E] nii
At level α, the critical value for this test is t(N − t, α/2).
P
100(1 − α)% confidence interval for a contrast θ = ciµi given by
t=
To test H0 : θ = θ0 (often 0) versus H1 : θ 6= θ0 a t use t-test:
v
u
i=t 2
u
X
ci
ˆ θ̂) = tM S[E]
SE(
n
i=1 i
Normality follows because θ̂ is a linear function of normal data Yij .
Standard error:
v
s
u i=t
X c2
c21 2 c22 2
c2t 2 u
i
SE(θ̂) =
σ + σ + · · · + σ = tσ 2
,
n1
n2
nt
n
i=1 i
θ̂ ∼ N (θ, Var(θ̂))
·
Q: What is the sampling distribution of θ̂?
Q’: That is, what are E(θ̂), SE(θ̂) and shape of distribution of θ̂?
Osborne, ST512
c θ̂) =
SE(
s
r
12 (−1)2
9.05
(9.05) =
= 2.127
+
n1
n2
2
96
′
,
Here µ = (µ1, µ2, µ3, µ4, µ5) .
θ5 = (
,
,
,
)µ
along with the complex contrast comparing Pen G. with mean of
other four antibiotics:
• θ4 = d′µ =?
• θ3 = c′µ =?
• θ2 = b′µ =?
• θ1 = a′µ = (1, −1, 0, 0, 0)µ
Code (next page) estimates all pairwise contrasts involving Pen. G:
−2.8 ± 2.13(2.127) or (−7.3, 1.7)
A 95% confidence interval is given by
So that the t statistic becomes
−2.8
= −1.32
2.127
which is not in the critical region, so that the sample mean binding
fractions for Penicillin G and Tetracyclin do not differ significantly.
Here,
Osborne, ST512
theta1
theta2
theta3
theta4
theta5
Parameter
Source
drug
Class
drug
-1;
0 -1;
0 0 -1;
0 0 0 -1;
-1 -1 -1 -1/divisor=4;
1.7602402
25.3102402
14.0602402
5.3352402
10.6666722
-7.3102402
16.2397598
4.9897598
-3.7352402
3.4958278
theta1
theta2
theta3
theta4
theta5
-1.30
9.76
4.48
0.38
4.21
t Value
F Value
40.88
95% Confidence Limits
-2.7750000
20.7750000
9.5250000
0.8000000
7.0812500
Standard
Error
Mean Square
370.205750
2.12777270
2.12777270
2.12777270
2.12777270
1.68215202
Type III SS
1480.823000
Estimate
DF
4
F Value
40.88
y Mean
22.93500
Mean Square
370.205750
9.054833
Values
1 2 3 4 5
Root MSE
3.009125
Sum of
Squares
1480.823000
135.822500
1616.645500
Coeff Var
13.12023
DF
4
15
19
Levels
5
Parameter
R-Square
0.915985
Source
Model
Error
Corrected Total
1
1
1
1
4
The GLM Procedure
Class Level Information
proc glm data=one;
class drug;
model y=drug/clparm;
estimate "theta1" drug
estimate "theta2" drug
estimate "theta3" drug
estimate "theta4" drug
estimate "theta5" drug
run;
Osborne, ST512
0.2118
<.0001
0.0004
0.7122
0.0008
Pr > |t|
Pr > F
<.0001
Pr > F
<.0001
97
Orthogonal contrasts
98
θ1 =
i=1
t
X
c i µi
and
and
θ2 =
i=1
t
X
diµi
θ2 = d1µ1 + · · · + dtµt
i=1
cidi = 0
(−1, 1, 0, 0, 0) and (0, 0, −1, 1, 0) orthogonal ?
θi and θj orthogonal =⇒ θ̂i and θ̂j are statistically independent.
(−1, 1, 0, 0, 0) and (0, −1, 1, 0, 0) orthogonal ?
(1, −1/2, −1/2, 0, 0) and (0, 0, 0, −1, 1) orthogonal ?
Examples:
Consider several contrasts, say k of them: θ1, . . . , θk . The set is
mutually orthogonal if all pairs are mutually orthogonal.
c1d1 + · · · ctdt =
t
X
Definition: The two contrasts θ1 and θ2 are mutually orthogonal if
the products of their coefficients sum to zero:
or
θ 1 = c 1 µ1 + · · · + c t µt
Let two contrasts θ1 and θ2 be given by
In the same way the SS[T OT ] can be partitioned into independent
components SS[T rt] and SS[E], the sum of squares for treatments,
SS[T rt] can be partitioned into t − 1 independent components.
Osborne, ST512
99
c21
n1
c2
+ · · · + ntt
θ̂12
SS[θ̂j ]
M S[E]
M S[E]
1
4
= 1.73.
2
+ (−1)
+
0
+
0
+
0
4
(−2.8)2
(Using F (0.05, 1, 15) = 4.54, is the value θ1 = 0 plausible?)
F =
on 1 numerator degree of freedom and N − t denominator degrees
of freedom. For θ1 = µ1 − µ2 in the binding fractions,
F =
To test H0 : θj = 0 versus H1 : θj 6= 0 use
E(SS[θ̂j ]) = σ 2.
There is one df associated with a sum of squares for an individual
contrast θ̂j and if θj = 0, then it can be shown that
SS[T rt] = SS(θ̂1) + SS(θ̂2) + · · · + SS(θ̂t−1)
If θ1, . . . , θt−1 are t − 1 mutually orthogonal contrasts, then
SS[θ̂1] == Sums of squares for contrasts
In the same way SS[T rt] was obtained for a treatment effect, a sum
of squares term can be obtained for a contrast:
Osborne, ST512
100
and
µM + µA
θ 2 = µC −
2
Q: Are these contrasts orthogonal?
Q: Are the estimated contrasts θ̂1 and θ̂2 independent?
Exercise: Compute SS[θ̂1] and SS[θ̂2]. Add em up.
θ 1 = µM − µA
Consider the following 2 contrasts:
Sum of Mean
Source
d.f. squares Square F
Treatments (or pharmacies) 2 113646 56823 5.81
Error
15 146753 9784
Total
17 260400
ȳi+
Cutter Abbott McGaw
255
105
577
264
288
515
342
98
214
331
275
413
234
221
401
217
240
260
273.8 204.5
396.7
Number of contaminants in IV fluids made by t = 3 pharmaceutical
companies
A new dataset:
Osborne, ST512
Parameter
C - avg of M and A
McGaw - Abbott
Contrast
C - avg of M and A
McGaw - Abbott
R-Square
0.436430
Source
Model
Error
Corrected Total
Class
firm
F Value
5.81
t Value
-0.54
3.37
F Value
0.29
11.32
con Mean
291.6667
Mean Square
2862.2500
110784.0833
Standard
Error
49.4559849
57.1068524
Contrast SS
2862.2500
110784.0833
Estimate
-26.750000
192.166667
DF
1
1
Root MSE
98.91197
Mean Square
56823.1667
9783.5778
Values
Abbott Cutter McGaw
Sum of
Squares
113646.3333
146753.6667
260400.0000
Coeff Var
33.91268
DF
2
15
17
Levels
3
proc glm order=formatted;
title "contaminant particles in IV fluids";
class firm;
model con=firm;
contrast ’C - avg of M and A’ firm -0.5 1 -0.5;
contrast ’McGaw - Abbott’ firm -1 0 1;
estimate ’C - avg of M and A’ firm -0.5 1 -0.5;
estimate ’McGaw - Abbott’ firm -1 0 1;
run;
contaminant particles in IV fluids
The GLM Procedure
data one;
infile "pharmfirm.dat";
input firm con;
format firm firmfmt.;
run;
proc format;
value firmfmt 1="Cutter" 2="Abbott" 3="McGaw";
run;
Osborne, ST512
Pr > |t|
0.5965
0.0043
Pr > F
0.5965
0.0043
Pr > F
0.0136
1
101
Multiple Comparisons
102
• Tukey
• Bonferroni
• Scheffé
Methods for simultaneous inference for multiple contrasts include
f we = Pr(at least one type I error)
Definition: When testing k contrasts, the experimentwise error rate
(or familywise) is
• probability of committing at least one type I error ?
• e.g. consider the case with t = 5 (antibiotic treatments): all
simple (pairwise) contrasts of the form θ = µi − µj
5
= 10 tests of significance each at level α = 0.05
•
2
• Can’t go carrying out many many tests of significance willy-nilly
Osborne, ST512
103
(Data courtesy of Cassi Myburg)
A context in which multiplicity is a big issue:
Microarray experiments, which may involve thousands of genes and tests
Osborne, ST512
Bonferroni
104
nj
X kj2
might have to be obtained
M S[E]
X b2j
nj
We have k = 4, α′ = 0.05/k = 0.0125, and t( 2 , 15) = 2.84.
α′
θ1 = µ1 − µ2, θ2 = µ1 − µ3, θ3 = µ1 − µ4, θ4 = µ1 − µ5
For the binding fraction example, consider only pairwise comparisons
with Penicillin:
′
t( α2 , ν)
s
M S[E]
s
α′
k1Ȳ1+ + k2Ȳ2+ + · · · + ktȲt+ ± t( , ν)
2
α′
b1Ȳ1+ + b2Ȳ2+ + · · · + btȲt+ ± t( , ν)
2
...
where ν denotes df for error.
using software.
and
Simultaneous 95% confidence intervals for the k contrasts given by
s
X a2j
α′
a1Ȳ1+ + a2Ȳ2+ + · · · + atȲt+ ± t( , ν) M S[E]
2
nj
Suppose interest lies in exactly k contrasts. The Bonferroni adjustment to α which controls f we is
α
α′ =
k
Osborne, ST512
105
Parameter
theta1
theta2
theta3
theta4
Estimate
2.7750000
-20.7750000
-9.5250000
-0.8000000
Standard
Error
2.12777270
2.12777270
2.12777270
2.12777270
Values
1 2 3 4 5
t Value
1.30
-9.76
-4.48
-0.38
98.75% Confidence Limits
-3.2606985
8.8106985
-26.8106985 -14.7393015
-15.5606985
-3.4893015
-6.8356985
5.2356985
Levels
5
(actually simultaneous 95% confidence intervals)
Parameter
theta1
theta2
theta3
theta4
Class
drug
proc glm data=one;
title "Bonferroni correction for 4 contrasts";
class drug;
model y=drug/clparm alpha=.0125;
estimate "theta1" drug -1 1;
estimate "theta2" drug -1 0 1;
estimate "theta3" drug -1 0 0 1;
estimate "theta4" drug -1 0 0 0 1;
run;
Bonferroni correction for 4 contrasts
The GLM Procedure
Class Level Information
Pr > |t|
0.2118
<.0001
0.0004
0.7122
In SAS, an adjustment for k = 4 can be achieved with care:
so that simultaneous 95% confidence intervals for θ1, θ2, θ3, θ4 take
the form
ȳ1 − ȳi ± 6.0
Substitution leads to
s
r
02
(−1)2 (−1)2 02
2
′
+
+ + ···
t(α , 15) M S[E]
= 2.84 (9.05) = 6.0
4
4
4
4
4
Osborne, ST512
c2i
ni
q
(t − 1)(F ∗ )M S[E](1/nj + 1/nk )
1)(F ∗)M S[E]
1
1
+
nj nk
=
1 1
(3 − 1)(3.68)9784( + ) = 154.9
6 6
r
conclusion about pairwise contrasts? (compare w/ Bonferroni)
(t −
and
s
ȳ·A = 204.5, ȳ·M = 396.67 ȳ·C = 273.83
For IV fluids,
If any two sample means differ by more than 7.44, they differ significantly.
For binding fraction data,
s
s
1
1
1 1
∗
= 7.44
+
(t − 1)(F )M S[E]
+
= (5 − 1)(3.06)9.05
ni nj
4 4
• ni, nj (from the data)
• ȳi+, ȳj+
• M S[E] (from the data)
• F (same critical value as for H0 : τi ≡ 0).
∗
• t (from the design)
Using α = 0.05, need to specify
ȳj+ − ȳk+ ±
where, F ∗ = F (α, t−1, N −t). For a pairwise comparisons of means,
µj and µk , this yields
Pt
q
P
(t − 1)(F ∗ )M S[E] t1
1 ci ȳi+ ±
For the IV data, q(3, 15, 0.05) = 3.67. Tukey’s 95% honestly significant difference (HSD) for pairwise comparisons of treatment means
in this balanced design are
r
9784
= 148.3
3.67
6
M S[E]
n
where q(t, N − t, α) denotes α level studentized range for t means
and N −t degrees of freedom. These studentized ranges can be found
in Table 10 of Ott and Longnecker.
r
H0 : θ = 0 vs H1 : θ 6= 0
|θ̂| > q(t, N − t, α)
reject H0 at level α if
to test
θ = µ j − µk
For simple contrasts of the form
Tukey’s method is better than Scheffé’s method when making
all pairwise comparisons in balanced designs (n = n1 = n2 = · · · =
nt). It is conservative, controlling the experimentwise error rate, and
has a lower type II error rate in these cases than Scheffé. (It is more
powerful.)
107
For simultaneous 95% confidence intervals for ALL contrasts, use
Osborne, ST512
Tukey
106
Another method: Scheffé
Osborne, ST512
6
6
6
N
McGaw
Cutter
Abbott
firm
396.67
273.83
204.50
A
A
B
B
Mean
Scheffe Grouping
6
6
6
N
McGaw
Cutter
Abbott
firm
Means with the same letter are not significantly different.
Alpha
0.05
Error Degrees of Freedom
15
Error Mean Square
9783.578
Critical Value of F
3.68232
Minimum Significant Difference
154.98
NOTE: This test controls the Type I experimentwise error rate.
Scheffe’s Test for con
The GLM Procedure
396.67
273.83
204.50
A
A
B
B
Mean
Tukey Grouping
Means with the same letter are not significantly different.
Alpha
0.05
Error Degrees of Freedom
15
Error Mean Square
9783.578
Critical Value of Studentized Range 3.67338
Minimum Significant Difference
148.33
NOTE: This test controls the Type I experimentwise error rate, but it
generally has a higher Type II error rate than REGWQ.
proc glm;
class firm;
model con=firm;
means firm/scheffe tukey;
run;
Tukey’s Studentized Range (HSD) Test for con
Osborne, ST512
108
Expected mean squares
109
ψT2 =
E[M S[T rt]; H0] = E[SS[T rt]/(t − 1); H0]
= σ2
1 X
(µi − µ)2.
t−1
Note that under H0 : µi ≡ µ and ψT2 = 0 so that
where
It can be shown that
E[M S[T rt]; H1] = E[SS[T rt]/(t − 1); H1]
1 X
ni(µi − µ)2
= σ2 +
t−1
1 X
(µi − µ)2( balanced case)
= σ2 + n
t−1
= σ 2 + nψT2
A: Note that the terms in the ΣΣ above do not depend on j, so
it is a sum of squares from t independent sample treatment means,
leaving t − 1 df for assessing variability.
Q: Why are there t−1 degrees of freedom associated with M S[T rt]?
Definition: The treatment mean square is given by
1 XX
SS[T rt]
=
(ȳi+ − ȳ++)2
M S[T rt] =
t−1
t−1 i j
!
ni
1 X
yij
ȳ++ = y ++ ȳi+ =
ni i=1
Osborne, ST512
SS[E]
N −t
110
E[M S[E]] = σ 2
E(Si2) = σ 2
=⇒ n1 − 1
n2 − 1
nt − 1
2
2
E[M S[E]] =
σ +
σ + ··· +
σ2
N −t
N −t
N −t
= σ2
Since E(Si2) = σ 2, M S[E] is unbiased for σ 2 regardless of H0 or H1:
=
1 X
(ni − 1)s2i
N − t i=1
n1 − 1 2
n2 − 1 2
nt − 1 2
s1 +
s2 + · · · +
s
=
N −t
N −t
N −t t
= “Sp2”
t
SS[E]
M S[E] =
N −t
i=t j=n
1 X Xi
(yij − ȳi+)2
=
N − t i=1 j=1
This is just a generalization of the pooled variance Sp2 to the case of
more than t = 2 groups:
M S[E] =
Definition: The error mean square is given by
Osborne, ST512
Sample size computations for one-way ANOVA
111
1 − β = Pr(M S[T ]/M S[E] > F ∗; H1 is true).
(1)
The power of the F -test conducted using α = 0.05 to reject H0 under
this alternative is given by
Recall that the critical region for the statistic F = M S[T ]/M S[E]
is everything bigger than F (α, t − 1, N − t) = F ∗.
Given α, µ1, . . . , µt, and σ 2, we can choose n to ensure a power of
at least β using the noncentral F distribution.
Of course, the answer depends on lots of things, namely, σ 2 and
how many treatment groups t we have and how much of a difference
among the means we hope to be able to detect, and with how big a
probability.
Q: Suppose that we intend to use a balanced design. How big does
our sample size n1 = n2 = . . . = nt = n need to be?
H1 : µi 6= µj for some i 6= j
versus the alternative
H 0 : µ 1 = µ 2 = · · · = µt = µ
Now consider the null hypothesis in a balanced experiment using
one-way ANOVA to compare t treatment means and α = 0.05:
Osborne, ST512
112
113
We’ll write SAS code to do some computations but one could also
use the procedure PROC POWER or other software such as R.
http://www.stat.uiowa.edu/~rlenth/Power/
Q: “What is an acceptable type II error rate, or what kind of power
are we looking for?”
One way to obtain an adequate sample size is trial and error. Software packages can be used to get probabilities of the form (1) for
various values of n. Russ Lenth’s website is also terrific and helpful:
We need the area to the right of F ∗ for the noncentral F distribution
with degrees of freedom 2 and 3(n − 1) and noncentrality parameter
γ = 0.54n. The following printout suggests the sufficiency of n = 19
for power of 1 − β = 0.8
F ∗ = F (3 − 1, 3(n − 1), 0.05).
To obtain probabilities of the form (1) we need the noncentrality
parameter γ :
τ2
τ3
τ1
γ = n[( )2 + ( )2 + ( )2] = n[−302 + −302 + (60)2]/1002 = 0.54n
σ
σ
σ
The α = 0.05 critical value for H0 is given by
A: Suppose that 1 − β = 0.8 should be good enough.
A: The alternative H1 : µ1 = µ2 = (µ−30) = 230, µ3 = µ+60 = 320
would be meaningful. Assume σ ≈ 100.
Q: “What alternative to H0 would be meaningful? What is σ? ”
An example: suppose that a balanced completely randomized design
(CRD) is to be used to test for a difference in the number of contaminant particles in IV fluid for three pharmaceutical companies. It is
believed that the standard deviation on a given observation is about
100 particles for each company. In order to test H0 : µ1 = µ2 = µ3
at level α = 0.05, how large does the common sample size, n, need
to be?
Osborne, ST512
This is the parameterization for the F distribution used in both SAS
and S+.
When some H1 is true and the sample size n is used in each group,
it can be shown that the F ratio has the noncentral F distribution
with noncentrality parameter
t
t τ 2
X
X
τj 2
j
nj
=n
γ=
σ
σ
j=1
j=1
H0 : τ1 = τ2 = · · · = τt = 0
Let τi = µi − µ for each treatment i so that
Osborne, ST512
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
6
9
12
15
18
21
24
27
30
33
36
39
42
45
48
51
54
57
60
63
66
69
72
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
5400
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
10000
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
1800
proc print;run;
OBS N T NU1 NU2 SUMTAU2 SIGMA2 SIGMA2U
data one;
do n=3 to 25; output; end;
run;
data one;
set one;
t=3;
nu1=t-1;
nu2=t*(n-1);
sumtau2=(-30)**2 + (-30)**2 + 60**2;
sigma2=10000;
*sigma2u=(2/3)*var(100,100,190);
*ncp=t*sigma2u/(2*sigma2);
ncp=n*sumtau2/sigma2;
qf=finv(0.95,nu1,nu2);
pf=probf(qf,nu1,nu2,ncp);
power=1-pf;
run;
Osborne, ST512
1.62
2.16
2.70
3.24
3.78
4.32
4.86
5.40
5.94
6.48
7.02
7.56
8.10
8.64
9.18
9.72
10.26
10.80
11.34
11.88
12.42
12.96
13.50
NCP
5.14325
4.25649
3.88529
3.68232
3.55456
3.46680
3.40283
3.35413
3.31583
3.28492
3.25945
3.23810
3.21994
3.20432
3.19073
3.17880
3.16825
3.15884
3.15041
3.14281
3.13592
3.12964
3.12391
QF
0.86663
0.81604
0.76402
0.71152
0.65935
0.60817
0.55853
0.51085
0.46546
0.42257
0.38233
0.34481
0.31002
0.27795
0.24850
0.22160
0.19712
0.17493
0.15489
0.13684
0.12065
0.10616
0.09324
PF
0.13337
0.18396
0.23598
0.28848
0.34065
0.39183
0.44147
0.48915
0.53454
0.57743
0.61767
0.65519
0.68998
0.72205
0.75150
0.77840
0.80288
0.82507
0.84511
0.86316
0.87935
0.89384
0.90676
POWER
114
115
OBS
1
2
3
4
5
6
7
8
9
N
2
3
4
5
6
7
8
9
10
T
5
5
5
5
5
5
5
5
5
NU1
4
4
4
4
4
4
4
4
4
NU2
5
10
15
20
25
30
35
40
45
SUMTAU2
54
54
54
54
54
54
54
54
54
data one;
do n=2 to 10; output; end;
run;
data one;
set one;
t=5; nu1=t-1; nu2=t*(n-1);
sumtau2=3**2+3**2+(-6)**2;
sigma2=9;
ncp=n*sumtau2/sigma2;
qf=finv(0.95,nu1,nu2);
pf=probf(qf,nu1,nu2,ncp);
power=1-pf;
run;
proc print;run;
SIGMA2
9
9
9
9
9
9
9
9
9
NCP
12
18
24
30
36
42
48
54
60
QF
5.19217
3.47805
3.05557
2.86608
2.75871
2.68963
2.64147
2.60597
2.57874
The noncentrality parameter is given by
3
−6
3
γ = n[( )2 + ( )2 + ( )2].
3
3
3
The following code should do the trick
PF
0.59246
0.22465
0.06437
0.01533
0.00319
0.00060
0.00010
0.00002
0.00000
τ1 = 3, τ2 = 3, τ3 = −6, τ4 = τ5 = 0.
Assume σ = 3 and we need to use α = β = 0.05.
so that
POWER
0.40754
0.77535
0.93563
0.98467
0.99681
0.99940
0.99990
0.99998
1.00000
H1 : µP = µ + 3, µT = µ + 3, µS = µ − 6, µE = µ, µC = µ
Another example: suppose we want to test equal mean binding fractions among antibiotics against the alternative
Osborne, ST512
Orthogonal polynomial contrasts
116
R-Square
0.296837
DF
3
68
71
Coeff Var
3.399254
Source
Model
Error
Corrected Total
Root MSE
34.57291
Sum of
Squares
34311.7666
81279.4678
115591.2344
One-way ANOVA table
AMBW21D Mean
1017.073
Mean Square
11437.2555
1195.2863
F Value
9.57
Pr > F
<.0001
(b) The appropriate critical value for a test with level α = 0.05
is F ∗ = 3.13. Draw a conclusion about the adequacy of the
linear model using α = 0.05: is there evidence that the linear
model is inadequate?
(a) Report the F -ratio for testing for a lack-of-fit of the linear
model.
3. Fitting the SLR model leads to SS[Reg] = 32742 and SS[T ot] =
115591.
(f) Report SS(θ̂1).
(e) Estimate the standard error of θ̂2.
(d) Provide an expression for the standard error of θ̂2.
(c) Report θ̂1, θ̂2, θ̂3.
SS[θ̂1] + SS[θ̂2] + SS[θ̂3] = SS[T rt]?
(b) True/false: If contrast sums of squares are obtained, then
• n = 18 pens, N = 72.
θ̂1 = −3ȳ1+ − ȳ2+ + ȳ3+ + 3ȳ4+
θ̂2 =
ȳ1+ − ȳ2+ − ȳ3+ + ȳ4+
θ̂3 = −ȳ1+ + 3ȳ2+ − 3ȳ3+ + ȳ4+
2. Consider the following three sample contrasts
(a) True/false: These estimated contrasts are orthogonal.
diet x: level of diet mean diet std. dev. Tukey
si
grouping
ȳi+
group protein
1
21.8
993
38
2
23.5
1003
28
3
25.2
1022
39
4
26.9
1050
32
117
1. Sketch a plot of mean bodyweight at 21 days against protein
content.
Some omitted exam questions:
Osborne, ST512
• thanks to P. Plumstead for data.
• completely randomized design with one factor, protein in diet,
with four equally spaced levels.
• Y : 21-day bodyweights of chickens
Example: poultry science experiment measures bodyweights of chickens from a = 4 diet groups, characterized by protein concentration
in diet.
Osborne, ST512
118
2
2
1
1
ȳ5+
SS(θ̂i)
R(β1|β0)
R(β2|β0, β1)
R(β1|β0)
R(β2|β0, β1)
R(β3|β0, β1, β2)
?
?
?
?
This is computationally (and otherwise) easier than fitting polynomial regressions of various degrees.
• large |θ̂3| indicates cubic association between y and x.
• large |θ̂2| indicates quadratic association between y and x.
• large |θ̂1| indicates linear association between y and x.
The contrast corresponding to a polynomial of degree p can be used
to test for a pth degree association:
µ(x) = β0 + β1x + β2x + · · · .
2
Rightmost column indicates extra SS in MLR of the form
Coefficients for
Factor Poly.
levels Degree contrast ȳ1+ ȳ2+ ȳ3+ ȳ4+
3
1
θ̂1
-1 0
1
1 -2 1
2
θ̂2
4
1
θ̂1
-3 -1 1
3
2
θ̂2
1 -1 -1 1
-1 3 -3 1
3
θ̂3
5
1
θ̂1
-2 -1 0
1
2
θ̂2
2 -1 -2 -1
-1 2
0 -2
3
θ̂3
4
θ̂4
1 -4 6 -4
The contrasts in problem 2 are called orthogonal polynomial contrasts. The table below gives coefficients for orthogonal polynomial
contrasts for balanced single-factor experiments with 3, 4, or 5 equally
spaced levels.
Osborne, ST512
F Value
27.39
1.31
0.00
9.57
F Value
9.57
t Value
5.23
1.15
0.04
Mean Square
32741.55648
1568.66674
1.54337
11437.25553
Standard
Error
36.4430498
16.2978273
36.4430498
Contrast SS
32741.55648
1568.66674
1.54337
34311.76658
Mean Square
11437.2555
1195.2863
Values
21.8 23.5 25.2 26.9
Sum of
Squares
34311.7666
81279.4678
115591.2344
Estimate
190.734127
18.670635
1.309524
DF
1
1
1
3
Contrast
cp linear
CP quadratic
CP cubic
all three
Parameter
cp linear
CP quadratic
CP cubic
DF
3
68
71
Levels
4
Source
Model
Error
Corrected Total
Class
CP
Pr > |t|
<.0001
0.2560
0.9714
Pr > F
<.0001
0.2560
0.9714
<.0001
1
119
Pr > F
<.0001
Proc glm;
title "protein concentration and chicken weights";
class cp;
MODEL AMBW21D=cp;
contrast ’cp linear’ cp -3 -1 1 3;
contrast ’CP quadratic’ CP 1 -1 -1 1;
contrast ’CP cubic’ CP -1 3 -3 1;
contrast ’all three’ CP -3 -1 1 3,
cp 1 -1 -1 1,
cp -1 3 -3 1;
/* ’all three’ tests that the 3-vector of contrasts is (0,0,0)’ */
estimate ’cp linear’ cp -3 -1 1 3;
estimate ’CP quadratic’ CP 1 -1 -1 1;
estimate ’CP cubic’ CP -1 3 -3 1;
RUN;
proc glm; /* no class statement will fit regression model*/
model ambw21d=cp cp*cp cp*cp*cp;
run;
proc glm;
model ambw21d=cp;
run;
protein concentration and chicken weights
The GLM Procedure
Osborne, ST512
Estimate
743.8748249
11.2196545
DF
1
70
71
t Value
14.28
5.26
Mean Square
32741.5565
1183.5668
Standard
Error
52.10077470
2.13317368
Sum of
Squares
32741.5565
82849.6779
115591.2344
The GLM Procedure
Pr > F
<.0001
0.2560
0.9714
Pr > F
<.0001
6
Pr > F
<.0001
Pr > |t|
<.0001
<.0001
F Value
27.66
Pr > |t|
0.9524
0.9959
0.9857
0.9714
F Value
27.39
1.31
0.00
F Value
9.57
3
120
Note MSE. Linear regression on x preferred to one-factor model for
Plumstead’s data. Multiple comparisons among treatment means
might be unnecessary.
Parameter
Intercept
CP
Source
Model
Error
Corrected Total
t Value
0.06
0.01
-0.02
0.04
Mean Square
32741.55648
1568.66674
1.54337
Mean Square
11437.2555
1195.2863
Standard
Error
17690.23708
2192.80559
90.32122
1.23628
Type I SS
32741.55648
1568.66674
1.54337
Sum of
Squares
34311.7666
81279.4678
115591.2344
protein concentration and chicken weights
Estimate
1060.706771
11.320308
-1.630049
0.044424
DF
1
1
1
Source
CP
CP*CP
CP*CP*CP
Parameter
Intercept
CP
CP*CP
CP*CP*CP
DF
3
68
71
The GLM Procedure
protein concentration and chicken weights
Source
Model
Error
Corrected Total
Osborne, ST512
121
The
The
The
The
Group
I
II
III
IV
I
II
III
IV
221
271
262
192
213
192
193
253
of
of
of
of
avg
std. dev.
162 ȳI = 194.7 s = 20
142 ȳII = 209.4 s = 41
265 ȳIII = 212.0 s = 40
289 ȳIV = 251.3 s = 32
women younger than 50
men younger than 50
women 50 years or older
men 50 years or older
Cholesterol level
202 183 185 197
189 209 227 236
224 201 161 178
248 278 232 267
population
population
population
population
Cholesterol measurements for random samples of nj ≡ 7 people from
four populations are given in the table below. The groups (cohorts)
are defined as follows:
An example of a 2 × 2 study
• nested vs. crossed designs (not described in packet)
• three-factor ANOVA
• a × b experiments
• 2 × 2 experiments
Topic: Multi-factor ANOVA
Reading: Rao, Ch. 13
Osborne, ST512
DF
3
Type I SS
12280.85714
3.46
F Value
Mean Square
4093.61905
F Value
3.46
y Mean
216.8571
4093.61905
1184.77381
Mean Square
Root MSE
34.42054
12280.85714
28434.57143
40715.42857
Sum of
Squares
28
Values
I II III IV
Pr > F
0.0323
0.0323
Pr > F
Conclusion so far is the cohort means, µi or µ + τi, are not plausibly
equal (using α = 0.05).
Source
cohort
3
24
27
Model
Error
Corrected Total
Coeff Var
15.87245
DF
Source
R-Square
0.301627
Levels
4
The GLM Procedure
Number of observations
Class
cohort
One-way ANOVA table:
Definition: If there are observations at all combinations of all factors,
the design is complete , otherwise it is incomplete .
In one-way analysis of cholesterol data, COHORT is the only factor.
This factor can be broken down into two factors in a two-way analysis:
AGE (factor A) and GENDER (factor B).
Definition: When the same number of units are used for each treatment, the design is balanced .
Note: The cholesterol study is NOT a completely randomized design,
as randomization of subjects to different levels of AGE and GENDER
isn’t possible.
Definition: In completely randomized designs experimental units
are randomly assigned to factor levels, or treatment groups.
Definition: A factor in an experiment or study is a variable whose
effect on the response is of primary interest. The values that a factor
takes in the experiment are called factor levels or treatments .
123
Yij = µi + Eij
= µ + τi + Eij
Osborne, ST512
Some terminology
122
One-way ANOVA Model:
i = 1, 2, 3, 4 j = 1, 2, . . . , 7 and Eij i.i.d. N (0, σ 2)
P
Parameters: µ, τ1, τ2, τ3, τ4, σ 2, with 41 τi = 0 constraint.
Osborne, ST512
Exercise
124
7. Specify the vectors defining these contrasts. For example, the
first contrast of cohort means can be written


µ1


µ 
θ1 = (−1, 1, 0, 0)′  2  = µ2 − µ1
 µ3 
µ4
6. Provide standard errors for all of these estimated contrasts
5. Estimate the mean difference between the differences estimated
in 1. and 2.
4. Estimate the mean difference between older and younger folks.
3. Estimate the mean difference between men and women.
2. Estimate the mean difference between old men and old women.
1. Estimate the mean difference in cholesterol between young men
and young women.
Osborne, ST512
125
(c) an age × gender interaction
(b) a gender effect
(a) an age effect
3. Obtain the α = 0.05 critical region for each test. Compare the
observed F -ratios to critical value and draw conclusions about
2. Formulate a test of H0 : θi = 0 for each of these three contrasts.
Obtain the F -ratio for each of these tests.
ni
1. Compute the sums of squares for the estimated contrasts in 3.,
4. and 5. using the exercise just completed and the fact that if
P
θ̂ = ciȳi+ then
θ̂2
SS[θ̂] = P 2 .
ci
Another exercise:
Q: True/False: SS(θ̂3) + SS(θ̂4) + SS(θ̂5) = SS[T rt]
Q: Are these contrasts orthogonal?
θ3 = (−1, 1, −1, 1)′µ
θ4 = (−1, −1, 1, 1)′µ
θ5 = (−1, 1, 1, −1)′µ
Consider the following contrasts of the cohort cholesterol means in
the population:
Osborne, ST512
126
3. Main effects are averages or sums of simple effects
1
µ(A) = (µ(AB1) + µ(AB2))
2
1
µ(B) = (µ(A1B) + µ(A2B))
2
Exercise: Classify the contrasts in the last exercise as simple, interaction or main effects.
- difference between simple age effects for men and women
- difference between simple gender effects for old and young folks
- interaction effect of AGE and GENDER.
µ(AB) = µ(AB2) − µ(AB1) = (µIV − µII − (µIII − µI ))
2. Interaction effects are differences of simple effects:
• µ(A1B) = µII − µI - simple effect of gender for young folks.
• µ(AB1) = µIII − µI - simple effect of age for women
1. Simple effects are simple contrasts.
Three kinds of effects in 2 × 2 designs:
Factor A: AGE has a = 2 levels - A1 : younger and A2 : older
Factor B: GENDER has b = 2 levels - B1 : female and B2 : male
Types of effects
Two-way ANOVA model for the cholesterol measurements:
Yijk = µ + αi + βj + (αβ)ij + Eijk
i = 1, 2 = a and j = 1, 2 = b and k = 1, 2, . . . , 7 = n.
P
P
iid
Eijk ∼ N (0, σ 2). Parameter constraints:
i αi =
j βj = 0 and
P
P
(αβ)
≡
0
for
each
j
and
(αβ)
≡
0
for
each
i.
ij
ij
i
j
Osborne, ST512
127
1
7
2
=
4
7
(24.6)2
F = 1056/1185 = 0.9
(−1)
1
+ (−1)
7 + 7 + 7
2
(24.6)2
which isn’t significant at α = 0.05 on 1,24 df .
and
SS(θ̂5) =
the associated sum of squares is
= 1056
θ̂5 = µ̂(AB) = (251.3 − 209.4) − (212 − 194.7) = 41.9 − 17.3 = 24.6
on 1 and 28 − 4 = 24 numerator, denominator df . For cholesterol
datathe estimated interaction effect is
F =
SS(θ̂)/((a − 1)(b − 1))
M S[E]
H1 : (αβ)ij 6= 0 for some i, j
use θ5 = µ(AB) and
vs.
H0 : (αβ)11 = (αβ)12 = (αβ)21 = (αβ)22 = 0
F test for interaction effect
To test for interaction,
• (b − 1) df for main effect of B
• (a − 1) df for main effect of A
• (a − 1)(b − 1) df for AB interaction
12281 = SS[T rt] = SS[θ̂3] + SS[θ̂4] + SS[θ̂5] = 5103 + 6121 + 1056
Partitioning the treatment SS into t − 1 orthogonal components
Osborne, ST512
128
use θ3 = µ(B) on 1 and 24 df
209.4 − 194.7 251.3 − 212
+
= 27
µ̂(B) =
2
2
(27)2
(27)2
=
= 5103
SS(θ̂3) = 1 2
1
(2)
( 12 )2
( 12 )2
( 12 )2
7
+
+
+
7
7
7
7
and
F = 5103/1185 = 4.3
since F (0.05, 1, 24) = 4.26 GENDER effect significant at α = 0.05.
H0 : β1 = β2 = 0 vs. H1 : β1 6= 0 or β2 6= 0
Similarly for the main effect of B: gender
F =
SS(θ̂4)
M S[E]
on 1, 24 df . The estimated main effect of AGE is
(251.3 − 209.4) (212 − 194.7) 59.2
+
=
= 29.6
µ̂(A) =
2
2
2
the associated sum of squares is
(29.6)2
(29.6)2
SS(θ̂4) = 1 2
=
= 6121
1
1
1
1
(2)
( 2 )2
( 2 )2
( 2 )2
7
7 + 7 + 7 + 7
and
F = 6121/1185 = 5.2
since F (0.05, 1, 24) = 4.26 AGE effect significant at α = 0.05.
use θ4 = µ(A) and
H0 : α1 = α2 = 0 vs. H1 : α1 6= 0 or α2 6= 00
F test for main effects
To test for main effect of A: AGE
Osborne, ST512
Confidence intervals for effects
129
ciȳi+) =
s
M S[E]
i
ni
X c2
is the estimated standard error of the estiX
√
c
SE(
The term under the
mated contrast:
For the cholesterol data, with t(0.025, 24) = 2.06 we have a 95%
confidence interval for the AGE × GENDER interaction effect:
r
4
1185 or 24.6 ± 2.06(26.0) or (−29, 78)
24.6 ± 2.06
7
a 95% confidence interval for the AGE effect:
r
1
1185 or 29.6 ± 2.06(13.0) or (2.7, 56.4)
29.6 ± 2.06
7
and a 95% confidence interval for the GENDER effect:
r
1
27.0 ± 2.06
1185 or 27.0 ± 2.06(13.0) or (0.15, 53.9).
7
If θ = c′µ, 100(1 − α)% confidence interval given by
q
P c2
θ̂ ± t(α/2, N − t) M S[E] nii
Osborne, ST512
SAS code for cholesterol problem
(SAS will overlook misspelling of contrast.
proc glm;
class gender age;
model y=age|gender;
run;
data one;
input cohort $ @;
do subj=1 to 7;
input y @;
if cohort="I" then do; gender="W"; age="y"; end;
else if cohort="II" then do; gender="M"; age="y";end;
else if cohort="III" then do; gender="W" ; age="o";end;
else if cohort="IV" then do; gender="M" ; age="o";end;
output;
end;
cards;
I
221
213
202
183
185
197
162
II
271
192
189
209
227
236
142
III
262
193
224
201
161
178
265
IV
192
253
248
278
232
267
289
;
run;
proc glm;
class cohort;
model y=cohort/clparm;
constrast "main effect of age
" cohort -1 -1 1 1;
constrast "main effect of gender" cohort -1 1 -1 1;
constrast "interaction effect
" cohort -1 1 1 -1;
estimate "main effect of age
" cohort -1 -1 1 1/divisor=2;
estimate "main effect of gender" cohort -1 1 -1 1/divisor=2;
estimate "interaction effect
" cohort -1 1 1 -1;
run;
Osborne, ST512
130
95% Confidence Limits
2.7206396
56.4222175
0.1492111
53.8507889
-78.2730065
29.1301493
1
131
Pr > F
0.0323
0.0488
0.3544
Pr > F
0.0323
Pr > |t|
0.0323
0.0488
0.3544
F Value
5.17
4.31
0.89
t Value
2.27
2.08
-0.94
Mean Square
6121.285714
5103.000000
1056.571429
Standard
Error
13.0097426
13.0097426
26.0194851
Contrast SS
6121.285714
5103.000000
1056.571429
Estimate
29.5714286
27.0000000
-24.5714286
DF
1
1
1
F Value
3.46
y Mean
216.8571
Mean Square
4093.61905
1184.77381
Root MSE
34.42054
Sum of
Squares
12280.85714
28434.57143
40715.42857
Coeff Var
15.87245
DF
3
24
27
Parameter
main effect of age
main effect of gender
interaction effect
Parameter
main effect of age
main effect of gender
interaction effect
Contrast
main effect of age
main effect of gender
interaction effect
R-Square
0.301627
Source
Model
Error
Corrected Total
Class Level Information
Class
Levels
Values
cohort
4
I II III IV
The SAS System
The GLM Procedure
SAS output (abbreviated) for cholesterol problem
Osborne, ST512
DF
1
1
1
Levels
2
2
Exercise:
Type I SS
6121.285714
5103.000000
1056.571429
F Value
3.46
Mean Square
6121.285714
5103.000000
1056.571429
F Value
5.17
4.31
0.89
y Mean
216.8571
Mean Square
4093.61905
1184.77381
Values
M W
o y
Root MSE
34.42054
Sum of
Squares
12280.85714
28434.57143
40715.42857
Coeff Var
15.87245
DF
3
24
27
Class
gender
age
The GLM Procedure
Pr > F
0.0323
0.0488
0.3544
Pr > F
0.0323
132
GENDER
female (j = 1) male (j = 2)
AGE younger (i = 1)
194.7
209.4
older (i = 2)
212.0
251.3
2. Estimate these effects
(a) µ(AB1)
(b) µ(AB2)
(c) µ(A1B)
(d) µ(A)
(e) µ(B)
(f) µ(AB)
1. Express the effects below in terms of model parameters αi, βj , (αβ)ij :
Source
age
gender
gender*age
R-Square
0.301627
Source
Model
Error
Corrected Total
Osborne, ST512
133
Source
TEMP
SUC
TEMP*SUC
R-Square
0.974809
Source
Model
Error
Corrected Total
DF
8
18
26
DF
2
2
4
Levels
3
3
Type I SS
293.1585185
309.9585185
27.1303704
F Value
87.07
Mean Square
146.5792593
154.9792593
6.7825926
F Value
162.00
171.28
7.50
y Mean
10.81481
Mean Square
78.7809259
0.9048148
Values
20 30 40
20 40 60
Root MSE
0.951218
Sum of
Squares
630.2474074
16.2866667
646.5340741
Coeff Var
8.795505
Class
TEMP
SUC
The SAS System
The GLM Procedure
Pr > F
<.0001
<.0001
0.0010
Pr > F
<.0001
a × b designs
An example: Entomologist records energy expended (y) by N = 27
honeybees at a = 3 temperature (A) levels (20, 30, 40oC) consuming
liquids with b = 3 levels of sucrose concentration (B) (20%, 40%, 60%)
in a balanced, completely randomized crossed 3 × 3 design.
Temp Suc
Sample
20
20 3.1 3.7 4.7
20
40 5.5 6.7 7.3
20
60 7.9 9.2 9.3
30
20
6
6.9 7.5
30
40 11.5 12.9 13.4
30
60 17.5 15.8 14.7
40
20 7.7 8.3 9.5
40
40 15.7 14.3 15.9
40
60 19.1 18.0 19.9
Osborne, ST512
3.0000000
4.0000000
5.0000000
6.0000000
7.0000000
8.0000000
9.0000000
10.0000000
11.0000000
12.0000000
13.0000000
14.0000000
15.0000000
16.0000000
17.0000000
18.0000000
19.0000000
mean energy expended
20
suc
20
40
temp
30
60
40
3 × 3 honeybee example continued
134
20
40
60
20
40
60
20
40
60
20
20
20
30
30
30
40
40
40
3
3
3
3
3
3
3
3
3
n
3.8333333
6.5000000
8.8000000
6.8000000
12.6000000
16.0000000
8.5000000
15.3000000
19.0000000
Exercise: Obtain an interaction plot for the cholesterol data.
0.80829038
0.91651514
0.78102497
0.75498344
0.98488578
1.41067360
0.91651514
0.87177979
0.95393920
--------------Y-------------Mean
SD
The plot above is called an interaction plot .
Level of
SUC
Level of
TEMP
Unlike 2×2 study, not possible to express interaction between factors
A: TEMP and B: SUCROSE using a single number (w/ 1 df ).
Osborne, ST512
Partitioning SS[T ot] in a × b design
Two-way ANOVA Model:
135
i
j
i
i
(ȳ+j+ − ȳ+++)2
i
j
k
XXX
k
k
k
(yijk − ȳij+)2
(ȳij+ − ȳi++ − ȳ+j+ + ȳ+++)2
j
j
(ȳi++ − ȳ+++)2
k
(yijk − ȳ+++)2
Square both sides, ×-products vanish. (See output, p. 133, note
that SS terms add up to model SS. Also, Type I and III SS equal.)
yijk = ȳ+++ + (ȳi++ − ȳ+++ ) + (ȳ+j+ − ȳ+++ ) + (ȳij+ − ȳi++ − ȳ+j+ + ȳ+++ ) + yijk − ȳij+
yijk − ȳ+++ = (ȳi++ − ȳ+++ ) + (ȳ+j+ − ȳ+++ ) + (ȳij+ − ȳi++ − ȳ+j+ + ȳ+++ ) + yijk − ȳij+
SS[E] =
SS[AB] =
j
XXX
XXX
SS[B] =
SS[A] =
i
XXX
XXX
SS[T ot] =
ȳij+−ȳ+++−(ȳi++−ȳ+++)−(ȳ+j+−ȳ+++) = ȳij+−ȳi++−ȳ+j++ȳ+++
Deviations:
total : yijk − ȳ+++
due to level i of factor A: ȳi++ − ȳ+++
due to level j of factor B: ȳ+j+ − ȳ+++
due to levels i of factor A and j of factor B after subtracting main
effects:
Yijk = µ + αi + βj + (αβ)ij + Eijk
(i = 1, 2, . . . , a and j = 1, 2, . . . , b and k = 1, 2, . . . , n)
Osborne, ST512
a × b example continued
i=1 j=1
3 X
3
X
(ȳij+ − ȳi++ − ȳ+j+ + ȳ+++)2 = 27.1
Another a × b design - no interaction
SS[B]/(b − 1)
on b − 1, N − ab df
M S[E]
FB =
Source
a
b
a*b
R-Square
0.917368
DF
2
3
6
Sum of
Squares
24.22
F Value
Mean Square
163.7986111
28.8955556
1.3386111
F Value
103.34
18.23
0.84
y Mean
13.88889
38.3923232
1.5850000
Root MSE
1.258968
Type I SS
327.5972222
86.6866667
8.0316667
10.5
10.1
17.5
13.3
13.7
19.2
14.0
15.4
21.0
12.5
14.5
18.9
Mean Square
36
Values
1 2 3
10 20 30 40
422.3155556
38.0400000
460.3555556
Coeff Var
9.064568
11
24
35
Model
Error
Corrected Total
SS[A]/(a − 1)
on a − 1, N − ab df
M S[E]
Levels
3
4
Number of observations
Class
a
b
Sample
7.9 9.2
8.1 8.6
15.3 16.1
11.2 12.8
11.5 12.7
16.6 18.5
12.1 12.6
13.7 14.4
18.0 20.8
9.1 10.8
11.3 12.5
17.2 18.4
The SAS System
The GLM Procedure
If one insists on main effects, the appropriate F -ratios are
ANOVA table
Variety Density k/hectare
1
10
2
10
3
10
1
20
2
20
3
20
1
30
2
30
3
30
1
40
2
40
3
40
DF
FA =
137
Pr > F
<.0001
<.0001
0.5484
<.0001
Pr > F
1
Yields on 36 tomato crops from balanced, complete, crossed design
with a = 3 varieties (A) at b = 4 planting densities (B) :
Osborne, ST512
Source
136
A: Because effect of one factor depends on the level of the other
factor, it doesn’t make sense to talk about main effects.
Q: Why not?
We could proceed to test for main effects, but we won’t.
F =
27.1/4
= 7.5
0.9
which is highly significant (p = 0.001) on 4,18 df .
SS[AB] = n
For honeybee data,
M S[AB]
F =
M S[E]
on (a − 1)(b − 1) and N − ab numerator, denominator df .
H0 : (αβ)ij ≡ 0 vs. H1 : (αβ)ij 6= 0 for some i, j
Test for interaction effect generalizes from p.127:
Osborne, ST512
138
N
12
12
12
N
9
9
9
9
Level of
a
1
2
3
Level of
b
10
20
30
40
--------------y-------------Mean
Std Dev
11.4777778
3.75458978
14.3888889
2.96835158
15.7777778
3.36480972
13.9111111
3.53250777
--------------y-------------Mean
Std Dev
11.3333333
1.88309867
12.2083333
2.34887142
18.1250000
1.73369023
Since there is no evidence of interaction, we proceed to analyze main
effects. The F -ratios for factors A and B are each highly significant
(p < 0.0001).
3. If interaction, analyze simple effects
2. If no interaction, analyze main effects
1. Check for interaction
Analysis of replicated two (or more) factor designs often proceed
according to the following directions:
Osborne, ST512
139
0.05
24
1.585
3.90126
1.6372
C
9
9
13.9111
11.4778
9
9
N
14.3889
15.7778
A
A
A
B
B
B
Mean
Tukey Grouping
10
40
20
30
b
Means with the same letter are not significantly different.
Alpha
Error Degrees of Freedom
Error Mean Square
Critical Value of Studentized Range
Minimum Significant Difference
NOTE: This test controls the Type I experimentwise error rate, but it
generally has a higher Type II error rate than REGWQ.
Tukey’s Studentized Range (HSD) Test for y
The GLM Procedure
A conventional look at main effects is just to make pairwise comparisons among marginal means, after averaging over other factors.
Pairwise comparisons of density means using Tukey’s procedure with
α = 0.05 are given below.
(Use means b/tukey;. to obtain the output.)
Osborne, ST512
Temperature at 25o C
Temperature at 35o C
Density of shrimp population at 80 shrimp/40l
Density of shrimp population at 160 shrimp/40l
Salinity at 10 units
Salinity at 25 units
Salinity at 40 units
iid
=
=
=
=
1, 2
1, 2
1, 2, 3
1, 2, 3
j
k
where ni++ denotes the number of observations at the ith level of
factor A.
i
Many constraints such as:
X
X
X
ni++αi =
n+j+βj =
n++k γk = 0
Eijkl ∼ N (0, σ 2)
i
j
k
l
Yijkl =
µ + αi + βj + γk
+ (αβ)ij + (αγ)ik + (βγ)jk
+
(αβγ)ijk + Eijkl
The response variable of interest is weight gain Yijkl after four weeks.
Three-way ANOVA Model:
A1:
A2:
B1:
B2:
C1:
C2:
C3:
Mean
10
0.000000
100.000000
200.000000
300.000000
400.000000
a_b
1
1
1
2
2
2
2
a
b
a*b
c
a*c
b*c
a*b*c
500.000000
DF
25_160
25_80
Level of c
25
35_160
15376.0000
21218.7778
8711.1111
96762.5000
300855.1667
674.3889
24038.3889
15376.0000
21218.7778
8711.1111
48381.2500
150427.5833
337.1944
12019.1944
Mean Square
35_80
F Value
14.64
40
5.30
7.31
3.00
16.66
51.80
0.12
4.14
F Value
y Mean
279.1667
Mean Square
42512.3939
2903.7778
Root MSE
53.88671
Type I SS
Coeff Var
19.30270
Source
R-Square
0.870301
DF
11
24
35
Sum of
Squares
467636.3333
69690.6667
537327.0000
Source
Model
Error
Corrected Total
In a balanced, complete, crossed design, N = 36 shrimp were randomized to abc = 12 treatment combinations from the factors below:
Osborne, ST512
The GLM Procedure
140
A three-factor example
Osborne, ST512
0.0304
0.0124
0.0961
<.0001
<.0001
0.8909
0.0285
Pr > F
Pr > F
<.0001
141
N
6
6
6
6
6
6
N
6
6
6
6
6
6
N
9
9
9
9
Level of
c
10
25
40
10
25
40
10
25
40
10
25
40
Level of
c
10
25
40
10
25
40
Level of
b
80
80
80
160
160
160
Level of
b
80
80
80
160
160
160
80
80
80
160
160
160
Level of
c
10
25
40
10
25
40
Level of
a
25
25
25
35
35
35
Level of
a
25
25
25
25
25
25
35
35
35
35
35
35
Level of
b
80
160
80
160
Level of
a
25
25
35
35
Osborne, ST512
N
3
3
3
3
3
3
3
3
3
3
3
3
--------------y-------------Mean
Std Dev
70.333333
17.156146
465.666667
87.648921
359.000000
59.858166
70.666667
16.623277
333.000000
108.282039
252.333333
11.372481
408.000000
51.117512
274.666667
47.961790
243.000000
36.166283
331.000000
30.116441
311.666667
42.665365
230.666667
46.971623
--------------y-------------Mean
Std Dev
239.166667
188.065326
370.166667
122.218520
301.000000
77.415761
200.833333
144.240655
322.333333
74.529636
241.500000
32.788718
--------------y-------------Mean
Std Dev
70.500000
15.109600
399.333333
114.206246
305.666667
69.987618
369.500000
56.450864
293.166667
45.375838
236.833333
38.096807
--------------y-------------Mean
Std Dev
298.333333
185.106051
218.666667
128.739077
308.555556
85.475305
291.111111
57.953525
142
Interpretation of second order interaction
143
Exercise: Obtain µ̂(ABC2) and µ̂(ABC3) as well as AB interaction
plots for C = 1, C = 2 and C = 3. Interpret the plots.
µ̂(ABC1) = 408 − 70 − (331 − 71) ≈ 77
A: We could compute µ̂(ABC1), µ̂(ABC2), µ̂(ABC3) and see if these
first order interactions, with C fixed, are the same. (We know they
are not by the FABC ratio and p-value.)
Q: How is the ABC interaction manifested here?
B
(C = 1) B1 B2
A A1 70 71
A2 408 331
B
(C = 2) B1 B2
A A1 466 333
A2 275 312
B
(C = 3) B1 B2
A A1 359.0 252
A2 243 231
To do this, look at three 2 × 2 tables as follows:
Consider the AB interaction at each of three levels, C1, C2, C3.
1st order interaction is between two factors
2nd order interaction is between three factors
Osborne, ST512
− ȳ111+} − ȳ|221+ {z
− ȳ121+}
|ȳ211+ {z
effect of A at B1C1 effect of A at B2C1
=
=
=
=
µ + α2 + β1 + γ1 + (αβ)21 + (αγ)21 + (βγ)11 + (αβγ)211
−µ − α1 − β1 − γ1 − (αβ)11 − (αγ)11 − (βγ)11 − (αβγ)111
−µ − α2 − β2 − γ1 − (αβ)22 − (αγ)21 − (βγ)21 − (αβγ)221
µ + α1 + β2 + γ1 + (αβ)12 + (αγ)11 + (βγ)21 + (αβγ)121
µ(ABC1) = −(αβ)11 + (αβ)12 + (αβ)21 − (αβ)22
− (αβγ)111 + (αβγ)121 + (αβγ)211 − (αβγ)221
These can be rearranged so that they agree with the ordering of the
treatment combinations employed by the ESTIMATE statement:
µ(ABC1) = (αβ)21 − (αβ)11 − (αβ)22 + (αβ)12
+ (αβγ)211 − (αβγ)111 − (αβγ)221 + (αβγ)121
Add these all up to get the contrast we’re interested in, µ(ABC1).
Note that all terms vanish except the second order parameters and
first order (αβ)ij parameters:
E(ȳ211+)
E(−ȳ111+)
E(−ȳ221+)
E(ȳ121+)
Using the model to specify parameters, we can write
µ̂(ABC1) =
µ[ABC3] = (−1, 1, 1, −1)(αβ)+(0, 0, −1, 0, 0, 1, 0, 0, 1, 0, 0, −1)(αβγ).
µ[ABC2] = (−1, 1, 1, −1)(αβ)+(0, −1, 0, 0, 1, 0, 0, 1, 0, 0, −1, 0)(αβγ).
Similarly for µ[ABC2] and µ[ABC3]:
µ[ABC1] = (−1, 1, 1, −1)(αβ)+(−1, 0, 0, 1, 0, 0, 1, 0, 0, −1, 0, 0)(αβγ).
and likewise for (αβγ), then the contrast µ[ABC1] on p. 25 can be
written
If (αβ) is a vector of AB interaction effects with the default ordering:


(αβ)11


 (αβ)12 
(αβ) = 

 (αβ)21 
(αβ)22
. . . , A2B1C1, A2B1C2, A2B1C3, A2B2C1, A2B2C2, A2B2C3
A1B1C1, A1B1C2, A1B1C3, A1B2C1, A1B2C2, A1B2C3, . . .
Similarly, with a CLASS a b c; statement, the following order is
used for second-order interaction parameters
ESTIMATE statement with two two-level and a three level factor
A1B1, A1B2, A2B1, A2B2
If A appears before B in the CLASS statement, SAS uses the following ordering for (αβ) terms when specifying contrasts in ESTIMATE
(or CONTRAST) statements:
145
To get SAS to estimate the interaction like µ(ABC1), the AB interaction at C = 1, you must specify the parameters involved.
We saw on the last page that
Osborne, ST512
ESTIMATE statement with two two-level factors
144
Getting interaction contrasts using the ESTIMATE statement in GLM
Osborne, ST512
77.333333
-169.666667
-94.333333
209.333333
-161.166667
Estimate
1
0
1
0
1
1
1
-1
1 0
-1
0 1
-1
0 0
1 2
62.2230159
62.2230159
62.2230159
76.2073196
76.2073196
Standard
Error
1.24
-2.73
-1.52
2.75
-2.11
t Value
0.2259
0.0118
0.1426
0.0112
0.0450
Pr > |t|
1 0 0 -1;
-1 -1 2 -1 -1 -2 1 1/divisor=2;
0 0 -1 0;
0 -1 0 0;
146
1
θ = µ(ABC1) − (µ(ABC2) + µ(ABC3))
2
Adding up all the coefficients in this combination yields the contrast
below to use with an ESTIMATE statement:
1 1
1 1
1 1
1 1
(−1, , , 1, − , − , 1, − , − , −1, , )
2 2
2 2
2 2
2 2
PS: Three-factor interactions are not easily interpreted. Effects
can sometimes be made additive through a transformation of the
response.
The FABC ratio indicates the three contrasts estimated above are not
plausibly (p = 0.05) equal. (µ̂(ABC1), µ̂(ABC2) and µ̂(ABC3) differ significantly). Interaction plots from p.24 suggest the comparison
theta1: ABC1
theta2: ABC2
theta3: ABC3
t1-av(t2+t3)
t2-av(t1+t3)
Parameter
proc glm;
class a b c;
model y=a|b|c;
estimate "theta1: ABC1" a*b -1 1
a*b*c -1 0 0 1 0
estimate "theta2: ABC2" a*b -1 1
a*b*c 0 -1 0 0 1
estimate "theta3: ABC3" a*b -1 1
a*b*c 0 0 -1 0 0
estimate "t1-av(t2+t3)" a*b*c -2
means a|b|c;
run;
Osborne, ST512
147
proc glm;
class additive uv;
model ycount=additive uv uv*additive;
estimate ’acetic uv=0 vs acetic uv=1’ uv 1 -1 uv*additive 1 -1 0 0 0 0;
estimate ’acetic uv=0 vs nothing uv=0’ uv 1 -1 uv*additive 1 0 -1 0 0 0;
estimate ’acetic uv=0 vs nothing uv=1’ uv 1 -1 uv*additive 1 0 0 -1 0 0;
estimate ’acetic uv=0 vs sorbate uv=0’ uv 1 -1 uv*additive 1 0 0 0 -1 0;
estimate ’acetic uv=0 vs sorbate uv=1’ uv 1 -1 uv*additive 1 0 0 0 0 -1;
estimate ’acetic uv=1 vs nothing uv=0’ uv 1 -1 uv*additive 0 1 -1 0 0 0;
estimate ’acetic uv=1 vs nothing uv=1’ uv 1 -1 uv*additive 0 1 0 -1 0 0;
estimate ’acetic uv=1 vs sorbate uv=0’ uv 1 -1 uv*additive 0 1 0 0 -1 0;
estimate ’acetic uv=1 vs sorbate uv=1’ uv 1 -1 uv*additive 0 1 0 0 0 -1;
estimate ’nothing uv=0 vs nothing uv=1’ uv 1 -1 uv*additive 0 0 1 -1 0 0;
estimate ’nothing uv=0 vs sorbate uv=0’ uv 1 -1 uv*additive 0 0 1 0 -1 0;
estimate ’nothing uv=0 vs sorbate uv=1’ uv 1 -1 uv*additive 0 0 1 0 0 -1;
estimate ’nothing uv=1 vs sorbate uv=0’ uv 1 -1 uv*additive 0 0 0 1 -1 0;
estimate ’nothing uv=1 vs sorbate uv=1’ uv 1 -1 uv*additive 0 0 0 1 0 -1;
estimate ’sorbate uv=0 vs sorbate uv=1’uv 1 -1 uv*additive 0 0 0 0 1 -1;
estimate ’uv=0 vs uv=1’ uv 1 -1;
estimate ’acetic vs nothing’ additive 1 -1;
estimate ’acetic vs sorbate’ additive 1 0 -1;
estimate ’nothing vs sorbate’ additive 0 1 -1;
I am still having trouble with the estimate statements, the only ones
that work for the additive*uv interaction are where we contrast the
same additive over the uv, can anything be done about this??
To: [email protected]
Subject: non estimatable estimate statements
Activity w/ ESTIMATE statement
A linear function θ(β) of parameters is estimable if and only if there
is a linear combination of Y whose expected value is θ.
Exercise: identify the estimable contrasts in each of the ESTIMATE
statements in the correspondence below, which pertains to a 3 × 2
study with factors and levels
Levels
Factor
A : additive acetic, nothing, sorbate
0,1.
B : uv
Osborne, ST512
148
149
effect of interest
estimate
1
µ + α1 + 2 (β1 + β2) 188.03
µ + α2 + 12 (β1 + β2)
µ + 12 (α1 + α2) + β1
µ + 12 (α1 + α2) + β2
µ + α1 + β1
µ + α2 + β1
µ + α1 + β2
µ + α2 + β2
Pr > |t|
<.0001
<.0001
age
jr
sr
y LSMEAN
188.025773
247.097938
Standard
Error
16.233482
15.842256
Pr > |t|
<.0001
<.0001
θ̂ = c11ȳ11+ + c12ȳ12+ + c21ȳ21+ + c22ȳ22+.
P c2ij
The coefficients are chosen so that E(θ̂) = θ and
nij is minimized.
Standard
Error
16.233482
15.842256
Marginal sample means are not real useful in this unbalanced study.
Q: How are group population means estimated then?
A: Least squares means (what would be estimated by marginal
means if design were balanced).
y LSMEAN
251.525773
183.597938
All of these quantities are estimated using linear combinations of the
treatment means of the form:
gender
m
w
The GLM Procedure
Least Squares Means
Invoking the command lsmeans age gender; will report least squares
estimates for the first four means above.
Population group
Young folks
Older folks
Men
Women
Young men
Older men
Young women
Older women
Parametric expressions for all the population means of interest are
given below for the additive model:
Osborne, ST512
Exercise: finish parametric expressions for expected values below:
1
E(Ȳ1++) = µ + α1 + (6β1 + β2)
7
1
E(Ȳ2++) = µ + α2 + (β1 + 7β2)
8
E(Ȳ+1+) =
E(Ȳ+2+) =
Yijk = µ + αi + βj + Eijk
Consider an additive two-factor ANOVA model:
Gender
Age
Male
Female
Marginal mean
young 271,192,189,209,
162
ȳ1++ = 212.3
227,236
old
289
262,193,224,201 ȳ2++ = 221.6
161,178,265
ȳ+1+ = 230.4
ȳ+2+ = 205.8
ȳ11 = 220.7, ȳ12 = 162, ȳ21 = 289, ȳ22 = 212.
Recall the 2 × 2 cholesterol study. Suppose the study is unbalanced
and the data are given by
Topic: Supplement on design imbalance
Reading: None
Osborne, ST512
150
c21 =
1
1
− 18/97, c22 = + 18/97
2
2
so that the estimate for the old folks mean is
18
18
1 18
1 18
ȳ11+ − ȳ12+ + ( − )ȳ21+ + ( + )ȳ22+ = 247.1.
97
97
2 97
2 97
Exercise: obtain least squares estimators and estimates of marginal
means for men and women as well as for each age×gender combination.
and
c11 = 18/97 = −c12
which is minimized at c11 = 66
97 by setting the derivative to zero and
solving. The least squares mean is then
66
1 66
66 1
66
θ̂ = ȳ11+ + (1 − )ȳ12+ + ( − )ȳ21+ + ( − )ȳ22+ = 188.03.
97
97
2 97
97 2
Similarly for old folks, the contrast with minimum variance has
c2
(1 − c11)2 ( 12 − c11)2 (c11 − 12 )2
c211 c212 c221 c222
+
+
+
= 11 +
+
+
n11 n12 n21 n22 n11
n12
n21
n22
c11 + c12 = 1(coeff for α1)
c21 + c22 = 0(coeff for α2)
1
c11 + c21 = (coeff for β1)
2
1
c12 + c22 = (coeff for β2)
2
Variance is then proportional to
Example: What are the coefficients for the contrast which estimates
the population mean for young folks, µ + α1 + 12 (β1 + β2) with minimum variance?
Osborne, ST512
i
j
k
151
c11 + c12
c21 + c22
c11 + c21
c12 + c22
=
=
=
=
with
SS[θ̂age] =
6
2
( 48
97 )
+
1
+
2
( −48
97 )
1
(−59.07)2
2
(−1− 48
97 )
+
2
(−1+ 48
97 )
7
which leads to
θ̂age = −59.07
48
97
1(coeff for α1)
−1(coeff for α2)
0(coeff for β1)
0(coeff for β2)
Var(θ̂age) is minimized when c11 =
where
= 6044.
θ̂age = α̂1 − α̂2 = c11ȳ11+ + c12ȳ12+ + c21ȳ21+ + c22ȳ22+
We can obtain the coefficients of the LS estimate of this contrast and
then use them to get SS[θ̂age], which is the sum of squares for the
age effect adjusted for gender, or type II sum of squares for age:
θage = α1 − α2.
Alternatively, consider the contrast
A: Might not be a good idea if factor B has an effect.
(This is the type I SS for Age if Age is the first factor entered into
the model, or the so-called unadjusted sum of squares for age.)
Q: Is there an age effect? Should we base our conclusion on
XXX
(ȳi++ − ȳ+++)2 = 325.6?
Osborne, ST512
152
proc glm;
class age gender;
model y=age gender/solution;
lsmeans gender age/stderr;
estimate "lsmean for young folks" intercept 2 age 2 0 gender 1 1/divisor=2;
estimate "lsmean for older folks" intercept 2 age 0 2 gender 1 1/divisor=2;
estimate "lsmean for men" intercept 2 age 1 1 gender 2 0/divisor=2;
estimate "lsmean for women" intercept 2 age 1 1 gender 0 2/divisor=2;
estimate "lsmean for young men" intercept 1 age 1 0 gender 1 0;
estimate "lsmean for young women" intercept 1 age 1 0 gender 0 1;
estimate "lsmean for old men" intercept 1 age 0 1 gender 1 0;
estimate "lsmean for old women" intercept 1 age 0 1 gender 0 1;
contrast "age effect" age 1 -1;
estimate "age effect" age 1 -1;
contrast "gender effect" gender 1 -1;
means gender age;
run;
/*
I
221 213 202 183 185 197 162
II 271 192 189 209 227 236 142
III 262 193 224 201 161 178 265
IV 192 253 248 278 232 267 289
*/
options ls=75;
data one;
input gender $ age $2. @;
do i=1 to 7;
input y @@;
output;
end;
cards;
w jr . . . . . . 162
m jr 271 192 189 209 227 236 .
w sr 262 193 224 201 161 178 265
m sr . . . . . . 289
;
run;
Osborne, ST512
Levels
2
2
Number of observations
Class
age
gender
28
Values
jr sr
m w
The SAS System
The GLM Procedure
Class Level Information
jr
sr
m
w
Estimate
213.1340206
-59.0721649
0.0000000
67.9278351
0.0000000
DF
1
1
Source
age
gender
Parameter
Intercept
age
age
gender
gender
DF
1
1
B
B
B
B
B
1
153
Pr > F
0.0457
0.0249
Pr > F
0.6144
0.0249
Pr > F
0.0669
Pr > |t|
<.0001
0.0457
.
0.0249
.
F Value
4.97
6.57
F Value
0.27
6.57
t Value
16.69
-2.23
.
2.56
.
Mean Square
6044.348306
7992.437592
Mean Square
325.629762
7992.437592
Standard
Error
12.77243521
26.50916484
.
26.50916484
.
Type III SS
6044.348306
7992.437592
Type I SS
325.629762
7992.437592
F Value
3.42
y Mean
217.2667
Mean Square
4159.03368
1217.23883
Root MSE
34.88895
Sum of
Squares
8318.06735
14606.86598
22924.93333
Coeff Var
16.05812
DF
2
12
14
Source
age
gender
R-Square
0.362839
Source
Model
Error
Corrected Total
NOTE: Due to missing values, only 15 observations can be used in this
analysis.
Osborne, ST512
age
jr
sr
gender
m
w
young folks
older folks
men
women
young men
young women
old men
old women
Standard
Error
16.2334818
15.8422561
16.2334818
15.8422561
13.7197962
26.2714097
26.2714097
12.7724352
26.5091648
t Value
11.58
15.60
15.49
11.59
16.18
5.86
10.70
16.69
-2.23
Pr > F
0.0457
0.0249
154
Pr > |t|
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
0.0457
F Value
4.97
6.57
Pr > |t|
<.0001
<.0001
Pr > |t|
<.0001
<.0001
Mean Square
6044.348306
7992.437592
Standard
Error
16.233482
15.842256
Standard
Error
16.233482
15.842256
Contrast SS
6044.348306
7992.437592
Estimate
188.025773
247.097938
251.525773
183.597938
221.989691
154.061856
281.061856
213.134021
-59.072165
DF
1
1
y LSMEAN
188.025773
247.097938
y LSMEAN
251.525773
183.597938
Least Squares Means
http://www.stat.ncsu.edu/people/dickey/courses/st512/crsnotes/rnotes unbal.htm
(These notes were adapted from one of Dr. Dickey’s lectures:)
Parameter
lsmean for
lsmean for
lsmean for
lsmean for
lsmean for
lsmean for
lsmean for
lsmean for
age effect
Contrast
age effect
gender effect
Osborne, ST512
155
2. Blocks are randomly assigned to each of the t treatments. (Restricted randomization, as opposed to a completely randomized
design.
1. matched sets of experimental units are formed, each consisting
of t units. Goal is reduced variability of response within a block
That is, the units within a block are homogeneous. Variance
between blocks is ok.
In a randomized block design (RBD),
Block Designs
Reading Ch. 15.3,15.4,15.6
Motivation - sometimes the variability of responses among experimental units is large, making detection of differences among treatment means µ1, µ2, . . . , µt difficult
Osborne, ST512
RBD - first example
156
Contact
Block j Desensitization
1
8
y12 = 11
2
3
9
16
4
5
24
Avg ȳi+
13.6
Therapy
Demonstration
Live
Participation Modeling ȳ+j
2
-2
2.67
1
0
4
12
6
9
11
2
9.67
19
11
18
9
3.4
Severity of acrophobia measured by HAT (Height Avoidance Test)
scores. The point of the study is to investigate the effectiveness of
the three therapies. The study will measure HAT scores before and
after therapy. There is considerable heterogeneity in the degree to
which acrophobia afflicts subjects. So N = 15 subjects will be put
into blocks according to their original HAT score, then one from
each block will be randomly assigned to a therapy: Let Yij denote
the change in HAT score for subject in block j assigned to treatment
i. (Bigger score means bigger reduction in fear.)
• “Live Modeling” - subject simply watches completion of task
• “Demonstration participation” - therapist talks subject through
task without any contact.
• “Contact desensitization ” - activity/task demonstrated then walked
through while a therapist is in constant contact with the subject.
Acrophopia can be treated in several ways.
Osborne, ST512
RBD example
i=1 j=1
a X
b
X
i=1 j=1
b
a X
X
i=1 j=1
a X
b
X
i=1 j=1
b
a X
X
(ȳij − ȳi+ − ȳj + ȳ++)2
j=1
(ȳ+j − ȳ++)2
(ȳi+ − ȳ++)2
b
X
i=1
a
X
(ȳ+j − ȳ++)2 = a
(ȳi+ − ȳ++)2 = b
(yij − ȳ++)2
157
yij − ȳ++ = (ȳi+ − ȳ++) + (ȳ+j − ȳ++) + (yij − ȳi+ − ȳ+j + ȳ++)
| {z } |
{z
} |
{z
}
error
therapy effect block effect
Note that
SS[E] =
SS[B] =
SS[A] =
SS[T ot] =
SS[T ot] = SS[A] + SS[B] + SS[E]
(Data taken from Larsen and Marx, 1986)
Source
Sum of Squares d.f. Mean Square F
A: Therapies
260.9
2
130.5
15.3
B: Blocks
438
4
109.5
12.8
Error
68.4
8
8.6
Total
767.3
14
Osborne, ST512
F -tests in the RBD
vs
H1 : not all equal.
158
For the HAT scores, FA = M S[A]/M S[E] = 130.5/8.6 = 15.3
which has p < 0.01 on 2, 8 df, providing strong evidence of a therapy
effect. Inference, including MCPs, for CONTRASTS involving fixed
effects is the same in the complete RBD as it isp
for other factorial
c
experiments with fixed effects. (E.g. SE(Ȳi+) = M S[E]/b )
The EM S for error is E(M S[E]) = σ 2, but only under the additivity assumption that there is no block-trt interaction. This assumption is required for inference about treatment effects in the absence
of replication, common to block designs.
Using level α, reject H0 if
M S[A]
> F (α, a − 1, N − a − b + 1)
F =
M S[E]
H0 : α1 = α2 = α3 = 0
Mean squares obtained by dividing SS by df :
SS[A]
M S[A] =
a−1
SS[B]
M S[B] =
b−1
SS[E]
M S[E] =
N −a−b+1
The primary hypothesis of interest is for a therapy effect:
where i = 1, . . . , a j = 1, . . . , b and Eij ∼ N (0, σ 2)
iid
Yij = µ + αi + βj + Eij
A model for RBD with fixed treatment (therapy) effects is
Osborne, ST512
(a − 1)(F ∗)M S[E]
r
i
X c2
159
ȳ2+ = 9,
ȳ3+ = 3.4
The Tukey minimum significant difference term in the RBD is
r
M S[E]
q(a, n − a − b + 1, α)
b
q
For the acrophobia RBD, the term is 4.04 ∗ 8.6
5 = 5.3
means therapy/scheffe tukey; will get the job done in SAS.
with ȳLM + significantly different from the other two. (LM brings
about significantly less improvement than the other two therapies.)
and
p
p
(a − 1)(F ∗)(M S[E])(1/5 + 1/5) = (3 − 1)(4.46)(8.6)(2/5) = 5.5
ȳ1+ = 13.6,
For the HAT scores,
b
where F ∗ = F (0.05, a − 1, N − a − b + 1). For simultaneous pairwise
differences, these look like
r
2
(a − 1)(F ∗)M S[E]
ȳi+ − ȳj+ ±
b}
{z
|
“minimum significant difference”
c1ȳ1+ + c2ȳ2+ + · · · + caȳa+ ±
look like
c 1 µ1 + c 2 µ2 + · · · + c a µa
Scheffè simultaneous 95% confidence intervals for contrasts like
Multiple comparisons among means in the RBD
Osborne, ST512
160
5
9.000
3.400
B
Live Modelling
Demonstration Pa
Contact Desensit
TREAT
5
9.000
3.400
B
5
5
13.600
A
A
A
N
Mean
Scheffe Grouping
Live Modelling
Demonstration Pa
Contact Desensit
TREAT
Alpha= 0.05 df= 8 MSE= 8.55
Critical Value of F= 4.45897
Minimum Significant Difference= 5.5226
NOTE: This test controls the type I experimentwise error rate but
generally has a higher type II error rate than REGWF for all
pairwise comparisons
Scheffe’s test for variable: DIFF
5
5
13.600
A
A
A
N
Mean
Tukey Grouping
Means with the same letter are not significantly different.
Alpha= 0.05 df= 8 MSE= 8.55
Critical Value of Studentized Range= 4.041
Minimum Significant Difference= 5.2843
NOTE: This test controls the type I experimentwise error rate, but
generally has a higher type II error rate than REGWQ.
Tukey’s Studentized Range (HSD) Test for variable: DIFF
Osborne, ST512
161
Yij = µ + αi + Bj + Eij
iid
systems
Tubex
25.0
26.0
25.3
24
24.2
26.2
24
20.9
36.9
25.8
ȳ+j
25.6
24.0
25.6
23.5
26.4
24.9
23.9
24.9
31.3
25.6
• Bj ∼ N (0, σB2 ) and Eij ∼ N (0, σ 2) denote random subject
(block) and error effects (B ⊥ E).
iid
• αi denote fixed system effects
• i = 1, . . . , 4 = a and j = 1, . . . , 9 = b
Model
Average times (seconds) for implementing
Subject Standard Vari-Ject Unimatic
1
35.6
17.3
24.4
2
31.3
16.4
22.4
36.2
18.1
22.8
3
4
31.1
17.8
21
39.4
18.8
23.3
5
6
34.7
17
21.8
34.1
14.5
23
7
36.5
17.9
24.1
8
9
40.7
16.4
31.3
ȳi+
35.5
17.1
23.8
Another example, blocks are random
(this material to be covered after random effects have been introduced)
A study investigates the efficiency of four different unit-dose injection systems. For each system, an individual subject (pharmacist or
nurse) measures the average time it takes to remove a unit of each
system from its outer package, assemble it, and simulate an injection.
(Data from Larsen and Marx, 1986.)
Osborne, ST512
Source
system
subject
Residual
DF
3
8
24
Mean Square
519.734074
22.175625
6.186366
162
Expected Mean Square
Var(Residual) + Q(system)
Var(Residual) + 4 Var(subject)
Var(Residual)
Type 3 Analysis of Variance
36
WORK.ONE
time
Variance Components
Type 3
Factor
Model-Based
Satterthwaite
Values
1 2 3 4
1 2 3 4 5 6 7 8 9
Total Observations
Levels
4
9
Sum of
Squares
1559.202222
177.405000
148.472778
Class
system
subject
Data Set
Dependent Variable
Covariance Structure
Estimation Method
Residual Variance Method
Fixed Effects SE Method
Degrees of Freedom Method
The SAS System
The Mixed Procedure
Model Information
proc mixed method=type3;
class system subject;
model time=system/ddfm=satterth;
random subject;
lsmeans system/adj=tukey cl pdiff;
run;
data one;
input subject system time;
cards;
1 1 35.6
2 1 31.3
...
8 4 20.9
9 4 36.9
;
run;
Osborne, ST512
1
system
1
2
3
4
Den
DF
24
system
1
2
3
4
Effect
system
system
system
system
DF
21.9
21.9
21.9
21.9
Means
33.3044
14.9266
21.5822
23.6266
Lower
Standard
Error
1.0637
1.0637
1.0637
1.0637
Least Squares
37.7178
19.3400
25.9956
28.0400
Upper
Pr > |t|
<.0001
<.0001
<.0001
<.0001
F Value
84.01
3.58
.
Pr > F
<.0001
t Value
33.38
16.11
22.36
24.29
F Value
84.01
Least Squares Means
Num
DF
3
Estimate
35.5111
17.1333
23.7889
25.8333
Effect
system
3.9973
6.1864
Type 3 Tests of Fixed Effects
subject
Residual
Error
Error Term
DF
MS(Residual)
24
MS(Residual)
24
.
.
Covariance Parameter Estimates
Cov Parm
Estimate
Alpha
0.05
0.05
0.05
0.05
Pr > F
<.0001
0.0072
.
163
Clearly, the injection system effects are highly significant, as is the
random block (or subject) effect, which has an estimated variance
component of
1
1
σ̂B2 = (M S[B] − M S[E]) = (22.2 − 6.2) = 4( squared seconds)
a
4
Effect
system
system
system
system
Source
system
subject
Residual
Osborne, ST512
2
3
4
3
4
4
DF
24
24
24
24
24
24
t Value
15.67
10.00
8.25
-5.68
-7.42
-1.74
<.0001
<.0001
<.0001
<.0001
<.0001
0.3242
Adj P
0.05
0.05
0.05
0.05
0.05
0.05
Alpha
15.9579
9.3023
7.2579
-9.0755
-11.1199
-4.4644
Lower
20.7977
14.1421
12.0977
-4.2356
-6.2801
0.3755
Upper
164
15.1433
8.4878
6.4433
-9.8900
-11.9345
-5.2789
Ȳi1+ = µ + αi1 + B̄ + Ēi1+
Ȳi2+ = µ + αi2 + B̄ + Ēi2+
Ȳi1+ − Ȳi2+ = αi1 − αi2 + Ēi1+ − Ēi2+
r
1 2
(σB + σ 2)
SE(Ȳi1+) =
b
r
2 2
σ
SE(Ȳi1+ − Ȳi2+) =
b
• For means, pesky mean random effects don’t wash out, necessitating a Satterthwaite df approximation
21.6122
14.9567
12.9122
-3.4211
-5.4655
1.1900
Adj
Upper
Adjustment
Tukey-Kramer
Tukey-Kramer
Tukey-Kramer
Tukey-Kramer
Tukey-Kramer
Tukey-Kramer
Adj
Lower
Pr > |t|
<.0001
<.0001
<.0001
<.0001
<.0001
0.0940
• For difference of means, pesky mean random effects wash out
Note the df columns:
1
1
1
2
2
3
system
system
system
system
system
system
Standard
Error
1.1725
1.1725
1.1725
1.1725
1.1725
1.1725
Differences of Least Squares Means
Estimate
18.3778
11.7222
9.6778
-6.6556
-8.7000
-2.0444
Differences of Least Squares Means
_system
_system
2
3
4
3
4
4
system
system
1
1
1
2
2
3
Effect
Effect
system
system
system
system
system
system
Osborne, ST512
165
2 3 1
1 2 3
3 1 2
Eg, random number generator permutes (1, 2, 3) to get (2, 3, 1). Placing the columns in this order leads to
2. rows
1. columns
Fertilizer trts randomized to row×column (or sunlight × iheight)
combinations by randomly permuting the
1 2 3
3 1 2
2 3 1
Non-randomized design (number indicates trt):
• location on bench a second blocking factor
• blocks to control for variability in exptl. units
• Experiment with ∼ 30 plants, 3 fertilizer trts.
Latin squares for experiments with two blocking factors
Osborne, ST512
166
2 1 3
3 2 1
1 3 2
2 3 1
3 1 2
1 2 3
3 1 2
1 2 3
2 3 1
Suppose nine rows are available. Three latin squares may be generated, one below the other. The one on top is closest to sunlight, the
one on bottom furthest. Columns correspond to initial height blocks.
2 3 1
3 1 2
1 2 3 −→ 1 2 3
3 1 2
2 3 1
Another random permutation of (1, 2, 3) is (3, 2, 1). Placing the rows
in this order leads to an unreplicated 3 × 3 × 3 design:
Osborne, ST512
167
• For each k, (ρ1(k), ρ2(k), ρ3(k)) = (3, 0, −3) are nested row effects
• (β1, β2, β3) = (3, 0, −3) are square effects
where
Yijkl = µ + τl + ρi(k) + κj + βk + Eijkl
Fixed effect model replicated design has extra term:
2. specify the marginal means for trt, row and column
1. specify the theoretical mean in each of the nine cells of the unreplicated design in square 1.
Exercises:
• Eijk ∼ N (0, σ 2) with σ 2 = 1.
iid
• (κ1, κ2, κ3) = (−2, 0, 2) are initial ht. effects.
• (ρ1, ρ2, ρ3) = (1, 0, −1) are sunlight effects.
• µ = 13, (τ1, τ2, τ3) = (−1, 0, 1) are trt effects
where
Yijk = µ + τk + ρi + κj + Eijk
SAS code to illustrate how such an experiment might go follows. To
consider an unreplicated 3 × 3 × 3 Latin square, ignore squares 2 and
3. The fixed effects model generated by the code is:
Osborne, ST512
data latinsq;
array sqware{3} (3,0,-3);
array slight1{3} (1,0,-1);
/* in square 1 */
array slight2{3} (1,0,-1);
/* in square 2 */
array slight3{3} (1,0,-1);
/* in square 3 */
array iheight{3} (-2,0,2);
/* initial height effects */
array treatment{3} (-1,0,1); /* fertilizer effects */
input square row col trt;
growth=round(growth,0.1);
sigma=1; /* try various values of sigma to generate the data */
if square=1 then do;
growth = 10 + sqware{square} + slight1{row} + iheight{col}
+ treatment{trt} + sigma*rannor(1234);
end;
else if square=2 then do;
growth = 10 + sqware{square} + slight2{row} + iheight{col}
+ treatment{trt} + sigma*rannor(1234);
end;
else do;
growth = 10 + sqware{square} + slight3{row} + iheight{col}
+ treatment{trt} + sigma*rannor(1234);
end;
height=col;
cards;
1 1 1 3
1 1 2 1
1 1 3 2
1 2 1 1
1 2 2 2
1 2 3 3
1 3 1 2
1 3 2 3
1 3 3 1
2 1 1 2
2 1 2 3
2 1 3 1
2 2 1 1
2 2 2 2
2 2 3 3
2 3 1 3
2 3 2 1
2 3 3 2
3 1 1 2
3 1 2 1
3 1 3 3
3 2 1 3
3 2 2 2
3 2 3 1
3 3 1 1
3 3 2 3
3 3 3 2 ;
Osborne, ST512
168
Source
row
col
trt
R-Square
0.969083
Source
Model
Error
Corrected Total
Levels
1
3
3
3
growth
LSMEAN
14.7000000
13.5000000
11.1000000
growth
LSMEAN
12.5333333
13.3666667
13.4000000
col
1
2
3
trt
1
2
3
F Value
10.45
F Value
9.88
20.03
1.44
growth Mean
13.10000
Mean Square
4.97333333
10.08000000
0.72333333
growth
LSMEAN
14.5000000
12.8333333
11.9666667
Type I SS
9.94666667
20.16000000
1.44666667
Root MSE
0.709460
Mean Square
5.25888889
0.50333333
9
Values
1
1 2 3
1 2 3
1 2 3
row
1
2
3
DF
2
2
2
Coeff Var
5.415724
DF
6
2
8
Sum of
Squares
31.55333333
1.00666667
32.56000000
Number of observations
Class
square
row
col
trt
data one; set latinsq; if square=1; run;
proc glm data=one;
class square row col trt;
model growth = row col trt;
lsmeans row col trt;
run;
The SAS System
The GLM Procedure
Osborne, ST512
Pr > F
0.0919
0.0476
0.4103
Pr > F
0.0899
1
169
DF
2
6
2
2
Source
square
row(square)
col
trt
row
1
2
3
1
2
3
1
2
3
Mean Square
77.6677778
5.5962963
48.3100000
7.7744444
Mean Square
25.0901852
0.9126984
growth
LSMEAN
12.3666667
10.0000000
7.7333333
growth
LSMEAN
9.0444444
10.1666667
10.8888889
col
1
2
3
trt
1
2
3
growth
LSMEAN
14.5000000
12.8333333
11.9666667
11.3333333
9.5000000
8.4333333
8.6333333
7.1333333
5.9666667
growth
LSMEAN
13.1000000
9.7555556
7.2444444
Type I SS
155.3355556
33.5777778
96.6200000
15.5488889
Sum of
Squares
301.0822222
12.7777778
313.8600000
square
1
1
1
2
2
2
3
3
3
square
1
2
3
DF
12
14
26
Source
Model
Error
Corrected Total
proc glm data=latinsq;
class square row col trt;
*model growth = row*square col trt;
model growth = square row(square) col trt;
lsmeans square row(square) col trt;
run;
The SAS System
The GLM Procedure
Osborne, ST512
F Value
85.10
6.13
52.93
8.52
F Value
27.49
Pr > F
<.0001
0.0025
<.0001
0.0038
Pr > F
<.0001
170
171
a
k=1
a
X
(ȳk − ȳ++)2 = SS[
(yij − ȳi+ − ȳ+j − ȳk + 2ȳ++)2 = SS[
j=1
(ȳ+j − ȳ++)2 = SS[
]
]
]
]
]
Note that ȳk is determined by the i, j combination. In the 3 × 3 × 3
scheme used for our plant heights, fertilizer k = 3 was assigned to
the first row and column so that in the last sum of squares above, for
i = 1, j = 1, ȳk is the third fertilizer treatment mean, ȳ3 = 13.40.
i=1 j=1
3
3 X
X
a
i=1 j=1
a
X
(yij − ȳ++)2 = SS[
(ȳi+ − ȳ++)2 = SS[
i=1 j=1
a
a X
X
a
a X
X
Exercise
Let Yij denote the observation in row i column j. Let Ȳk denote the
treatment mean for level k of the treatment factor. For an unreplicated latin square, identify these sums of squares:
Osborne, ST512
3
1
2
4
4
3
1
2
2
4
3
1
data watermelons;
input row col trt yield; cards;
1 1 1 1.75
1 2 3 1.43
1 3 4 1.28
1 4 2 1.66
2 1 2 1.7
2 2 1 1.78
2 3 3 1.40
2 4 4 1.31
3 1 4 1.35
3 2 2 1.73
3 3 1 1.69
3 4 3 1.41
4 1 3 1.45
4 2 4 1.36
4 3 2 1.65
4 4 1 1.73
;
proc glm;
class row col trt;
model yield = row col trt;
estimate "micronutrient effect A-B" trt 1 -1 1 -1/divisor=2;
contrast "micronutrient effect A-B" trt 1 -1 1 -1;
estimate "placement effect" trt 1 1 -1 -1/divisor=2;
contrast "placement effect" trt 1 1 -1 -1;
estimate "placement-x-nutrient interaction " trt 1 -1 -1 1;
contrast "placement-x-nutrient interaction " trt 1 -1 -1 1;
lsmeans row col trt; run;
1
2
4
3
• Treatment factor: Fertilizer (4 levels, 2 factors)
(1=broad A,2=broad B,3=band A,4=band B)
• Blocking factors: plot rows, plot columns
A 4 × 4 × 4 example taken from Ott and Longnecker
Osborne, ST512
172
DF
1
1
1
Contrast
micro.effect A-B
place.effect
interaction
Pr > |t|
<.0001
<.0001
0.0069
Pr > F
<.0001
<.0001
0.0069
Pr > F
0.1810
0.0004
<.0001
Symbol is value of micronutrient.
yield LSMEAN
1.73750000
1.68500000
1.42250000
1.32500000
1
Pr > F
<.0001
mean_yield |
1.8 +
|
A
|
B
|
1.6 +
|
|
|
1.4 + A
| B
|
|
1.2 +
|
---+-----------------------------------------+-band
broadcast
Plot of mean_yield*placement.
trt
1
2
3
4
F Value
180.00
3645.00
16.20
F Value
2.27
32.93
1280.40
F Value
438.53
t Value
13.42
60.37
-4.02
Mean Square
0.02250000
0.45562500
0.00202500
Mean Square
0.00028333
0.00411667
0.16005000
Mean Square
0.05481667
0.00012500
Standard
Error
0.00559017
0.00559017
0.01118034
Contrast SS
0.02250000
0.45562500
0.00202500
Type I SS
0.00085000
0.01235000
0.48015000
Sum of
Squares
0.49335000
0.00075000
0.49410000
Estimate
0.07500000
0.33750000
-0.04500000
DF
3
3
3
Source
row
col
trt
Parameter
micro.effect A-B
place.effect
interaction
DF
9
6
15
The SAS System
The GLM Procedure
Source
Model
Error
Corrected Total
Osborne, ST512
173
yield LSMEAN
1.42250000
1.73750000
1.32500000
1.68500000
yield LSMEAN
1.37375000
1.71125000
placement
band
broadcast
band
broadcast
placement
band
broadcast
Mean Square
0.00028333
0.00411667
0.02250000
0.45562500
0.00202500
yield LSMEAN
1.58000000
1.50500000
Type I SS
0.00085000
0.01235000
0.02250000
0.45562500
0.00202500
micronutrient
A
B
DF
3
3
1
1
1
F Value
438.53
F Value
2.27
32.93
180.00
3645.00
16.20
yield Mean
1.542500
Mean Square
0.05481667
0.00012500
Values
1 2 3 4
1 2 3 4
A B
band broadcast
Root MSE
0.011180
Sum of
Squares
0.49335000
0.00075000
0.49410000
Levels
4
4
2
2
Coeff Var
0.724819
DF
9
6
15
micronutrient
A
A
B
B
Source
row
col
micronutrient
placement
micronutri*placement
R-Square
0.998482
Source
Model
Error
Corrected Total
Class
row
col
micronutrient
placement
proc glm data=watermelons;
class row col micronutrient placement;
model yield = row col micronutrient|placement;
lsmeans micronutrient|placement;
run;
The SAS System
The GLM Procedure
Osborne, ST512
Pr > F
0.1810
0.0004
<.0001
<.0001
0.0069
Pr > F
<.0001
174
175
103
115
59
121
93
75
98
54
101
105
62
94
100
101
75
ȳi+
83.6
97.5
63.1
77.5
91.0
si
22.6
11.2
15.0
25.9
28.0
where τi denotes the difference between the mean birthweight of population of offspring from sire i and µ, mean of whole population.
Yij = µ + τi + Eij
A: One-way fixed effects model?
Q: Statistical model for these data?
Birthweights
Sire # Level
Sample
177
1
61 100 56 113 99
75 102 95 103 98
200
2
201
3
58 60 60 57 57
57 56 67 59 58
202
4
203
5
59 46 120 115 115
• N = 40, completely randomized.
• t = 5 sires, each mated to a separate group of n = 8 dams.
• Genetics study w/ beef animals. Measure birthweight Y (lbs).
An example using one-way random effects model
• Expected mean squares for mixed models
• Subsampling
• Mixed effects models
• One-way random effects model to study variances
Topic: Mixed Models for factorial designs
Reading: Rao Ch. 14.1,14.2,14.3
Osborne, ST512
The one-way random effects model
fixed
random
• E11, . . . , Etn ∼ N (0, σ 2)
iid
• τ1, τ2, . . . , τt unknown model parameters
with
fixed
µ + |{z}
τi + Eij for i = 1, 2, . . . , t and j = 1, . . . , n
Yij = |{z}
|{z}
No particular interest in τ1, τ2, . . . , τ5 from the fixed effects model:
For beef animal genetic study, with t = 5 and n = 8, the random
effects T1, T2, . . . , T5 reflect sire-to-sire variability.
• conceptually different from one-way fixed effects model
• σT2 and σ 2 are called variance components • T1, T2, . . . denote random effects, drawn from some population
of interest. That is, T1, T2, . . . is a random sample !
Features
• E11, . . . , Etn ∼ N (0, σ 2)
iid
177
The ANOVA table is almost the same, it just has a different expected
mean squares column:
Source
SS
df
MS
Expected MS
Treatment SS[T ] t − 1 M S[T ]
σ 2 + nσT2
Error
SS[E] N − t M S[E]
σ2
Total
SS[T ot] N − 1
Sums of squares and mean squares - same as in one-way fixed effects
ANOVA:
XX
SS[T ] =
(ȳi+ − ȳ++)2
XX
SS[E] =
(yij − ȳi+)2
XX
SS[T ot] =
(yij − ȳ++)2
Model parameters: σ 2, σT2 , µ
Contrast this situation with the binding fractions. Why not model
antibiotic effects random? Why fixed? (See Ch. 17 for more discussion.)
• Sire effects is a population in its own right.
• T1, T2, T3, T4, T5 a random sample of sire effects
E(Yij ) and Var(Yij )
• T1, T2, . . . , Tt independent of E11, . . . , Etn
random
One-way random effects model continued
Exercise: Using the random effects model, specify
Osborne, ST512
• Two components to variability in data: σ 2, σT2
iid
random
176
• T1, T2, . . . , Tt ∼ N (0, σT2 )
with
fixed
Yij = |{z}
µ + |{z}
Ti + Eij for i = 1, 2, . . . , t and j = 1, . . . , n
|{z}
Osborne, ST512
179
σ2
Cov(Yij , Yik )
= 2 T 2
Var(Yij )Var(Yik ) σT + σ
3. What do you do in that case?
2. How?
General questions:
• The estimated correlation between any two calves with the same
sire for a male parent, or the estimated intrasire correlation coefficient, is 0.20
= 0.29
= 0.20
1. Is it possible for an estimated variance component to be negative?
117+464
82.6
117
117+464
√
• The estimated standard deviation of a birthweight, 24.1 is 29%
of the estimated mean birthweight, 82.6.
d =
CV
ρ̂I =
3. Estimate how much of this variation is not due to the sire effect.
For sires,
• Bigger values of ρI correspond to (bigger/smaller?) random treatment effects.
• Interpretation: the correlation between two responses receiving
the same level of the random factor.
ρI = p
Intraclass correlation coefficient
Note: this is not estimated by Coeff Var in PROC GLM output.
Coefficient of variation (CV):
p
p
Var(Yij )
σT2 + σ 2
=
CV (Yij ) =
|E(Yij |)
|µ|
Other parameters of interest in random effects models
Osborne, ST512
Interpretations:
178
2. Estimate how much of this variation is due to the sire effect.
1. What is the estimated variance of such a calf?
Consider the birthweight of a randomly sampled calf.
Specific questions pertaining to this study:
µ̂ = 82.6
σ̂ 2 = 464 (lbs2)
1398 − 464
σ̂T2 =
8
= 117 (lbs2)
µ̂ = ȳ++
σ̂ 2 = M S[E]
M S[T ] − M S[E]
σ̂T2 =
n
For sires data, ȳ++ = 82.6 and
Source SS df MS Expected MS
Sire 5591 4 1398
σ 2 + 8σT2
Error 16233 35 464
σ2
Total 21824 39
Estimating parameters of one-way random effects model
Osborne, ST512
Type III Expected Mean Square
Root MSE
21.53585
bw Mean
82.55000
Mean Square
1397.78750
463.79286
Values
177 200 201 202 203
Sum of
Squares
5591.15000
16232.75000
21823.90000
Coeff Var
26.08825
DF
4
35
39
Levels
5
The GLM Procedure
Pr > F
0.0309
180
(σ 2 =Var(Error) and σT2 =Var(sire).)
sire
Var(Error) + 8 Var(sire)
----------------------------------------------------------------------------
Source
R-Square
0.256194
Source
Model
Error
Corrected Total
Class
sire
proc glm;
class sire;
model bw=sire;
random sire;
run;
F Value
3.01
Using PROC GLM for random effects models
data one;
input sire @;
do i=1 to 8;
input bw @; output;
end;
cards;
177 61 100 56 113 99 103 75 62
200 75 102 95 103 98 115 98 94
201 58 60 60 57 57 59 54 100
202 57 56 67 59 58 121 101 101
203 59 46 120 115 115 93 105 75
;
run;
Osborne, ST512
Testing a variance component - H0 : σT2 = 0
181
M S[T ]
M S[E]
F =
A: “Yes.”
Q: “Isn’t this just just like the F -test for one-way ANOVA with fixed
effects?”
1398
= 3.01 > 2.64 = F (0.05, 4, 35)
464
so H0 is rejected at α = 0.05. (The p-value is 0.0309)
For the sires,
reject H0 at level α if F > F (α, t − 1, N − t)
F =
Recall that σT2 = Var(Ti), the variance among the population of
treatment effects.
Osborne, ST512
1
N
i=1 j=1
i=1 j=1
n
t X
X
(µ + Ti + Eij )
Yij
82.6 ± 2.78(5.91) or (66.1, 99.0).
Sires data: ȳ++ = 82.6, M S[T ] = 1398, nt = 40. Critical value
t(0.025, 4) = 2.78 yields the interval
and a 95% confidence interval for µ given by
q
S[T ]
Ȳ++ ± t(0.025, t − 1) M nt
M S[T ]
nt
Ȳ++ − µ
q
∼ tt−1
Var(Ȳ++) = Var(T̄ + Ē++)
σ 2 σ2
= T +
t
nt
1
=
(nσT2 + σ 2)
nt
1
=
E(M S[T ]).
nt
If the data are normally distributed, then
PP
Eij )/N , so that
where T̄+ = (T1 + · · · + Tt)/t and Ē++ = (
= µ + T̄+ + Ē++
1
=
N
Ȳ++ =
n
t X
X
∼ χ2N −t
M S[T ]
M S[E]
/
∼ Ft−1,N −t
σ 2 + nσT2
σ2
(which explains the F test for H0 : σt2 = 0)
so that ρI gets left in the middle yields the confidence interval
yields the c.i. at the top o’ the page.
σ
• Rearranging the probability statement below


M S[T ]
2
2
α
α
σ +nσ
1−α = Pr F (1 − , t − 1, N − t) < M S[E]T < F ( , t − 1, N − t)
2
2
2
•
• Ratio of independent χ2 RVs divided by df has an F distribution
• M S[T ] and M S[E] are independent
• (N −
t) M σS[E]
2
T
S[T ]
2
• (t − 1) σM2+nσ
2 ∼ χt−1
These formulas arrived at via some distributional results:
Note the asymmetry and disagreement with test of H0 : σT2 = 0
For the sires, Fobs = 3.01 and F0.025 = 3.179, F0.975 = 0.119. The
formula gives (−0.01, 0.75).
where Fα/2 = F ( α2 , t − 1, N − t) and Fobs is the observed F -ratio for
treatment effect from the ANOVA table.
A 95% confidence interval for ρI can be obtained from the expression
Fobs − Fα/2
Fobs − F1−α/2
< ρI <
Fobs + (n − 1)Fα/2
Fobs + (n − 1)F1−α/2
183
A 95% confidence interval for µ derived by consideration of SE(Ȳ++):
Osborne, ST512
Confidence interval for ρI
182
Interval Estimation of some model parameters
Osborne, ST512
Label
mean
116.75
463.79
sire
Residual
Label
mean
Standard
Error
5.9114
Estimate
Estimate
82.5500
Values
177 200 201 202 203
Lower
66.1373
Estimates
DF
4
Estimates
0.05
0.05
Alpha
Upper
98.9627
t Value
13.96
29.9707
305.11
Lower
Upper
Pr > |t|
0.0002
7051.37
789.17
bw
Variance Components
REML
Profile
Model-Based
Containment
Covariance Parameter Estimates
Levels
5
Cov Parm
Class
sire
Dependent Variable
Covariance Structure
Estimation Method
Residual Variance Method
Fixed Effects SE Method
Degrees of Freedom Method
proc mixed cl;
class sire;
model bw=;
random sire;
estimate "mean" intercept 1/cl;
run;
The SAS System
The Mixed Procedure
Model Information
Using PROC MIXED for random effects models
Osborne, ST512
Alpha
0.05
1
184
185
Q: What are the mean and variance of the χ235 distribution?
Rearranging to get σ 2 in the middle yields the 100(1−α)% confidence
interval for σ 2:
!
"
(N − t)M S[E] (N − t)M S[E]
.
,
χ2α/2
χ21−α/2
(N − t)
M S[E]
∼ χ2N −t
σ2
setting up the probability statement
M S[E]
α
2
2 α
1−α = Pr χ (1 − , N − t) < (N − t)
< χ ( , N − t)
2
σ2
2
This can be derived using the distributional result
A 95% confidence interval for this variance component is given by
(40 − 5)464
(40 − 5)464
< σ2 <
53.2
20.6
or
35
35
464 < σ 2 <
464
53.2
20.6
2
or (305.2, 789.5)lbs
The estimated residual variance component for the sire data was
σ̂ 2 = M S[E] = 464 lbs2.
More interval estimation for variance components
Osborne, ST512
Interval estimation for σT2
186
M S[T ]2
t−1
+ MNS[E]
−t
2
2
b = (8 × 117) = 1.76
df
2
1398
4642
4 + 35
b =
df
(nσ̂T2 )2
1−α/2,df
χ20.025,1.76 = 6.87
yielding the 95% confidence interval
1.76(117) 1.76(117)
6.87
0.29
or
(30, 7051)
χ20.975,1.76 = 0.029,
Using the CL option in the MIXED statement will request this confidence interval and will use this approximation to df and will not
round to the nearest integer df :
For the sire data,
where
α/2,df
A: In a similar fashion, but the confidence level based on Satterthwaite’s approximation to the degrees of freedom of the linear combination of M S terms:
"
!
b σ̂ 2
b σ̂ 2
df
df
T
T
,
χ2 b χ2
b
Q: How can we get a 95% confidence interval for σT2 ?
The estimated variance component for the random sire effect was
σ̂T2 = 117.
Osborne, ST512
Review of one-way random effects ANOVA
The model
iid
random
random
iid
187
σT2
σ 2 +σT2
b
α/2,df
b
1−α/2,df
Exercise: match up the formulas for confidence intervals below with
their targets, ρI , σ 2, σT2 , µ:
q
S[T ]
Ȳ++ ± t(0.025, t − 1) M nt
F −F
Fobs −Fα/2
obs
1−α/2
Fobs +(n−1)F1−α/2 , Fobs +(n−1)Fα/2
(N −t)M S[E]
(N −t)M S[E]
,
χ2
χ2
α/2
1−α/2 2
b σ̂
b σ̂ 2
df
df
T
T
,
2
2
χ
χ
• Two observations from same treatment group not independent
– ρI = Corr(Yij , Yik ) =
• Several functions of these parameters of interest
√ 2 2
σ +σT
– CV (Y ) =
µ
• Only three parameters: µ, σ, σT2
• (T1, T2, . . . randomly drawn from pop’n of treatment effects.)
Remarks:
T1, T2, . . . , Tt ∼ N (0, σT2 ) independent of E11, . . . , Etn ∼ N (0, σ 2)
fixed
Yij = |{z}
µ + |{z}
Ti + Eij for i = 1, 2, . . . , t and j = 1, . . . , n
|{z}
with
Osborne, ST512
Inference
- for these levels used in this expt
- for the popn of levels
Goal
- estimate varcomps
- estimate longrun means
Another expt
- would use same levels
- would involve new levels sampled
from same popn
X
X
X
X
X
X
X
X
Random Fixed
Modelling factorial effects: fixed, or random?
A guide
Levels
- selected from conceptually ∞ popn of
collection of levels
- finite number of possible levels
Osborne, ST512
188
189
Suc
20
40
60
20
40
60
20
40
60
3.1
5.5
7.9
6
11.5
17.5
7.7
15.7
19.1
Sample
3.7
6.7
9.2
6.9
12.9
15.8
8.3
14.3
18.0
4.7
7.3
9.3
7.5
13.4
14.7
9.5
15.9
19.9
Type of Administration (j)
(j = 1)30mg × 1
(j = 2)15mg × 2
(i = 2) Brand II tablet (j = 1)20mg × 1
(j = 2)10mg × 2
(i = 3) Insulin injection (j = 1) before breakfast
(j = 2) before supper
Drug (i)
(i = 1) Brand I tablet
Mean ȳj(i)
15.7
19.7
20
17.3
28
33
Variance s2j(i)
6.3
9.3
1
6.3
4
9
• First factor is drug: brand I tablet, brand II tablet, insulin
injection
• Second factor is type of administration (see table)
2. Experiment to study effect of drug and method of administration
on fasting blood sugar in a random sample of N = 18 diabetic
patients. (dataset on website is blsugar.dat)
Temp
20
20
20
30
30
30
40
40
40
• at three TEMPERATURES (20, 30, 40oC)
• consuming three levels of SUCROSE (20%, 40%, 60%)
1. Entomologist records energy expended (y) by N = 27 honeybees
Two-factor designs
with factors that are fixed/random and nested/crossed
Topic: Mixed Models for factorial experiments
Reading: Rao, Ch.14
Osborne, ST512
190
Transformation?
Before
Washer
Day
1
70070.00
(79034.49)
75890.00
2
(74551.32)
95260.00
3
(03176.00)
Data courtesy
Location
After
After
After
Washer
mic. rinse chill tank
48310.00
12020.00 11790.00
(34166.80) (3807.24) (7832.05)
52020.00
8090.00
8690.00
(17686.27) (4848.01) (5526.19)
33170.00
6200.00
8370.00
(22259.08) (5028.81) (5720.15)
of Michael Bashor, General Mills
4. An expt measures Campylobacter counts in N = 120 chickens in
a processing plant, at four locations, over three days. Means (std)
for n = 10 chickens sampled at each location tabulated below:
Sample
1
2
3
4
Lab
1
2200 3000 210 270
2200 2900 200 260
2600 3600 290 360
2
2500 3500 240 380
1900 2500 160 230
3
2100 2200 200 230
2600 2800 330 350
4
4300 1800 340 290
5
4000 4800 370 500
3900 4800 340 480
(Data from Oehlert, 2000)
3. An experiment is conducted to determine variability among laboratories (interlaboratory differences) in their assessment of bacterial concentration in milk after pasteurization. Milk w/ various
degrees of contamination was tested by randomly drawing four
samples of milk from a collection of cartons at various stages
of spoilage. Y is colony-forming units/µl. Labs think they’re
receiving 8 independent samples
Osborne, ST512
191
3
1
18.3 14.1
18.7 13.8
19.0 14.2
al (1996)
2
2
3
1
19.0 11.9 15.3
18.5 12.4 15.9
18.2 12.0 16.0
3
4
2
3
1
2
3
19.5 16.5 7.3 8.9 11.3
20.1 17.2 7.8 9.4 10.9
19.3 16.9 7.0 9.3 10.5
Treatment
DD
DD
AL
AL
AH
AH
BL
BL
BH
BH
Dark
1
1
0
0
0
0
0
0
0
0
Source
D
D
A
A
A
A
B
B
B
B
Intensity
D
D
L
L
H
H
L
L
H
H
Pot
1
2
1
2
1
2
1
2
1
2
Seedling 1
32.94
34.76
30.55
32.37
31.23
30.62
34.41
34.07
35.61
33.65
Seedling 2
35.98
32.40
32.64
32.04
31.09
30.42
34.88
33.87
35.00
32.91
6. Plantheights from 20 pots randomized to 10 treatment combinations. (See Table 14.2 from Rao.)
Plant i
1
Leaf j
1
2
k = 1 11.2 16.5
k = 2 11.6 16.8
k = 3 12.0 16.1
Data from Neter, et
5. An experiment to assess the variability of a particular acid among
plants and among leaves of plants:
Osborne, ST512
Yijk
Yijk
Yijk
Yijk
Yijk
Yijk
= µ + Ai + Bj + (AB)ij + Eijk crossed/random
=
µ + αi + βj(i) + Eijk
nested/fixed
=
µ + Ai + Bj(i) + Eijk
nested/random
= µ + αi + Bj + (αB)ij + Eijk crossed/mixed
=
µ + αi + Bj(i) + Eijk
nested/mixed
= µ + αi + βj + (αβ)ij + Eijk
crossed/fixed
192
• RANDOM effects are used when it makes sense to think of
LEVELS of factor as random sample from a population.
Recall
• for Model 5, Ai, Bj(i) are all independent
P
• for Model 6,
αi = 0
• for Model 3, Ai, Bi, (AB)ij are all independent
P
• for Model 4,
αi = 0 and Bj , (αB)ij are all independent
• CAPITAL letters represent RANDOM effects
P
P
P
P
• for Model 1,
αi = βj = i(αβ)ij = j (αβ)ij ≡ 0
P
P
• for Model 2,
αi = j βj(i) ≡ 0
• GREEK symbols parameterize FIXED, unknown treatment means
In the models above, (which are not ordered according to the six
prior datasets)
1.
2.
3.
4.
5.
6.
Six types of two-factor models
Fixed and/or random effects that are either crossed or nested
Osborne, ST512
Yijk = µ +
• Model:
• Crossed or nested?
• Fixed or random?
• Second factor:
• First factor:
3. Measuring bacterial concentration in milk
Yijk = µ +
• Model:
• Crossed or nested?
• Fixed or random?
• Second factor:
• First factor:
2. Change in fasting blood sugar for diabetics
Yijk = µ +
• Model:
• Crossed or nested?
• Fixed or random?
• Second factor:
• First factor:
1. Energy expended by honeybees.
+Eijk
+Eijk
+Eijk
Identifying the appropriate model for our 6 examples:
Osborne, ST512
193
+Eijk
+Eijk
194
Yijk = µ +
• First factor:
• Second factor:
• Fixed or random?
• Crossed or nested?
• Model:
+Eijk
6. Effect of light source and intensity on plant heights (Rao Table
14.2)
Yijk = µ +
• First factor:
• Second factor:
• Fixed or random?
• Crossed or nested?
• Model:
5. Acids in leaves of plants
Yijk = µ +
• First factor:
• Second factor:
• Fixed or random?
• Crossed or nested?
• Model:
4. Measuring bacteria counts in chickens at processing plant
Osborne, ST512
Tables of expected means squares (EMS)
(compare w/ OL tables 17.10, 17.17 and 17.31)
195
df
A, B fixed
a−1
σ 2 + nbψA2
b−1
σ 2 + naψB2
2
(a − 1) σ 2 + nψAB
×(b − 1)
ab(n − 1)
σ2
A fixed B random
2
σ 2 + nbψA2 + nσαB
2
σ 2 + naσB2 + nσαB
2
σ 2 + nσαB
σ2
A, B random
2
σ 2 + nbσA2 + nσAB
2
2
σ + naσB2 + nσAB
2
σ 2 + nσAB
σ2
df
A, B fixed
a−1
σ 2 + nbψA2
2
a(b − 1) σ 2 + nψB(A)
ab(n − 1)
σ2
A, B random
2
σ 2 + nbσA2 + nσB(A)
2
σ 2 + nσB(A)
σ2
A fixed B random
2
σ 2 + nbψA2 + nσB(A)
2
σ 2 + nσB(A)
σ2
2. If a factor X is fixed. Treat it like it is random and then just
2
replace the varcomp for X with the effect size, ψX
.
1. If a factor X with index i is random then EM S(X) is a linear
combo of σ 2 and varcomps for all random effects containing index
i. Coefficients for varcomps are limits of indexes NOT listed
(summed over) in random effects.
Help with computing expected mean squares
(without sum-to-zero assumptions on random effects)
where ψ 2 and σ 2 values are defined on the next page.
Source
A
B(A)
Error
When factor B is NESTED in factor A, expected means associated
with sums of squares are given in the table below:
Error
Source
A
B
AB
When factors A and B are CROSSED, and no sum-to-zero assumptions are made on random effects, expected means associated with
sums of squares are given in the table below:
Osborne, ST512
1
1
b
X
a
βi2
αi2
b
a
b
effect size of factor B
effect size of factor A
error variance
variance component for factor B
variance component for interaction
The term effect size is often used in power considerations and sometimes involves division by σ 2.
σ 2 = Var(Eijk))
2
σB(A)
= Var(Bj(i))
= Var((AB)ij )
= Var(Bi)
variance component for factor B
effect size of factor B
σB2
XX
1
=
β2
a(b − 1) i=1 j=1 j(i)
variance component for factor A
2
σAB
196
XX
1
=
(αβ)2ij effect size of interaction
(a − 1)(b − 1) i=1 j=1
1
b−1
1
a−1
a
X
σA2 = Var(Ai)
2
ψB(A)
2
ψAB
ψB2 =
ψA2 =
Osborne, ST512
197
M S[B]
M S[AB]
M S[A]
M S[AB]
M S[AB]
M S[E]
F Value
191.44
1150.89
37.37
2.94
Pr > F
<.0001
<.0001
<.0001
0.0161
1
M S[A] 0.5756
=
= 37.37(p < 0.0001)
M S[E] 0.0154
The correct F -ratio and p-value for testing for random LAB (A)
effect:
M S[A]
0.5756
F =
=
= 12.72(p = 0.0003)
M S[AB] 0.0452
F =
The wrong F -ratio and p-value for testing for random LAB (A) effect:
The SAS System
The GLM Procedure
Dependent Variable: ly = log(y)
Sum of
Source
DF
Squares
Mean Square
Model
19
56.03510844
2.94921623
sample
3
53.18978788
17.72992929
lab
4
2.30248803
0.57562201
sample*lab
12
0.54283253
0.04523604
Error
20
0.30810726
0.01540536
Corrected Total
39
56.34321569
Note the departure from fixed effects analysis, where M S[E] is always used in the denominator.
3. To test for main effect of B, use FB =
2. To test for main effect of A, use FA =
1. To test for interaction effect, use FAB =
Milk example (p.2): F -tests and estimating variance components.
• EM S tables yield estimating equations for variance components
• EM S tables dictate which F -ratios test which effects
Using expected mean squares to analyze data in mixed models
Osborne, ST512
Estimating variance components
198
M S[E]
M S[AB]
M S[A]
M S[B]
=
=
=
=
0.0154
0.0452
0.5756
17.7299
σ̂ 2
2
σ̂AB
σ̂A2
σ̂B2
=
=
=
=
M S[AB]−M S[E]
n
M S[A]−M S[AB]
nb
M S[B]−M S[AB]
na
M S[E]
=
=
=
=
0.0452−0.0154
2
0.5756−0.0452
8
17.7299−0.0452
10
= 0.0154
= 0.01492
= 0.0663
= 1.768
into the system of equations yields estimated variance components:
Substitution of
M S[E] = σ̂ 2
2
M S[AB] = σ̂ 2 + nσ̂AB
2
2
= σ̂ + 2σ̂AB
2
M S[A] = σ̂ 2 + nbσ̂A2 + nσ̂AB
2
= σ̂ 2 + 8σ̂A2 + 2σ̂AB
2
M S[B] = σ̂ 2 + naσ̂B2 + nσ̂AB
2
2
2
= σ̂ + 10σ̂B + 2σ̂AB
The estimated variance components satisfy the following system of
equations:
Osborne, ST512
Type I SS
53.18978788
2.30248803
0.54283253
F Value
1150.89
37.37
2.94
ly Mean
6.815577
Mean Square
17.72992929
0.57562201
0.04523604
Root MSE
0.124118
Pr > F
<.0001
<.0001
0.0161
Pr > F
<.0001
Type III Expected Mean Square
Var(Error) + 2 Var(lab*sample) + 10 Var(sample)
Var(Error) + 2 Var(lab*sample) + 8 Var(lab)
Var(Error) + 2 Var(lab*sample)
DF
3
4
12
Coeff Var
1.821098
F Value
191.44
199
Source
lab
sample
DF
4
3
Type III SS
2.30248803
53.18978788
Mean Square
0.57562201
17.72992929
F Value
12.72
391.94
Pr > F
0.0003
<.0001
Tests of Hypotheses Using the Type III MS for lab*sample as an Error Term
Source
sample
lab
lab*sample
Source
sample
lab
lab*sample
R-Square
0.994532
proc glm;
class lab sample;
model ly=sample|lab;
random sample lab sample*lab;
test h=lab sample e=sample*lab;
lsmeans sample*lab;
run;
The GLM Procedure
Dependent Variable: ly
Sum of
Source
DF
Squares
Mean Square
Model
19
56.03510844
2.94921623
Error
20
0.30810726
0.01540536
Corrected Total
39
56.34321569
data one;
infile "milk.dat" firstobs=4;
input sample lab y;
ly=log(y);
run;
Osborne, ST512
1.76847
0.06630
0.01492
0.01541
ly
200
σ̂ 2 =
=
=
=
µ̂ =
2
σ̂AB
σ̂A2
σ̂B2
0.0154
0.0149
0.0663
1.7685
6.82(log scale)
Var(Ȳ+++) = Var(Ā+) + Var(B̄+) + Var((AB)++) + Var(Ē+++)
σ2
σ2
σ2 σ2
= A + B + AB +
a
b
ab
abn
Ȳ+++ = µ + Ā+ + B̄+ + (AB)++ + Ē+++
• The standard error of Ȳ+++ can be derived by
are
Yijk = µ + Ai + Bj + (AB)ij + Eijk
• The estimated parameters (µ+ variance components) of the model
• There is evidence of variabilility due to laboratory× sample interaction; interlaboratory effects vary by sample.
Q: At the end of the day, what is the conclusion from the analysis of
this crossed, random effects experiment?
Var(sample)
Var(lab)
Var(sample*lab)
Var(Error)
Variance Component
proc varcomp;
class sample lab;
model y=sample|lab;
run;
Variance Components Estimation Procedure
Osborne, ST512
r
201
SE(Ȳ+++) =
σA2 σB2
σ2
σ2
+
+ AB +
a
b
ab
abn
can be estimated by substitution of estimated variance components
(σ̂ 2), which leads to
r
σ̂ 2
σ̂A2 σ̂B2
σ̂ 2
c Ȳ+++) =
+
+ AB +
SE(
a
b
ab
abn
= lots of algebra and cancellations
r
1
(M S[A] + M S[B] − M S[AB])
=
nab
For the milk data, we have
r
1
c
(0.58 + 17.73 − 0.05) = 0.6757
SE(Ȳ+++) =
40
For a 95% confidence interval, we have a problem: we don’t know
how many df are associated with a t statistic based on this estimated
SE.
The standard error
Estimation of standard error and approximation of df
Osborne, ST512
ST511 Flashback:
Unequal variances independent samples t-test:
202
(c1M S1 + c2M S2)2
(c1M S1)2/df1 + (c2M S2)2/df2
where M Si = Si2 and ci = 1/ni.
b =
df
Ȳ1 − Ȳ2 − (µ1 − µ2)
T = p 2
.
S1 /n1 + S22/n2
For small n1, n2, this quantity does not have the standard normal
distribution. nor does the version where Sp2 is used in the denominator. An approximate solution
is to use the student t distribution
with df approximated by the Satterthwaite approximation: Consider the test statistic
H0 : µ1 − µ2 = 0 v. H1 : µ1 − µ2 6= 0
Assumptions:
• Y11, . . . , Y1n1 and Y21, . . . , Y2n2 are independent random samples
from normal distributions with unknown µ1, µ2, σ1, σ2 and σ12 6= σ22.
Note the large difference in the sample variances.
ȳ1 = 133.2, s21 = 26.0
ȳ2 = 92.8, s22 = 195.4
n1 = 11, n2 = 9
Summary statistics:
Example: Suspended particulate matter Y (in micrograms per cubic meter) in homes with smokers (Y1) and without smokers (Y2):
smokers 133 128 136 135 131 131 130 131 131 132 147
no smokers 106 85 84 95 104 79 72 115 95
Osborne, ST512
203
(29.4, 51.4)
40.4 ± 10.97
40.4 ± 2.236(4.91)
This problem aka the Behrens-Fisher problem.
These data would lead to the rejection of H0 : µ1 = µ2 = 0 versus
the two-tailed alternative. The observed test statistic is given by
40.4
133.2 − 92.8
=
tobs = p
= 8.2 (p < 0.0001)
26/11 + 195.4/9 4.91
or
or
or
A 95% confidence interval for the mean difference between homes
with and without a smoking occupant, µ1 − µ2 is given by
p
133.2 − 92.8 ± 2.236 26/11 + 195.4/9
The 97.5th percentile of the t distribution with df = 9.74 is t(0.025, 9.74) =
2.236
2
b = (2.36 + 21.71) = 9.74
df
2
2
2.36
21.71
10 +
8
ST511 Flashback continued:
For the air pollution in homes with a smoking occupant data, c1M S1 =
26/11 = 2.36, c2M S2 = 195.4/9 = 21.71 and
Osborne, ST512
0
1
Equal
Unequal
18
9.74*
DF
13.98
5.0955
10.064
Std Dev
82.032
129.76
-49.91
Variances
T-Tests
9.443
3.5603
7.6046
Lower CL
Std Dev
9
11
N
Method
Folded F
Variable
y
8
Num DF
10
Den DF
Equality of Variances
Pooled
Satterthwaite
Diff (1-2)
y
y
y
y
y
smoke
Variable
0
1
Method
Diff (1-2)
y
y
y
Variable
smoke
Lower CL
Mean
The SAS System
The TTEST Procedure
Statistics
Variable
proc ttest;
class smoke;
var y;
run;
data one;
infile "smokers.dat" firstobs=2;
input y smoke;
label y="suspended particulate matter";
run;
Osborne, ST512
7.53
F Value
-8.93
-8.23*
t Value
26.783
8.9422
14.883
Upper CL
Std Dev
92.778
133.18
-40.4
Mean
0.0045
Pr > F
<.0001
<.0001
Pr > |t|
4.66
1.5363
4.5235
Std Err
103.52
136.6
-30.9
Upper CL
Mean
1
204
205
(log scale)
or
6.82 ± 2.08
6.82 ± 3.08(0.6757)
Using t(0.025, 3.18) = 3.08, a 95% confidence interval for the mean
µ among the population of all labs and samples is given by
The degrees of freedom associated with this linear combination is
approximated by
(0.6757)4
b =
= 3.18
df
1
1
1
( 40
17.73)2/3 + ( 40
0.58)2/4 + ( 40
0.045)2/12
b =
df
(c1M S1 + c2M S2 + · · · + ck M Sk )2
(c1M S1)2/df1 + (c2M S2)2/df2 + · · · + (ck M Sk )2/dfk
Recall that for the milk data, we have
r
1
c
(M S[A] + M S[B] − M S[AB])
SE(Ȳ+++) =
r 40
1
=
(0.58 + 17.73 − 0.05)
40
= 0.6757
(a
linear combination of mean square terms), use the
Satterthwaite
approximation: To approximate the df associated with a t statistic based on a standard error of the form
p
c 1 M S1 + c 2 M S2 + · · · + c k M Sk
The two-way random effects model for milk data
Satterthwaite’s approximation (cont’d)
Osborne, ST512
Effect
Intercept
0.05
0.05
0.05
0.05
Alpha
0.5664
0.02233
0.005761
0.009017
Lower
Standard
Error
0.6757
Lower
4.7325
DF
3.18
Upper
8.8987
t Value
10.09
Solution for Fixed Effects
1.7685
0.06630
0.01492
0.01541
Estimate
Effect
Intercept
Estimate
6.8156
sample
lab
sample*lab
Residual
Cov Parm
24.8486
0.7260
0.09261
0.03213
Upper
Pr > |t|
0.0016
ly
Variance Components
REML
Profile
Model-Based
Satterthwaite
Covariance Parameter Estimates
Dependent Variable
Covariance Structure
Estimation Method
Residual Variance Method
Fixed Effects SE Method
Degrees of Freedom Method
proc mixed cl;
class sample lab;
model ly=/s ddfm=satterth cl;
random sample lab sample*lab;
run;
The SAS System
The Mixed Procedure
Model Information
data one;
infile "milk.dat" firstobs=4;
input sample lab y;
ly=log(y);
run;
Osborne, ST512
Alpha
0.05
1
206
1.5
1.5
2.0
2.0
2.5
sample
2.5
log transformed data
sample
Lab 1
Lab 2
Lab 3
Lab 4
Lab 5
untransformed data
3.0
3.0
3.5
3.5
plot(x=sample,y=log(bacteria),pch=lab)
title("log transformed data")
postscript()
milk.data <- read.table("milk.dat",skip=3,col.names=c("sample","lab","bacteria"))
attach(milk.data)
postscript(file="milkplot1.ps")
par(mfrow=c(2,1)) # A 2x1 template (for two plots in a single column)
plot(x=sample,y=bacteria,pch=lab)
title("untransformed data")
legend(2.5,4400,pch=1:5,legend=c("Lab 1","Lab 2","Lab 3","Lab 4","Lab 5"))
1.0
1.0
Osborne, ST512
bacteria
log(bacteria)
3000
0 1000
8.0
7.0
6.0
5.0
4.0
4.0
207
Osborne, ST512
mean of y
2000 4000
0
mean of log(y)
6 7 8
1
1
3
lab
4
5
2
3
lab
4
5
sample
Interaction plot for milk - log(counts)
2
sample
Interaction plot for milk - raw counts
2
1
4
3
2
1
4
3
208
A nested design
209
Mean Variance Mean
ȳj(i)
s2j(i)
ȳ+(i)
15.7
6.3
17.7
19.7
9.3
20
1
18.7
17.3
6.3
28
4
30.5
33
9
Definition: Factor B is nested in factor A if there is a new set of
levels of factor B for every different level of factor A.
(This is exercise 13.35. Grand mean is ȳ+++ = 22.3.)
Type of
Administration (j)
30mg × 1
15mg × 2
Brand II tablet 20mg × 1
10mg × 2
Insulin injection before breakfast
before supper
Drug
(i)
Brand I tablet
• Second factor is type of administration (see table)
• First factor is drug: brand I tablet, brand II tablet, insulin injection
Experiment to study effect of drug and method of administration on
fasting blood sugar in diabetic patients
Osborne, ST512
210
k
i
j
k
j
k
i
j
k
XXX
i
XXX
(yijk − ȳij+)2
(ȳij+ − ȳi++)2
(ȳi++ − ȳ+++)2
The ANOVA table looks like
Sum of
Mean
Source
d.f.
squares
Square
F
SS[T ]
M S[A]
A
a−1
SS[A]
M S[A] = (a−1)
FA = M
S[E]
P
SS[B(A)]
M S[B(A)]
P
B(A)
i (bi − 1) SS[B(A)] M S[B(A)] =
(bi −1) FB(A) = M S[E]
P
SS[E]
Error N − bi
SS[E]
M S[E] = (N
−t)
Total
N −1
SS[T OT ]
P
If b1 = b2 = · · · ba = b then (bi − 1) = a(b − 1) and dfE = ab(n − 1).
SS[E] =
SS[B(A)] =
SS[A] =
j
XXX
i
SS[T ot] can be broken down into components reflecting variability
due to A, B(A) and variability not due to either factor (SS[E]):
SS[T ot] = SS[A] + SS[B(A)] + SS[E]
XXX
SS[T ot] =
(yijk − ȳ+++)2
for i = 1, 2, . . . , a, j = 1, 2, . . . , bi, k = 1, 2, . . . , n
Yijk = µ + αi + βj(i) + Eijk
Analysis of variance in nested designs
Consider a two-factor design in which factor B is nested in factor
A. Let Yijk denote the k th response at level j of factor B within level
i of factor A. A model:
Osborne, ST512
Inference from nested designs
211
Mean Variance Mean
ȳ+(i)
ȳj(i)
s2j(i)
15.7
6.3
17.7
19.7
9.3
20
1
18.7
17.3
6.3
28
4
30.5
33
9
Q1: How many df associated with SS[A]?
Q2: How many df associated with SS[B(A)]?
Q3: How many df associated with SS[E]?
SS[A] = 2(3)[(17.7 − 22.3)2 + (18.7 − 22.3)2 + (30.5 − 22.3)2
= 611.4
SS[B(A)] = 3[(15.7 − 17.7)2 + (19.7 − 17.7)2 + (20.0 − 18.7)2
+(17.3 − 18.7)2 + (28 − 30.5)2 + (33 − 30.5)2]
= 72.2
SS[E] = 72
Type of
Administration (j)
30mg × 1
15mg × 2
Brand II tablet 20mg × 1
10mg × 2
Insulin injection before breakfast
before supper
Drug
(i)
Brand I tablet
For the diabetics blood sugar data, with ȳ+++ = 22.3 and means
To test H0 : αi ≡ 0, use FA on a − 1 and dfE degrees of freedom.
P
To test H0 : βj(i) ≡ 0, for all i, j, use FB(A) on (bi − 1) and dfE
degrees of freedom.
Osborne, ST512
effect of B within A=1
effect of B within A=2
effect of B within A=3
A=1 mean - A=2 mean
A=1 mean - A=3 mean
A=2 mean - A=3 mean
Parameter
Source
a
b(a)
R-Square
0.904713
Source
Model
Error
Corrected Total
Estimate
Type I SS
611.4444444
72.1666667
4.0000000
-2.6666667
5.0000000
-1.0000000
-12.8333333
-11.8333333
DF
2
3
2.00000000
2.00000000
2.00000000
1.41421356
1.41421356
1.41421356
2.00
-1.33
2.50
-0.71
-9.07
-8.37
Pr > F
<.0001
0.0344
Pr > F
<.0001
212
0.0687
0.2072
0.0279
0.4930
<.0001
<.0001
Pr > |t|
F Value
50.95
4.01
t Value
Mean Square
305.7222222
24.0555556
Standard
Error
F Value
22.79
y Mean
22.27778
Mean Square
136.7222222
6.0000000
Root MSE
2.449490
Sum of
Squares
683.6111111
72.0000000
755.6111111
Coeff Var
10.99522
DF
5
12
17
The GLM Procedure
proc glm;
class a b;
model y=a b(a);
output out=two p=p r=r;
means a b(a)/lsd;
estimate "effect of B within A=1" b(a) -1 1;
estimate "effect of B within A=2" b(a) 0 0 -1 1;
estimate "effect of B within A=3" b(a) 0 0 0 0 -1 1;
estimate "A=1 mean - A=2 mean" a 1 -1;
estimate "A=1 mean - A=3 mean" a 1 0 -1;
estimate "A=2 mean - A=3 mean" a 0 1 -1;
run;
data one;
infile "blsugar.dat" firstobs=2 dlm=’09’x;
input a b rep y;
drug=a; admin=b;
run;
Osborne, ST512
213
• The following contrasts may be of interest:
1
θ1 = µ1(3) − (µ1(1) + µ2(1) + µ1(2) + µ2(2))
4
1
θ2 = µ2(3) − (µ1(1) + µ2(1) + µ1(2) + µ2(2))
4
Exercise: Estimate them and test their significance (H0 : θi = 0).
• The effect of type of drug (factor A) is highly significant (p <
0.0001). Unadjusted pairwise comparisons indicate that the insulin injections yield greater changes, on average, in blood sugar
than either pill and the mean changes brought by the pills don’t
differ significantly.
with an (estimated) standard error of SE = 2 =?.
ȳ32+ − ȳ31+ = 5mg/dl
• The administration effect B (nested in the type of drug effect A)
is statistically significant (p = 0.0344). This is due mostly to the
before breakfast/supper difference, which is estimated to be
Conclusions?
Osborne, ST512
More Two-factor mixed models (Ch. 14)
214
Location
After
After
After
Washer
mic. rinse chill tank
48310.00
12020.00 11790.00
(34166.80) (3807.24) (7832.05)
52020.00
8090.00
8690.00
(17686.27) (4848.01) (5526.19)
33170.00
6200.00
8370.00
(22259.08) (5028.81) (5720.15)
of Michael Bashor, General Mills
1
2
3
1
16.5 18.3 14.1
16.8 18.7 13.8
16.1 19.0 14.2
2
2
3
1
19.0 11.9 15.3
18.5 12.4 15.9
18.2 12.0 16.0
3
4
2
3
1
2
3
19.5 16.5 7.3 8.9 11.3
20.1 17.2 7.8 9.4 10.9
19.3 16.9 7.0 9.3 10.5
Treatment
DD
DD
AL
AL
AH
AH
BL
BL
BH
BH
Dark
1
1
0
0
0
0
0
0
0
0
Source
D
D
A
A
A
A
B
B
B
B
Intensity
D
D
L
L
H
H
L
L
H
H
Pot
1
2
1
2
1
2
1
2
1
2
Seedling 1
32.94
34.76
30.55
32.37
31.23
30.62
34.41
34.07
35.61
33.65
Seedling 2
35.98
32.40
32.64
32.04
31.09
30.42
34.88
33.87
35.00
32.91
• Study of light source and intensity on plant height.
Plant i
Leaf j
1
k = 1 11.2
k = 2 11.6
k = 3 12.0
• An experiment to assess the variability of a particular acid among
plants and among leaves of plants:
Before
Washer
Day
1
70070.00
(79034.49)
75890.00
2
(74551.32)
3
95260.00
(03176.00)
Data courtesy
– Crossed design with two factors
∗ Location (4 levels)
∗ Day (3 levels)
– 4 × 3 layout, n = 10 chickens per combo
• Expt measures campylobacter counts in N = 120 chickens in a
processing plant
Osborne, ST512
215
-2
-1
0
1
2
-100000
0
100000
200000
r
300000
0
9
30000
1
40000
p
10
2
50000
60000
3
70000
11
4
80000
1
2
3
predicted_log
location
8
20000
location
10000
4
12
90000 100000
Analysis of Campylobacter counts on chickens data
Residual plots (resid .vs ŷ) for bacteria counts, after fitting two factor
fixed effects models (similar plots for mixed models):
Osborne, ST512
residual_log
ly=log(y);
216
proc mixed method=type3; * to get ANOVA table with EMS terms;
*proc mixed cl; * to get asymmetric confidence intervals ;
class day location;
model ly=location/ddfm=satterth;
random day day*location;
lsmeans location/adj=tukey;
run;
proc gplot data=two;
title "residuals versus predicted";
plot residual*predicted=location/haxis=axis1 vaxis=axis2 legend=legend1;
plot residual_log*predicted_log=location/haxis=axis1 vaxis=axis2 legend=legend1;
run; */
proc mixed method=type3 cl;
class day location;
model ly=location/ddfm=satterth outp=predz;
random day day*location;
lsmeans location/adj=tukey;
run;
/*proc glm;
* the old way of doing things, before PROC MIXED ;
class day location;
model ly=day|location;
random day day*location;
test h=location e=day*method;
lsmeans location/pdiff;
*wrong;
run;*/
axis1 offset=(1,1) label=(height=3);
axis2 offset=(1,1) label=(height=3 angle=90);
legend1 label=(height=2);
proc glm;
class day location;
model y ly=location|day;
output out=two r=residual residual_log p=predicted predicted_log;
run;
/*
symbol1 value=dot color=black;
symbol2 value=square color=black;
symbol3 value=triangle color=black; symbol4 value=diamond color=black;
data one;
/* Bashor data */
infile "bashor.dat" firstobs=3;
input day location y;
run;
Osborne, ST512
217
Values
1 2 3
1 2 3 4
0.548657
0.755594
1.393677
Mean Square
32.621796
Expected Mean Square
Var(Residual) + 10
Var(day*location) + Q(location)
Var(Residual) + 10
Var(day*location) + 40 Var(day)
Var(Residual) + 10
Var(day*location)
Var(Residual)
Type 3 Analysis of Variance
59.254946
4.533565
2.787355
Sum of
Squares
97.865388
Type 3 Analysis of Variance
Levels
3
4
Error
Source
Error Term
DF
F Value
Pr > F
location
MS(day*location)
6
43.17
0.0002
day
MS(day*location)
6
1.84
0.2375
day*location MS(Residual)
108
1.38
0.2303
-------------------------------------------------------------------------------------
108
6
day*location
Residual
2
DF
3
day
Source
location
Class
day
location
WORK.ONE
ly
Variance Components
Type 3
Factor
Model-Based
Satterthwaite
Class Level Information
Data Set
Dependent Variable
Covariance Structure
Estimation Method
Residual Variance Method
Fixed Effects SE Method
Degrees of Freedom Method
------------------------------------------------------------------------------------The SAS System
1
The Mixed Procedure
Model Information
Osborne, ST512
Effect
location
location
location
location
location
location
Den
DF
6
Estimate
10.8870
10.4953
8.8745
8.9394
Standard
Error
0.1747
0.1747
0.1747
0.1747
DF
7.33
7.33
7.33
7.33
F Value
43.17
Least Squares Means
Num
DF
3
_location
2
3
4
3
4
4
Estimate
0.3917
2.0125
1.9476
1.6208
1.5559
-0.06488
Standard
Error
0.2244
0.2244
0.2244
0.2244
0.2244
0.2244
DF
6
6
6
6
6
6
t Value
1.75
8.97
8.68
7.22
6.93
-0.29
Pr > |t|
0.1316
0.0001
0.0001
0.0004
0.0004
0.7823
Pr > |t|
<.0001
<.0001
<.0001
<.0001
1156981
145734
0.7303
Upper
t Value
62.33
60.09
50.81
51.18
Pr > F
0.0002
0.002071
0.002844
0.4274
Differences of Least Squares Means
location
1
2
3
4
Effect
location
location
1
1
1
2
2
3
Effect
location
location
location
location
0.05
0.05
0.05
Type 3 Tests of Fixed Effects
0.01595
0.02069
0.5487
(generated by 2nd run of PROC MIXED)
Estimate
Alpha
Lower
day
day*location
Residual
Cov Parm
218
Effect
location
location
location
location
location
location
location
1
1
1
2
2
3
_location
2
3
4
3
4
4
Adjustment
Tukey-Kramer
Tukey-Kramer
Tukey-Kramer
Tukey-Kramer
Tukey-Kramer
Tukey-Kramer
Adj P
0.3801
0.0004
0.0005
0.0015
0.0018
0.9907
------------------------------------------------------------------------------------Differences of Least Squares Means
*
*
*
*
*
Osborne, ST512
219
day
2
3
1
2
location
3
Yijk = µ + αi + Bj + (αB)ij + Eijk
1
2
w/ variance components σB2 , σαB
, σ2.
Model
8
9
10
11
4
Theory for mixed/crossed model used to analyze Campylobacter data
Discussion of MIXED output
Osborne, ST512
Predicted
Campylobacter analysis, continued
FAB =
M S[AB] 0.76
=
= 1.38
M S[E]
0.55
= 0, use
220
2
σ̂αB
=
σ̂ 2 = M S[E] = 0.55
M S[AB] − M S[E]
2
σ̂αB
=
n
0.76 − 0.55
= 0.021
=
10
M S[B] − M S[AB]
σ̂B2 =
na
1.39 − 0.76
= 0.016
=
40
M S[AB] − M S[E] 0.76 − 0.55
=
= 0.021
n
10
Intepretation: there is no evidence that day-to-day variability varies
by location. The estimated variance component is itself very small.
on (a − 1)(b − 1) = 6 and ab(n − 1) = 108 df . The p-value is 0.2303.
providing no evidence of a random day × location interaction effect.
The variance component for this random effect is estimated by
To test H0 :
2
σαB
Fixed Factor A: location
Random Factor B: day
Osborne, ST512
Implied correlation structure
221
Cov(Yijk1 , Yijk2 )
2
σ 2 + σB2 + σαB
Cov(Bi, Bi) + Cov((αB)ij , (αB)ij )
2
σ 2 + σB2 + σαB
2
2
σB + σαB
2
σ 2 + σB2 + σαB
Cov(Y1jk1 , Y2jk2 )
2
σ 2 + σB2 + σαB
Cov(Bi, Bi)
2
2
σ + σB2 + σαB
2
σB
2
2
σ + σB2 + σαB
0.016+0.021
0.016+0.021+0.55
0.016
0.016+0.021+0.55
=
=
0.037
.587
0.016
.587
= 0.027
= 0.063
What about the correlation of two observations on different days?
Which is which?
•
•
Estimates of these correlations are
=
=
Corr(Y1jk1 , Y2jk2 ) =
=
=
Corr(Yijk1 , Yijk2 ) =
Recall that Yijk = µ + αi + Bj + (αB)ij + Eijk .
• at different locations?
• at the same location?
What is the correlation of two observations taken on the same day
Osborne, ST512
32.6
M S[A]
=
= 43.2
FA =
M S[AB] 0.76
on a − 1 = 3 and (a − 1)(b − 1) = 6 df , which is significant (p =
0.0002).
Var(Ȳ4++ − Ȳ3++) 6= σ 2(
1
1
+ )
nb nb
Ȳ2++ − Ȳ1++
α2 + B̄ + αB 2+ + E 2++
(α1 + B̄ + αB 1+ + E 1++)
α2 − α1 + αB 2+ − αB 1+ + E 2++ − E 1++
Var(θ̂) = Var(αB 2+) + Var(αB 1+) + Var(E 2++) + Var(E 1++)
σ2
σ2
= 2 αB + 2
b
nb
2 2
2
(σ + nσαB
=
)
nb
which can be estimated nicely on (a − 1)(b − 1) = 6df by
ˆ θ̂) = 2 M S[AB]
Var(
nb
for the chickens, where y 4++ − y 3++ = −0.06 the SE is
r
q
2
c
Var(θ̂) =
0.76 = 0.22
3 ∗ 10
Since t(0.025, 6) = 2.45, a 95% c.i. for θ given by −0.06±2.45(0.22).
which has variance
θ̂ =
=
−
=
What is SE(θ̂) and how can it be estimated?
(Why not?)
Note that
θ̂ = ȳ4++ − ȳ3++ = 8.940 − 8.875 = −0.065
To estimate the a pairwise comparison among location means, such
as, α4 − α3, consider
223
Consider testing for a fixed effect of location. That is, test the hypothesis that average bacteria counts are constant across the locations,
H0 : α1 = α2 = α3 = α4 = 0
Osborne, ST512
Campylobacter analysis, continued
222
Some analysis of fixed effects
Osborne, ST512
Campylobacter analysis, continued
224
1
1202
((4−1)0.76)2
6
0.1754
2
+ 1.39
2
= 7.33
with dfAB = 6, dfB = 2. Since t(0.025, 7.33) = 2.34, a 95% c.i. for
the population mean of location 1, for example, is 10.9 ± 2.34(0.175) .
ˆ =
df
but the df must be approximated using the Satterthwaite approach
Y i++ = µ+ αi + B + αB i+ + E i++
Var(Y i++) = Var(B) + Var(αB i+) + Var(E i++)
σB2
σ2
σ2
=
+ αB +
b
b
nb
1
2
2
(nσB + nσαB
=
+ σ2)
nb
estimated by
1
2
c i++) =
(nσ̂B2 + nσ̂αB
+ σ̂ 2)
Var(Y
nb
=
algebra yields a linear combo of multiple EMS terms
1
=
{(a − 1)EM S[AB] + EM S[B]}
nab
The standard error is estimated easily enough:
r
1
c i++) =
{(a − 1)M S[AB] + M S[B]}
SE(Y
nab
r
1
{(4 − 1)0.76 + 1.39}
=
√ 120
= 0.03 = 0.175
Reporting standard errors for sample means of levels of fixed factor,
like LOCATION means, is a little messier:
Osborne, ST512
225
proc gplot;
title "plant acids";
plot y*plant=leaf/haxis=axis1;
run;
symbol1 value=dot h=1.5;
symbol2 value=diamond h=1.5;
symbol3 value=plus h=1.5;
axis1 value=(h=2) offset=(10);
goptions colors=(black) dev=pslepsf;
*goptions colors=(black);
proc mixed cl method=type3;
*proc mixed cl;
class plant leaf;
model y=/s cl;
random plant leaf(plant);
run;
data one;
infile "plantacid.dat";
input y plant leaf rep;
run;
options ls=75 nodate;
SAS code to fit two-factor random effects model for plant acid data
Nested or crossed?
Osborne, ST512
4
3
plant
leaf
1 2 3 4
1 2 3
Values
plant
leaf(plant)
Residual
Cov Parm
plant
leaf(plant)
Residual
Cov Parm
leaf(plant)
Residual
4.88
185.39
F Value
0.0324
<.0001
Pr > F
0.05
0.05
0.05
Alpha
-10.3930
0.1142
0.07706
Lower
10.1068
7.7684
0.1264
Estimate
0.05
0.05
0.05
Alpha
2.6599
3.5322
0.07706
Lower
/*Covariance Parameter Estimates*/
10.1068
7.7684
0.1264
Estimate
499.70
28.7787
0.2446
Upper
30.6066
15.4227
0.2446
Upper
MS(Residual)
.
MS(leaf(plant))
Error Term
114.392963
23.431667
0.126389
Mean Square
Covariance Parameter Estimates
plant
leaf(plant)
Source
Var(Residual) + 3 Var(leaf(plant))
+ 9 Var(plant)
Var(Residual) + 3 Var(leaf(plant))
Var(Residual)
343.178889
187.453333
3.033333
plant
3
8
24
plant
leaf(plant)
Residual
Expected Mean Square
DF
Source
Sum of
Squares
Type 3 Analysis of Variance
Levels
Class
The Mixed Procedure
Class Level Information
Source
Osborne, ST512
24
.
8
Error
DF
226
Intercept
Effect
Osborne, ST512
y
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Intercept
1
3
DF
8.00
t Value
2
leaf
1
plant
2
plant acids
8.5882
Lower
3
3
19.9341
Upper
Solution for Fixed Effects
1.7826
Standard
Error
Effect
14.2611
Estimate
Solution for Fixed Effects
4
0.0041
Pr > |t|
0.05
Alpha
227
228
σ̂ 2 = M S[E] = 0.13
M S[B(A)] − M S[E]
2
σ̂B(A)
=
n
23.4 − 0.13
= 7.8
=
3
M S[A] − M S[B(A)]
σ̂A2 =
nb
114.4 − 23.4
=
= 10.1
9
So there is some evidence of both a random plant effect and a random leaf effect, nested in plant. The magnitudes of these effects
are quantified by the estimated variance components. The statistical
significance addressed by the p-values.
To test for random effect of factor A (plant), H0 : σA2 = 0,
114.4
M S[A]
=
= 4.88
F =
M S[B(A)]
23.4
on a − 1 = 3 and (b − 1)a = 8df with p = 0.0324.
Reminder: Watch that denominator M S!
M S[B(A)] 23.4
=
= 185.4
F =
M S[E]
0.13
on (b − 1)a = 8 and (n − 1)ab = 24 df (p-value < 0.0001).
2
To test for random effect of nested factor B (leaf), H0 : σB(A)
= 0,
2
w/ variance components σ 2, σA2 , σB(A)
.
Yijk = µ + Ai + Bj(i) + Eijk
Random, nested model
Discussion of MIXED output and analysis of plant acid data
Osborne, ST512
229
2
σA2 + σB(A)
σ2
10.1+7.8
10.1+7.8+0.13
10.1
10.1+7.8+0.13
=
=
17.9
18.0
10.1
18.0
= 0.56
= 0.99
This means that two measurements taken on the same leaf are almost
perfectly correlated. Almost all the variation in any measurement can
be explained by the leaf and plant effects.
•
•
σA2
σA2
2
+
+ σB(A)
Estimates of these correlations are
=
2
σ 2 + σA2 + σB(A)
Cov(Yij1k1 , Y2j2k2 )
Corr(Yij1k1 , Yij2k2 ) = 2
2
σ + σA2 + σB(A)
Cov(Ai, Ai)
= 2
2
σ + σA2 + σB(A)
=
Cov(Yijk1 , Yijk2 )
2
σ 2 + σA2 + σB(A)
Cov(Ai, Ai) + Cov(Bj(i), Bj(i))
=
2
σ 2 + σA2 + σB(A)
Corr(Yijk1 , Yijk2 ) =
Recall that Yijk = µ + Ai + Bj(i) + Eijk .
• and different leaves?
• and the same leaf?
Implied correlation structure for plant acids
What is the correlation of two observations taken from the same
plant
Osborne, ST512
Treatment
DD
DD
AL
AL
AH
AH
BL
BL
BH
BH
Dark
1
1
0
0
0
0
0
0
0
0
Source
D
D
A
A
A
A
B
B
B
B
Intensity
D
D
L
L
H
H
L
L
H
H
Pot
1
2
1
2
1
2
1
2
1
2
Seedling 1
32.94
34.76
30.55
32.37
31.23
30.62
34.41
34.07
35.61
33.65
Seedling 2
35.98
32.40
32.64
32.04
31.09
30.42
34.88
33.87
35.00
32.91
30
31
32
33
34
35
36
y
AH
plant
AL
pot
heights
BH
1
2
light
treatment
and
BL
DD
treatments
• experimental units are 10 pots (points on graph).
• treatments are light sources, intensities,
• Response (y) is seedling height,
Osborne, ST512
230
Yijk = µ + αi + Pj(i) + Eijk
Experiment with light treatments on seedlings
iid
Eijk ∼ N (0, σ 2) (Pj(i) ⊥ Eijk )
231
proc mixed method=type3 cl;
*proc mixed data=planthts cl;
class pot treatment;
model y=treatment;
random pot(treatment);
*lsmeans treatment/diffs adj=tukey;
lsmeans treatment/diffs;
estimate "main effect of source" treatment 1 1 -1 -1/divisor=2;
estimate "main effect of intensity" treatment 1 -1 1 -1/divisor=2;
estimate "interaction " treatment 1 -1 -1 1;
contrast "main effect of source" treatment 1 1 -1 -1;
contrast "main effect of intensity" treatment 1 -1 1 -1;
contrast "interaction " treatment 1 -1 -1 1;
run;
For treatment effects, use M S(P ot(treatments)) as error term.
For example, for H0 : α1 = α2 = · · · = 0, use
M S(treatment)
∼ F5−1,5(2−1) or F4,5
F =
M S(Pot(treatment))
Be careful not to use
M S(treatment)
F =
M S(E)
For these data, we get
10.27
F =
= 8.4(df = 4, 5, p = .0192)
1.22
providing evidence of a treatment effect on plant heights. SAS code
to fixed the mixed effects model (output follows):
iid
Pj(i) ∼ N (0, σP2 ),
αi - treatment effects for i = 1, 2, 3, 4, 5
Pj(i) - pot effects, nested in treatments, j = 1, 2 for each i.
Eijk - seedling/experimental errors, k = 1, 2
Osborne, ST512
Pr > F
0.0192
0.3793
.
Label
main effect of source
main effect of intensity
interaction
Estimate
-2.9300
-0.5375
-1.0450
Num
DF
1
1
1
0.008606
0.5011
Lower
-0.7831
0.5011
Den
DF
5
5
5
F Value
28.09
0.95
0.89
Standard
Error
0.5528
0.5528
1.1057
Estimates
0.05
0.05
Alpha
0.05
0.05
Contrasts
0.09802
1.0264
/*
pot(treatment)
Residual
*/
Label
main effect of source
main effect of intensity
interaction
Estimate
0.09802
1.0264
Covariance Parameter Estimates
F Value
8.40
1.19
.
Upper
0.9792
3.1612
MS(Residual)
.
t Value
-5.30
-0.97
-0.95
Pr > F
0.0032
0.3756
0.3880
DF
5
5
5
Pr > |t|
0.0032
0.3756
0.3880
Error Term
MS(pot(treatment))
3.993E31
3.1612
Expected Mean Square
Var(Residual) + 2 Var(pot(treatment))
+ Q(treatment)
Var(Residual) + 2 Var(pot(treatment))
Var(Residual)
Cov Parm
pot(treatment)
Residual
Source
treatment
pot(treatment)
Residual
Mean Square
10.270192
1.222470
1.026420
Type 3 Analysis of Variance
Sum of
Squares
41.080770
6.112350
10.264200
Type 3 Analysis of Variance
pot(treatment)
Residual
Source
treatment
DF
4
5
10
Levels
Values
2
1 2
5
AH AL BH BL DD
Type 3 Analysis of Variance
Source
treatment
pot(treatment)
Residual
Class
pot
treatment
The SAS System
The Mixed Procedure
Class Level Information
Osborne, ST512
10
.
Error
DF
5
1
232
treatment
AH
AH
AH
AH
AL
AL
AL
BH
BH
BL
treatment
AH
AL
BH
BL
DD
Standard
Error
0.5528
0.5528
0.5528
0.5528
0.5528
DF
5
5
5
5
5
t Value
55.79
57.70
62.03
62.06
61.54
_treatment
AL
BH
BL
DD
BH
BL
DD
BL
DD
DD
Estimate
-1.0600
-3.4525
-3.4675
-3.1800
-2.3925
-2.4075
-2.1200
-0.01500
0.2725
0.2875
Standard
Error
0.7818
0.7818
0.7818
0.7818
0.7818
0.7818
0.7818
0.7818
0.7818
0.7818
Differences of Least Squares Means
Estimate
30.8400
31.9000
34.2925
34.3075
34.0200
DF
5
5
5
5
5
5
5
5
5
5
t Value
-1.36
-4.42
-4.44
-4.07
-3.06
-3.08
-2.71
-0.02
0.35
0.37
Pr > |t|
<.0001
<.0001
<.0001
<.0001
<.0001
Pr > |t|
0.2332
0.0069
0.0068
0.0097
0.0281
0.0275
0.0422
0.9854
0.7416
0.7281
233
(In practice, if all 10 pairwise comparisons were of interest, Tukey’s
adjustment can be considered. Here, the output was too wide for the
page, and there was greater interest in factorial effects (source, light,
interaction), anyway.)
Effect
treatment
treatment
treatment
treatment
treatment
treatment
treatment
treatment
treatment
treatment
Effect
treatment
treatment
treatment
treatment
treatment
Least Squares Means
Output for plant heights and light sources, cont’d
Osborne, ST512
234
5
pot(sour*inten*dark)
Cov Parm
pot(sour*inten*dark)
Residual
10.264200
6.112350
1.092025
1.155625
34.339600
Estimate
0.09802
1.0264
1.026420
1.222470
1.092025
1.155625
34.339600
Mean Square
4.493520
Error Term
MS(pot(sour*inten*dark))
MS(pot(sour*inten*dark))
MS(pot(sour*inten*dark))
MS(pot(sour*inten*dark))
MS(Residual)
.
Covariance Parameter Estimates
Source
dark
source(dark)
intensity(dark)
source*intensi(dark)
pot(sour*inten*dark)
Residual
10
1
source*intensi(dark)
Residual
1
intensity(dark)
DF
1
Sum of
Squares
4.493520
1
Error
DF
5
5
5
5
10
.
F Value
3.68
28.09
0.95
0.89
1.19
.
Pr > F
0.1133
0.0032
0.3756
0.3880
0.3793
.
Expected Mean Square
Var(Residual) + 2
Var(pot(sour*inten*dark)) +
Q(dark,source(dark),intensity
(dark),source*intensi(dark))
Var(Residual) + 2
Var(pot(sour*inten*dark)) +
Q(source(dark),source*intensi(dark))
Var(Residual) + 2
Var(pot(sour*inten*dark)) +
Q(intensity(dark),source*
intensi(dark))
Var(Residual) + 2
Var(pot(sour*inten*dark))
+ Q(source*intensi(dark))
Var(Residual) + 2
Var(pot(sour*inten*dark))
Var(Residual)
Values
1 2
AH AL BH BL DD
A B D
D H L
0 1
Type 3 Analysis of Variance
1
Levels
2
5
3
3
2
source(dark)
Source
dark
Class
pot
treatment
source
intensity
dark
proc mixed method=type3;
class pot treatment source intensity dark;
model y=dark source(dark) intensity(dark) source*intensity(dark) dark ;
random pot(source*intensity*dark);
lsmeans dark source(dark) intensity(dark) source*intensity(dark);
run;
The SAS System
The Mixed Procedure
Using nested factorial effects to get SAS to produce appropriate contrast sums
of squares for factorial effects analysis of plant height and light source data
Osborne, ST512
A
A
B
B
D
A
B
D
source
H
L
D
H
L
H
L
D
intensity
dark
0
1
0
0
1
0
0
1
0
0
0
0
1
Estimate
32.8350
34.0200
31.3700
34.3000
34.0200
32.5662
33.1038
34.0200
30.8400
31.9000
34.2925
34.3075
34.0200
Least Squares Means
Standard
Error
0.2764
0.5528
0.3909
0.3909
0.5528
0.3909
0.3909
0.5528
0.5528
0.5528
0.5528
0.5528
0.5528
DF
5
5
5
5
5
5
5
5
5
5
5
5
5
t Value
118.79
61.54
80.25
87.74
61.54
83.31
84.68
61.54
55.79
57.70
62.03
62.06
61.54
Pr > |t|
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
235
1.09
M S(interaction(dark))
=
= .89(p = .3880)
M S(Pot(dark*source*intensity)) 1.22
d ijklm , Yijklm ) =
Corr(Y
1
2
σ̂ 2
+
σ̂P2 (T )
σ̂P2 (T )
=
.098
= .088
.098 + 1.02
M S(pot(treatment)) − M S(E) 1.22 − 1.02
b)
=
= 0.098(df = df
2
2
Correlation structure? Intrapot correlation?
σ̂P2 (T ) =
σ̂ 2 = M S(E) = 1.02(df = 10)
Degrees of freedom: (df =?, ?)
Estimation of variance components:
F =
For treatment effects, use M S(P ot(treatments)) as the error term.
For example, for H0 : γjk ≡ 0 (intensity effect is constant across light
types), use
Yijklm = µ + δi + αj(i) + βk(i) + (αβ)jk(i) + Pl(ijk) + Eijklm
Inference for light effects
Model for treatment combination “ijk” and pot l, seedling m:
Effect
dark
dark
source(dark)
source(dark)
source(dark)
intensity(dark)
intensity(dark)
intensity(dark)
source*intensi(dark)
source*intensi(dark)
source*intensi(dark)
source*intensi(dark)
source*intensi(dark)
Osborne, ST512
236
(This is a good place to go back and look at RBD expt w/ random block effects. The experiment measured assembly times for iv
injection systems.)
Osborne, ST512
237
1
1
2
2
3
3
1
2
1
2
1
2
53.4
46.5
54.3
57.2
55.9
57.4
y
Source
df
A : Pesticide
a−1=2
Error or “plots” (n − 1)a = (2 − 1)3 = 3
Total
an − 1 = 5
One-way ANOVA for A effect and plot effects:
pest
plot
• Plots, n = 2 per level of A, total of na = 6 plots
For the moment, ignore B, or fix the level of B.
• B : Irrigation treatment, b = 4 levels (called ‘treatment, trt’)
• A : Pesticide treatment, a = 3 levels
Factors:
Consider an experiment to study effects of irrigation and aerially
sprayed pesticide on yields of different varieties of corn.
Repeated measures models
Topic: Split-plots: a repeated measures design
Reading: Rao, Ch. 16.
Osborne, ST512
238
1
2
1
2
1
2
1
1
2
2
3
3
53.4
46.5
54.3
57.2
55.9
57.4
B1
53.8
51.1
56.3
56.9
58.6
60.2
B2
58.2
49.2
60.4
61.6
62.4
57.2
B3
59.5
51.3
64.5
66.8
64.5
62.7
B4
Suggestion: draw a picture of the layout.
• Plots are ‘subjects’ in repeated measures terminology, where time
is often the within subjects factor.
• B - a within plots or within subjects factor
• A - a between plots or between subjects factor
Source
df
B: treatments
b−1=3
A×B
(a − 1)(b − 1) = 6
B×plot(A)
(b − 1)(n − 1)a = 9
aka Subplot error
The ANOVA table on the preceding page is at the whole plot level.
Sources of variation for the split-plot level:
Each row corresponds to one of na = 6 plots. Each plot is divided
into b = 4 subplots and levels of factor B are assigned to these at
random.
plot
pest
Split-plot design
Levels of factor B are randomly assigned to b = 4 subplots within
each of the na = 6 plots in a split-plot design:
Osborne, ST512
239
Size effects for fixed factors same as in prior 2-factor models. For
our example, αi denote pesticide effects, βj denote irrigation effects,
(αβ)ij are interactions. F -tests for fixed effects guided by EM S
column above
Eijk ∼ N (0, σ 2)
iid
Sk(i) ∼ N (0, σs2)
iid
Random effects and variance components:
Here, i = 1, . . . , a and j = 1, . . . , b and k = 1, . . . , ni where ni
denotes the number of plots treated with level i of factor a. If ni is
constant, call it n.
Yijk
random error component
z }| {
Sk(i) + Eijk
= µ + αi + βj + (αβ)ij +
.
|
{z
}
µij :(fixed component)
Where variance components and size effects pertain to the model for
a completely randomized split-plot design:
Source
df
EMS
A : Pesticide
a − 1 = 2 σ 2 + bσs2 + bnψA2
Plot(A)
(n − 1)a = (2 − 1)3 = 3
σ 2 + bσs2
B: treatments
b−1=3
σ 2 + naψB2
2
A×B
(a − 1)(b − 1) = 6
σ 2 + nψAB
2
B×plot(A)
(b − 1)(n − 1)a = 9
σ
Subplot error
Total
abn − 1 = 23
Osborne, ST512
23
df
EMS
2 σ 2 + bσs2 + bnψA2
3
σ 2 + bσs2
3
σ 2 + naψB2
2
6
σ 2 + nψAB
2
9
σ
F p−value
3.9 0.1452
10.1 0.0031
18.7 0.0003
1.3 0.3607
240
σ̂ 2 = M S[E] = 3.2 and σˆs2 = (M S[S(A)] − M S[E])/4 = 7.3
Estimated varcomps:
F = M S[S(A)]/M S[E] = 32.6/3.2
For random effect of whole plots, on 3, 9 df:
F = M S[AB]/M S[E] = 4.1/3.2
For pesticide by irrigation interaction, on 6, 9 df:
F = M S[B]/M S[E] = 60.2/3.2
For irrigation effect, on 3, 9 df:
F = M S[A]/M S[S(A)] = 128.1/32.6
For pesticide effect, on 2, 3 df:
• M S[E] denotes error or subplot mean square
• M S[S(A)] denotes mean square for WHOLE plots (nested in A)
Source
MS
A : Pesticide
128.1
Whole plot error M S[S(A)] = 32.6
B: treatments
60.2
A×B
4.1
B×plot(A)
M S[E] = 3.2
(Subplot error)
Total
For the corn yields data on p. 2,
Osborne, ST512
Pairwise comparisons
241
Estimate
2
nb M S[S(A)]
2
na M S[E]
2
2
2
n (σ̂ + σ̂s )
2
n M S[E]
2
2
2
n (σ̂ + σ̂s )
Variance
2
2
2
nb (σ + bσs )
2 2
na σ
2
2
(σ
+ σs2)
n
2 2
nσ
2
2
2
n (σ + σs )
df
(n − 1)a
(n − 1)(b − 1)a
messy
(n − 1)(b − 1)a
messy
/* parms statement can be used to keep SAS from dropping random
effects w/ negative estimated varcomps */
proc mixed method=type3;
class field pest trt irr cv;
model y=trt|pest/ddfm=satterth;
random field(pest);
*parms /nobound;
lsmeans trt pest/pdiff; /* can use adj=bon; to adjust for multiplicity */
*lsmeans trt|pest/pdiff; /* if there were interaction */
run;
To analyze data from a CRSPD in SAS, consider using PROC MIXED
instead of PROC GLM:
Comparison
Ȳi1++ − Ȳi2++
Ȳ+j1+ − Ȳ+j2+
Ȳi1j+ − Ȳi2j+
Ȳij1+ − Ȳij2+
Ȳi1j1+ − Ȳi2j2+
Skipping the algebra, the standard errors for all of these comparisons, save # 3 and #5, can be estimated ‘cleanly.’ That is, with
single M S terms and integer df . (See table 16.6, careful of errata)
5. Interaction effects: ȳi1j1+ − ȳi2j2+
4. Simple effects of B: ȳij1+ − ȳij2+
3. Simple effects of A: ȳi1j+ − ȳi2j+
2. Main effects of B: ȳ+j1+ − ȳ+j2+
1. Main effects of A: ȳi1++ − ȳi2++
Several kinds of pairwise comparisons of treatment means:
Osborne, ST512
1
1
1
1
1
2
2 3
2
2
2 3 4
4.081806
32.602083
3.228750
60.232639
128.137917
Mean Square
MS(Residual)
MS(field(pest))
MS(Residual)
MS(Residual)
trt
pest
pest*trt
field(pest)
242
9
3
9
9
Error
DF
18.66
3.93
1.26
10.10
F Value
0.0003
0.1452
0.3607
0.0031
Pr > F
Var(Residual) + Q(trt,pest*trt)
Var(Residual) + 4 Var(field(pest))
+ Q(pest,pest*trt)
Var(Residual) + Q(pest*trt)
Var(Residual) + 4 Var(field(pest))
Var(Residual)
Expected Mean Square
Type 3 Analysis of Variance
24.490833
97.806250
29.058750
Error Term
6
3
9
pest*trt
field(pest)
Residual
180.697917
256.275833
Sum of
Squares
Source
3
2
DF
2
3
2
2
4
field
pest
irr
cv
trt
Values
Type 3 Analysis of Variance
Levels
Class
Class Level Information
WORK.ONE
y
Variance Components
Type 3
Factor
Model-Based
Satterthwaite
The SAS System
The Mixed Procedure
Model Information
Data Set
Dependent Variable
Covariance Structure
Estimation Method
Residual Variance Method
Fixed Effects SE Method
Degrees of Freedom Method
trt
pest
Source
Osborne, ST512
1
pest
1
1
2
trt
trt
trt
trt
trt
trt
pest
pest
pest
1
1
1
2
2
3
2
3
3
9
3
9
Den
DF
54.1167
56.1500
58.1667
61.5500
52.8750
59.7500
59.8625
Estimate
1.3274
1.3274
1.3274
1.3274
2.0187
2.0187
2.0187
Standard
Error
DF
4.9
4.9
4.9
4.9
3
3
3
18.66
3.93
1.26
F Value
Least Squares Means
3
2
6
Num
DF
2
3
4
3
4
4
_trt
-2.0333
-4.0500
-7.4333
-2.0167
-5.4000
-3.3833
-6.8750
-6.9875
-0.1125
Estimate
1.0374
1.0374
1.0374
1.0374
1.0374
1.0374
2.8549
2.8549
2.8549
Standard
Error
9
9
9
9
9
9
3
3
3
DF
243
-1.96
-3.90
-7.17
-1.94
-5.21
-3.26
-2.41
-2.45
-0.04
0.0816
0.0036
<.0001
0.0838
0.0006
0.0098
0.0952
0.0919
0.9710
Pr > |t|
<.0001
<.0001
<.0001
<.0001
0.0001
<.0001
<.0001
Pr > |t|
t Value
40.77
42.30
43.82
46.37
26.19
29.60
29.65
t Value
0.0003
0.1452
0.3607
Pr > F
Differences of Least Squares Means
_pest
1
2
3
4
trt
7.3433
3.2287
Estimate
Type 3 Tests of Fixed Effects
field(pest)
Residual
Cov Parm
Covariance Parameter Estimates
trt
pest
pest*trt
Effect
trt
1
2
3
trt
trt
trt
trt
pest
pest
pest
Effect
pest
Effect
Osborne, ST512
40
50
60
y
70
Osborne, ST512
1
1
1
2
3
23
pest
1 1 1 1
2
1
1
2
2
3
3
trt
2 2 2 2
3 3 3 3
3
1
3
1
2
3
2
4
1
1
3
23
2
244
245
Irr CV
no 1
no 2
yes 1
yes 2
40
50
60
y
70
1
2
1
1
2
1
2
1
1
cv
1 1 1
irr
1
2 2 2
2
1.0 1.2 1.4 1.6 1.8 2.0
1
2
1
2
1
1
1
2
2
1
2
2
2
2
The 3 df for the within plot factor B can be broken up into three
1 df components due to main effect of irr, main effect of CV and
interaction. Same with the AB interaction. The plot below averages
over pesticide and field:
B
1
2
3
4
Corn yield, irrigation, pesticide and cultivars continued
So the treatment effect B is highly significant (p = 0.0003). Are
there particular comparisons among the three treatments that are of
interest? There are because real experiment is actually slightly more
complicated than previously described. The factor B is really a 2 × 2
combination of irrigation and cultivar:
Osborne, ST512
1
1
1
2
2
2
2
3
9
cv
irr*cv
pest
pest*irr
pest*cv
pest*irr*cv
field(pest)
Residual
DF
irr
Source
2
3
2
2
4
field
pest
irr
cv
trt
1
1
1
1
1
2
2 3
2
2
2 3 4
Values
5.357500
97.806250
29.058750
1.385833
17.747500
256.275833
2.733750
44.010417
133.953750
Sum of
Squares
2.678750
32.602083
3.228750
0.692917
8.873750
128.137917
2.733750
44.010417
133.953750
Mean Square
246
1
Var(Residual) +
Q(irr,irr*cv,pest*irr,pest*irr*cv)
Var(Residual) +
Q(cv,irr*cv,pest*cv,pest*irr*cv)
Var(Residual) +
Q(irr*cv,pest*irr*cv)
Var(Residual) + 4 Var(field(pest)) +
Q(pest,pest*irr,pest*cv,pest*irr*cv)
Var(Residual) +
Q(pest*irr,pest*irr*cv)
Var(Residual) +
Q(pest*cv,pest*irr*cv)
Var(Residual) + Q(pest*irr*cv)
Var(Residual) + 4 Var(field(pest))
Var(Residual)
Expected Mean Square
Type 3 Analysis of Variance
Levels
Class
The SAS System
The Mixed Procedure
Class Level Information
proc mixed method=type3;
class field pest irr cv trt;
*model y=trt|pest/ddfm=satterth;
model y=irr|cv|pest/ddfm=satterth;
random field(pest);
*parms /nobound;
*lsmeans trt pest/pdiff adj=tukey;
lsmeans irr cv /pdiff;
lsmeans irr*cv;
run;
Osborne, ST512
irr
cv
Effect
1
irr
irr
irr
cv
cv
irr*cv
irr*cv
irr*cv
irr*cv
Effect
1
1
2
2
1
2
1
cv
irr
2
55.1333
59.8583
56.1417
58.8500
54.1167
56.1500
58.1667
61.5500
Estimate
1.2219
1.2219
1.2219
1.2219
1.3274
1.3274
1.3274
1.3274
Standard
Error
DF
3.61
3.61
3.61
3.61
4.9
4.9
4.9
4.9
7.3433
3.2287
Least Squares Means
field(pest)
Residual
Estimate
Covariance Parameter
Estimates
2
_cv
-4.7250
-2.7083
Estimate
0.7336
0.7336
Standard
Error
9
9
DF
-6.44
-3.69
t Value
45.12
48.99
45.95
48.16
40.77
42.30
43.82
46.37
0.0001
0.0050
0.3815
0.1452
0.1171
0.8109
0.4670
0.0031
.
Pr > F
247
0.0001
0.0050
Pr > |t|
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
Pr > |t|
41.49
13.63
0.85
3.93
2.75
0.21
0.83
10.10
.
F Value
t Value
9
9
9
3
9
9
9
9
.
Error
DF
Differences of Least Squares Means
_irr
1
2
1
2
1
2
cv
MS(Residual)
MS(Residual)
MS(Residual)
MS(field(pest))
MS(Residual)
MS(Residual)
MS(Residual)
MS(Residual)
.
irr
cv
irr*cv
pest
pest*irr
pest*cv
pest*irr*cv
field(pest)
Residual
Cov Parm
Error Term
Type 3 Analysis of Variance
Source
Osborne, ST512
Split-plot in blocks (RCBSPD)
248
2
1
1
2
2
1
1
2
3
4
5
6
1
1
2
2
3
3
pest
1
2
3
4
5
6
plot
53.4
46.5
54.3
57.2
55.9
57.4
B1
53.8
51.1
56.3
56.9
58.6
60.2
B2
58.2
49.2
60.4
61.6
62.4
57.2
B3
59.5
51.3
64.5
66.8
64.5
62.7
B4
1
At the whole plot level (ignoring the split-plot factor), the df in an
ANOVA for pesticide effects are given by
Source
df
A : Pesticide 2
Farms
Error
Total
5
so that an F -ratio for the pesticide effect is based on df = 2, 2
farm
Obs
The SAS System
In the split-plot experiment with pesticide as the whole plot factor and irrigation×CV as the split-plot factor, suppose the six plots
come from two farms, with three plots in each farm. Suppose that
the three pesticide treatments are randomized to plots within farms.
Renumbering plots (1,2,1,2,1,2) as (1,2,3,4,5,6) and supposing plots
(2,3,6) come from farm 1 and plots (1,4,5) from farm 2, the data are
given as
In the randomized block split-plot design, sets of homogeneous plots
are be formed and levels of the whole plot factor are assigned to the
plots within these sets in a restricted randomization. Assignment of
levels of the split-plot factor are as in the CRSPD.
Osborne, ST512
249
iid
Source
df
EMS
2
2
A
a−1
σ + bσsr
+ brψA2
2
2
Blocks
r−1
σ + bσsr + abσr2
2
Whole plot error (r − 1)(a − 1)
σ 2 + bσsr
(Block×A)
B
b−1
σ 2 + arψB2
2
AB
(a − 1)(b − 1)
σ 2 + rψAB
2
Error
a(b − 1)(r − 1)
σ
(B×Blocks(A))
Total
abr − 1
2
Rk ∼ N (0, σr2) and SRik ∼ N (0, σsr
). All random errors are mutually independent.
iid
• k denotes block.
• j denotes level of B,
• i denotes level of A,
where
Yijk = µ + αi + Rk + βj + (αβ)ij + (SR)ik + +Eijk
= µij + Rk + (SR)ik + +Eijk
In general, for a RCBSPD with a levels of a whole-plot level (A)
randomized to r blocks (for a total of ra plots) and b levels of a
split-plot factor (B) within each plot, the model and ANOVA table
are given by
Osborne, ST512
3
6
1
2
9
farm*pest
Residual
DF
2
trt
pest*trt
farm
Source
pest
Values
1 2
1 2 3 4 5 6
1 2 3
1 2 3 4
Cov Parm
farm
farm*pest
Residual
19.292917
3.228750
60.232639
4.081806
59.220417
Effect
pest
trt
pest*trt
Num
DF
2
3
6
250
Estimate
3.3273
4.0160
3.2287
Den
DF
2
9
9
F Value
6.64
18.66
1.26
Pr > F
0.1309
0.0003
0.3607
Expected Mean Square
Var(Residual) + 4 Var(farm*pest)
+ Q(pest,pest*trt)
Var(Residual) + Q(trt,pest*trt)
Var(Residual) + Q(pest*trt)
Var(Residual) + 4 Var(farm*pest)
+ 12 Var(farm)
Var(Residual) + 4 Var(farm*pest)
Var(Residual)
Type 3 Tests of Fixed Effects
38.585833
29.058750
180.697917
24.490833
59.220417
Mean Square
128.137917
Type 3 Analysis of Variance
Levels
2
6
3
4
Sum of
Squares
256.275833
Class
farm
plot
pest
trt
data one; /* ’one’ is original dataset */
set one;
if plot in (2,3,6) then farm=1;
else farm=2;
run;
proc mixed method=type3;
class farm plot pest irr cv trt;
model y=pest|trt;
random farm farm*pest;
run;
The Mixed Procedure
Osborne, ST512
251
1
2
3
4
5
6
7
8
9
10
11
12
Obs
Data:
1
1
1
2
2
2
3
3
3
4
4
4
week
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
TQ
c
c
g
s
c
g
s
c
g
s
c
g
s
speed
variety
47
48
46
43
51
48
47
50
44
44
53
48
Low
Low
Medium
g
variety
49
51
49
46
55
49
51
53
50
49
53
51
Medium
49
54
51
49
53
48
51
53
50
51
58
52
High
high
s
Split-plot in blocks (RCBSPD)
Researchers for an ice cream manufacturer conduct an expt. to study
the effects of variety (sweetcharlie, camarosa, and gaviota) and mixing speed (slow, medium and fast) on ice cream quality. One batch of
each variety of strawberries is sampled on Monday over four consecutive weeks. Each batch is divided into three parts, which are randomized to the three mixing speeds and three quarts of iced cream
are produced, stored for one month, then tested for texture quality,
on a scale from 1-100.
Osborne, ST512
r−1=3
(a − 1)(r − 1) = 6
b−1=2
(a − 1)(b − 1) = 4
(b − 1)(r − 1)a = 18
abr − 1 = 35
Block
V×B
Speed
V×S
Error
Total
df Expected MS
a−1=2
i = 1, 2, 3 = a( variety )
j = 1, 2, 3 = b( speed )
k = 1, 2, 3, 4 = r( week )
Yijk = µij + Bk + (αB)ik + Eijk
ANOVA sketch
Source
Variety
where
Model
Osborne, ST512
252
proc gplot data=one;
title "Ice cream texture quality";
plot tq*variety=speed/haxis=axis1 vaxis=axis2;
*plot tq*speed=variety;
run; quit;
proc mixed data=one method=type3;
class week variety speed;
model tq=variety|speed;
random week week*variety;
lsmeans variety speed/diff;
lsmeans variety*speed;
run;
symbol1 value=dot c=black h=1.2;
symbol2 value=plus c=black h=1.2;
symbol3 value=diamond c=black h=1.2;
data one;
infile "strawberryice.dat" firstobs=3;
input week variety $ speed $ tq;
run;
*goptions dev=ps;
axis1 offset=(1 cm,1 cm) label=(height=2 "variety")
value=(height=2);
axis2 offset=(1 cm,1 cm) label=(height=2 "TQ")
value=(height=2);
/* SAS code for split-plot in blocks */
Osborne, ST512
253
Dimensions
1 2 3 4
c g s
Low Medium high
Values
2
2
4
3
6
18
variety
speed
variety*speed
week
week*variety
Residual
148.666667
112.166667
2.166667
19.222222
33.111111
37.666667
Sum of
Squares
Var(Residual) + 3 Var(week*variety)
+ Q(variety,variety*speed)
Var(Residual) + Q(speed,variety*speed)
Var(Residual) + Q(variety*speed)
Var(Residual) + 3 Var(week*variety)
+ 9 Var(week)
Var(Residual) + 3 Var(week*variety)
Var(Residual)
week*variety
Residual
speed
variety*speed
week
Expected Mean Square
variety
MS(Residual)
.
MS(Residual)
MS(Residual)
MS(week*variety)
MS(week*variety)
Error Term
74.333333
56.083333
0.541667
6.407407
5.518519
2.092593
Mean Square
3
16
16
1
36
Source
Type 3 Analysis of Variance
DF
Type 3 Analysis of Variance
Covariance Parameters
Columns in X
Columns in Z
Subjects
Max Obs Per Subject
4
3
3
Levels
Source
week
variety
speed
Class
WORK.ONE
tq
Variance Components
Type 3
Factor
Model-Based
Containment
The SAS System
The Mixed Procedure
Model Information
Data Set
Dependent Variable
Covariance Structure
Estimation Method
Residual Variance Method
Fixed Effects SE Method
Degrees of Freedom Method
Osborne, ST512
1
254
c
c
c
g
g
g
s
s
s
c
g
s
variety
variety
variety
speed
speed
speed
variety*speed
variety*speed
variety*speed
variety*speed
variety*speed
variety*speed
variety*speed
variety*speed
variety*speed
0.09877
1.1420
2.0926
Estimate
Fit Statistics
week
week*variety
Residual
Cov Parm
Low
Medium
high
Low
Medium
high
Low
Medium
high
Low
Medium
high
speed
48.0000
52.6667
48.8333
47.4167
50.5000
51.5833
45.2500
48.7500
50.0000
50.5000
53.0000
54.5000
46.5000
49.7500
50.2500
Estimate
0.6961
0.6961
0.6961
0.5424
0.5424
0.5424
0.9129
0.9129
0.9129
0.9129
0.9129
0.9129
0.9129
0.9129
0.9129
Standard
Error
Least Squares Means
-2 Res Log Likelihood
AIC (smaller is better)
AICC (smaller is better)
BIC (smaller is better)
variety
13.47
26.80
0.26
1.16
2.64
.
F Value
Covariance Parameter
Estimates
6
18
18
6
18
.
Error
DF
Type 3 Analysis of Variance
variety
speed
variety*speed
week
week*variety
Residual
Source
Effect
Osborne, ST512
6
6
6
18
18
18
18
18
18
18
18
18
18
18
18
DF
118.2
124.2
125.3
122.4
0.0060
<.0001
0.9004
0.3990
0.0516
.
Pr > F
68.95
75.66
70.15
87.41
93.10
95.10
49.57
53.40
54.77
55.32
58.06
59.70
50.94
54.50
55.05
t Value
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
<.0001
Pr > |t|
255
c
c
g
variety
variety
variety
speed
speed
speed
Low
Low
Medium
speed
g
s
s
_variety
Medium
high
high
_speed
-4.6667
-0.8333
3.8333
-3.0833
-4.1667
-1.0833
Estimate
variety
c
c
g
Effect
variety
variety
variety
speed
speed
speed
Low
Low
Medium
speed
g
s
s
_variety
Medium
high
high
_speed
-4.87
-0.87
4.00
-5.22
-7.06
-1.83
0.0028
0.4183
0.0071
<.0001
<.0001
0.0832
Pr > |t|
0.9590
0.9590
0.9590
0.5906
0.5906
0.5906
Standard
Error
t Value
Differences of Least Squares Means
variety
Differences of Least Squares Means
Effect
Osborne, ST512
6
6
6
18
18
18
DF
256