This research was conducted while the author, being a student in UNC at
Chapel Hill, N. C. was supported by the Biostatistics Department of this
University.
J
I
AN EXTENSION OF THE T-METHOD TO
~NBALAN~ED
LINEAR MODELS OF FIXED EFFECTS
'J
by
Yosef Hochberg
Department of Biostatistics
iversity of North Carolina a~ Chapel Hill
.
In~~itute
; j
1
of Statistics Mimeo Series No. 902
)
•
DECEMBER 1973
..
-..,:~;.
~ ).~;',:. ~.....
1irI'"
-£.,
':i'
"
-t:...... .•.•.. .
,.
"
.
'
I
1.
Introduction and Sununary
Among the various procedures for simultaneous interval estimation, Tukey's
I-method gives the shortest confidence intervals for pairwise contrasts when it
is applicable.
(~f.
Miller (1966) Ch. 2 and Scheffl (1959) Ch. 3).
However,
the T-method in its exact formulation, (excluding some conservative procedures
discussed in [2]) is restricted to well balanced designs.
In this paper we discuss a generalized T-type procedure for simultaneous
interval estimation of all estimable parametric functions in general fixed
effects, univariate linear models.
The new method, which we subsequently
abbreviate as the GT method, is an extension of a generalized T-procedure for
the unbalanced one way ANOVA model given in [7).
e
In section 2 we derive the new procedure in full rank and in less than
full rank models.
section 3.
Some important features of the procedure are discussed in
In order to gain insight into the types of models where the new
procedure is expected to produce satisfactory confidence intervals for pairwise contrasts, the discussion of section 3 is followed by examples from three
important designs.
In section 5 we discuss a modified conservative GT procedure
which constitutes a combination of a GTI procedure as in [2) and the GT method.
The proofs of some of the theorems are given in an appendix for purposes o'f a
convenient reading.
i
!
2.
The General Setup and the GT Method.
Suppose we are given a model for univariate data being linear in the
k
fixed effects e=(8 , ••• ,e )'£m •
l
k
We do not exclude from the model some other
concomitant parameters, such as fixed block effects or regression coefficients on
fixed covariates.
For purposes of inference, the usual assumptions on the normal
distribution of the error terms are made.
Many of such models usually fall
2
under the following general setup at their inference stage.
"4IJ'
We refer the'
reader to [3] Ch. 11 and to [1] for two major examples, namely, the general
incomplete block design and general designs with covariables, respectively.
The general univariate fixed effects linear normal model at an
inference stage
Based on a class IB of matrices B=«b.~ j »
.. -1 , ••• , k'
~,J-
(usually JB is
the class of generalized inverses of the matrix involved in the normal
equations for 8), a class {a} of random vectors
e (usually the
of solutions for 8 in the normal equations) can be generated.
k
t e ={£EIR
:
-
E(£'O)=£'8} be invariant to any 8E{a}.
~_
class
Let
These assumptions together
with the f~llowing four constitute a model subsequently referred to as the
general normal setup.
i)
ii)
lJ3 includes symmetric p. d. (positive definite) matrices.
te
is a linear subspace of dimension e.
The set {£'8: V£E
-
-
S.e }
constitute the space of estimable parametric functions.
iii)
For all &1'&2E"!e' Cov(~la'~2a) = cr2~lB~2 invariant to any choice
The quantity 0 2 is the unknown variance of the error term
of BElli.
involved in the original model.
iv)
An unbiased estimator s~ of 0 2 is available such that vs~/02 is a
central chi-square variable with V d.f.
Let
&=(~l""'£k)'Emk, put ~1
= max{£i'O},
~i=max{-£i'O} and
define
(2.1)
We denote by
q~~~(q~~~)
...
R(~) (R(~»
the range (augmented range) of a
vector~.
By
we denote the I-a percentile of the studentised range (augmented'
range) distribution with parameters k,v.
(Cf. [5] Ch. 2 for definitions).
3
We now describe the G'i' method in full rank models, Le., when
our general normal setup implies a unique B and a unique
Theor~m~
te =lRkwhich
in
e.
. 2.1
Let Q: kxk be a matrix of real elements satisfying
Q QI
=B
(2.2)
then, the probability is I-a that all linear parametric functions
~'8
simultaneously satisfy the event
(2.3)
Proof.
Fi~st
we note that B is p.d. and thus there always exists a matrix Q
satisfying (2.2) and Q ia non-singular.
The linear combination ~'(8-8) can be
written as
(2.4)
The vector Q-l(8-8) is normal with zero mean and dispersioncr 2 I.
Now, the
event
(2.5)
is equivalent to the event
. { I~ 'Q -1 (8-8)
I~ sv
l.,hich in terms of the origi.nal
~'s
~H(~'), Y}!£IRk }
can be written as
{1~'QQ-I(e-8)I ~ sv~H(~'Q),
y
~t::IRk}
On equating the probability of (2.5) to I-a we get
~=qk(a)
,v and the Theorem
now follows from the equivalence of (2.5) and (2.7) and from (2.4).
Note that Spj~tvoll and Stoline's method in [7] is a GT method in the
special case when B is diagonal.
(2.6)
These authors
u~e
for Q the matrix
(2.7)
4
•
1/2
1/2
DJ.ag(b
, ••• ,b
).
ll
kk
We also note that for any given B a matrix Q satisfying
~
(2.2) exists but is not unique, and that the GT procedure is not invariant
under all possible choices of a matrix Q.
also in
Spj~tvoll
This non-invariance property exists
and Stoline's diagonal case, and even in the original T-method
which can always be subjected to an orthogonal transformation of the original
vector
-e.
A discussion on the multiplicity of choices of a
Q matrix in (2.2)
and the choice of an appropriate one is given in section.3.
A simple GT3 procedure in models of less than full rank is based on the
fact that the original model can always be reparametrized to a full rank model
by transforming 8 to a vector
estimable
~arametric
ee :
functions.
eXl constituting a basis for the space of
ee
Suppose that
ee = L8,
is given by
L: exk
This brings us to a full rank model with
ee =L8
(2.8)
and
ee
~ N(e ,a 2 LBL').
e
The
procedure embodied in Theorem 2.1 can now be used in the transformed setup to
provide simultaneous interval estimation of all linear parametric functions (of
8 ) with an exact I-a confidence level.
e
The fact that various different
choices of a basis 8 for the estimable functions exist does not increase the
e
set of different possible GT procedures.
A discussion of this point is given
in the Appendix by Lemma A.
An altenlative GT method in models of less than full rank is based on
. the following Theorem, the proof of which is given in the Appendix.
alternative procedure we assume that
te
For this
is the contrasts subspace in IRk •
This subspace of ~kis hereafter denoted by Ck •
Theorem 2.2
Let BElli be any symmetric p.d. matrix.
that all contrasts
S.'8
simultaneously satisfy
The probability is at least I-a
5
{C'eE[C'S
.ECk}
..
.. -+ s vq(a)M(C'Q
k,v .. B)], V .C
(2 ~9)
where Q Q'=B and 8is any element in {a}.
BB
There are few reasons for willing to use, in some cases, a conservative
procedure as given by Theorem 2.2, rather than an exact procedure obtained by
first transforming to a full rank model.
The quantity
1.
approximate it by
q~~~
is not tabulated.
It is well known that we may
q~~~ if a < .05 and k > 3,' (Cf. [6], p. 78). However, if
one wants to be on the safe side, especially when a > .05 is used, the conservative procedure might be more appropriate.
2.
In mmlY cases the parametric functions of interest are easier to
"
specify in terms of the original parameters e than in terms of the transformed
vector
ee .
3.
In some problems it is easier to find an appropriate (to be discussed
in the next section) Q matrix corr.esponding to a certain patterned BElli (any of
which can be used as long as it is symmetric and p.d.) than a Q corresponding
to the uniquely determined matrix
L~L'.
In various important unbalanced designs, we may get an exact I-a confidence level for the simultaneous estimation of all contrasts based on
q~~~.
This is given by the following Theorem the proof of which is given in the
This Theorem assumes the setup and notation of Theorem 2.2.
Appendix.
For
convenience we write Q for QB.
Theorem 2.3
Let
.-
~1'."'~k
constitute the k rows of Q,
~i:
lxk (i=l, ... ,k) and
'
suppose the condition
q.l_ = q.l V (l<i<j<k).
~K
Ji<.
--
(2.10)
6
is satisfied.
The probability is l-a that all contrasts simultaneously satisfy
e
(2.11)
3.
On the Choice of theQ Matrix
The various" procedures introduced in section 2, entirely depend on the
specific choices of Q matrices satisfying (2.2) for a given B matrix.
1.
1/2
1/2
1 / 2 . ..
Let D
= Diag (Al ' ••• , Ak ) where
t~e
.
Ai's are the k pOS1tive
latent roots of B and let R=(rl, ••• ,r ) where the r. 's are any corresponding
- k
-1
normalized latent vectors.
Clearly Q can be chosen as
Q = RD l / 2
(3.1)
A
The multiplicity of choices for a Q matrix in this class is well known.
smaller class is the set of lower triangular matrices T satisfying
T T'
=B
(3.2)
k
,Clearl.y, Q canbechosen--a-s· any '0f ,the 2possihle matrices T.
many other types of matrices Q satisfying (2.2) do exist.
However,
Moreover, searches
for satisfactory Q matrices in the above two classes have been non-fruitful
so far, for various designs.
Can we always find a Q satisfying (2.2) which gives shorter confidence
intervals for all pairwise contrasts simultaneously than any other Q?
ansvler to this question is that generally we cannot!
The
A simple case when an
optimal choice of Q for the class of pairwise contrasts exists, is the case
when B is of the form B=aI.
uniform diagonal.
A matrix B of that structure is said to be
The optimal choice in this case is Q=a l / 2r which gives
the original T-method.
To see this, first note that on letting c(ij)E:C
k
denote the vector of coefficients for the (i,j)th pairwise contrast
M(c('.,)Q)
- 1J
q .)
= M(qi- -J
(3.3)
e
7
where
~i
is the ith row of Q. (i=l •••• ,k).
Next note that (q.-q.)(qi-qj)' = 2a
~1
~J
~
-
and it is easily verified that
min
-tElR
k
--
- {M(t')}
: t't=2a
= a 1/2 •
(3.4)
-
As mentioned above, such an optimal choice for Q does not always exist.
can easily be verified in
In crude
spj~tvol1
This
and Stoline's general diagonal case.
we may say that in general one would- try to obtain a Q
tel~S
matrix with a structure as close as possible to a uniform diagonal matrix.
It seems that this cannot be expected in designs which are more than moderately
unbalanced.
However, in many important designs, Q can be chosen as to have the
same structure as that of B.
In such cases if B is sufficiently close to a
uniform diagonal matrix. we may expect a good performance of a'GT procedure
based on a Q matrix chosen to be structured as B.
This approach is an extension
and Sto1ine's approach where they choose Q = Diag(b 1/2 , ••• ,b 1/2 )
ll
kk
in the diagonal case when B is given by Diag{bl1, ••• ,bkk)' We further explore
of the
Spj~tvo1l
this method in the next section by demonstrating it in three different designs.
4.
Some Examples
Example 4.1
ANOCVA with one concomitant variable and equal sample
size~•.
(See [1]).
This is an example
used.
~f
a full rank model where the original T-method can not be
Let Y.. (j=l, ••• ,n; i=l, ••• ,k) be N=kn independent random variables with
~J
a common variance 0 2 and expectations given by
E Y., =
~J
= L
tJ'~l
/n,
- X..
~J
Y.~
e.
~
+ bXi " j=l, ••• ,n; i=l, ••. ,k
J
n
= L
t. = Yi,/n,
J
J l
L. k 1 Lj =n 1
~=
- 2
(X ~J
.. -X.)
,
~
(4.1)
8
A
b ; S
/S
xyxx
, SSA ; S2 /S
b
xyxx
and V = N-k-l.
The least squares estimator of 8
...
adjusted for the concomitant variable is 8i
i
given by
(4.2)
where
~=(b1, ••• ,bk)"
b
i
e. 's is
The dispersion of the
This is a full rank model.
=
Xi/s~2
(i=l, ••• ,k).
1.
given by
The unbiased estimator of cr 2
is
(4.4)
•
To apply a GT procedure as given by Theorem 2.1, we look for a Q matrix with
the same structure as that of B in (4.3), i.e., Q; aI + hh', for some aEJR1
and
2;
k
(hl, ••• ,hk)'EIR •
2 we
To find the vg1ue of a and
~
write the equation
(4.5)
We get:
a2
lin and h = k l / 2b for·some scalar k satisfying the equation
;
k(2a+k~'e) =
(4.6)
1
The resulting confidence intervals for the pairwise contrasts based on the GT
method are given by
- - + s"qk
- (ex)\)«l/n) 1/2 +
8.-8.E[8.-8.
1.
J
1.
J ,-
v
,
Ih.-h. IM(h'»]
~
J
-
(l<i<j~k)
(4.7)
From (4.7) and the above formulas for the values of the h.
's it is clear that
-1.
the closer the values of X. and X. are, the shorter is the confidence interval
~
Example 4.2
J
A group divisible PBTB design.
(See [3] eh. 12).
This is an example of a model of less than full rank where the original T-method
9
cannot be used.
Consider. a design with k treatments, b blocks, each of the
treatments being tested in r blocks, each of the blocks consisting of d plots.
Let n
ij
block.
be the number of times the ith treatment is being tested in the jth
Put R=r
~:
kXk
D~dI:
bXh.and N=«n
ij
»:
kXb.
We assume the additive
fixed effects linear model Yij = ~+8i+Sj+~ijm' where the 8i 's and Sj'S are the
treatment and block effects respectively,
~
is the overall mean and the
are i.i.d. normal with a zero mean and variance
a2 •
ith treatment and the jth block totals respectively.
Denote by t
We let
i•
~ijm
and b . the
.J
!=(tl.""'~.)',
2=(b. l , ••• ,b.b)' and get the normal equations for the treatment effects having
eliminated blocks as
(4.8)
Suppose the design is a PBIB(2) design with parameters b, k, r, d, AI' 1. 2 '
i,j,t='1,2.
i
Pj~'
We put
(4.9)
As shown by C. R. Rao, (cf. [3] p. 257) a solution matrix B=«b ..
1.J
».1. , J"-1
. k
- ,...,
for the equations (4.8) is given as
b
ii
b ij
-1
=
B22 dll r ;
:;
0
h
ij
otherwise
=
-dB 12 11
-1
r
if (i,j) are first associates
(l~i<j<k)
}
(4.10)
This symmetric p.d. matrix B is one member of the class B of all solution
matrices to the equations (4.8).
The solution vector 8=B(t-ND- l b) satisfies
-
-
(4.11)
Clearly this setup pertains to the general normal model introduced in section 2.
10
The class mis the set of all solution matrices to (4.8).
To use GT procedures
as given by Theorems 2.2 and 2.3 we may use any symmetric p.d. matrix from m •
However, as we subsequently
particularly
conv~nient
exampl~fy,
the choice of B given by (4.10) is
for a derivation of an appropriate Q matrix.
Suppose
the PBIB(2) design is a group divisible design with m groups and n treatments
in each group, mn=k.
In this case B is of the form
(4.12)
B = I 0 U, I: mXm, U: nXn
where U'-"(a-c)I+cll' with a and c as given by (4.10) and 0 denotes the direct
product operation.
If Q is to be of the same structure as B, we must have
Q= I @P
where P
=
(4.13)
tg-f)I + fll' and such that (2.2) is satisfied.
This results in the
equations
I:
II:
(n-l)f 2 + g2
=a
(4.14)
(n-2)f 2 + 2gf = c
which are easily solved for f and g.
In this
cas~with
e
that choice of Q,Theorem
2.3 can be used and the resulting confidence intervals for the pairwise contrasts
are given by
(4.15)
where s~ is the usual mean 58 for error based on \l=bd-k-b+l d.f.
If n=l the
design is a balances! incomplete block design in which Tukey'sT-method can be
applied.
It comes out as a special case of the GT method for n=1.
It is
obvious that for large values of m and low values of n the GT procedure can
be expected to produce the best results in a comparison with any of the other
possible methods.
Before c1osi.ng this section we will bring a numerical example
to demonstrate this fact.
e
11
Example 4.3
A complete randomized block design with missing data.
(See [4J
p. 173).
Consider a complete randomized block design with k treatments and b blocks.
Suppose that two observations are missing in one block which without loss of
generality will be assumed to belong to the {k-l)th and the kth treatments.
The initial assumed model is
(4.l6)
= 8. + S. + e .. ; (i=l,. •• ,k; j=l, ... ,b)
J
~
S. 's
where the 8. 's and the
are fixed unknown treatment and block effects
J
~
~J
respectively and the e .. 's are i.i.d. normal variables with zero means and
~J
variance
(12.
Denote by
T~
~
the total of the available observations for the ith
.-
treatment; i=k-l, k, and let B denote the total of the k-2 observations in the
block where the data is not complete.
available observations.
~
Let G denote the sum of all bk-2
Following a well known method, due to F. Yates, we
estimate the missing observations and then use the data and the estimates to
analyze a 'complete' block design rather than having to lland1e an incomplete
block design.
Following that method, on deu'Ot.ing the estimates of the missing
A
A.
values by Y - l and Y we obtain the equations
k
k
(4.17)
Solving the equations we get
A
(k-2) (b-1)Y k _1 = (k-l)Tk_l+Tk+bB-G
A
(k-2) (b-l)Y
k
(4.18)
k k_l +bB-G
= (k-1)T +T
. Denote by Ti (i.=l, .•• ,k) the total of the observations (including estimated
values) for the ith treatment.
The estimates of the 8. 's are given as
~
~i = Ti/b (i=l, ••• ,k)·
The dispersion matrix of the
..
8i 's
is a 2 B where B=«b
(4.19)
ij
»
is given by
12
B
=[
n:
(k-2)x (k-2)
o
1
k-l
.
n = (l/b)I and V = (w-r)I + rll' where w = b[l + (b-l)(b-2)] and
t=1/[b(b-l)(b-2)]~
The error mean SS here is calculated from the complete data
but has two d.f. less than that in an actual complete block design.
Q of the same structure satisfying QQ'=B is
Q= [
(1/b)1/2 I : (k-2)X (k-2)
o
give~
A matrix
by
o
(4.21)
(f-g) I+gll ' :
intervals for the pairwise contrasts
1<i<j<k-2
1<i<k-2<j<k
(4.22_
It is almost readily obvious that for any value of b and a moderately large
vlaue of k the GT method will give shorter intervals than the S-method and the
conservative procedures discussed in [2] for all pairwise contrasts.
A numerical demonstration for example 4.2.
Suppose we have the following group divisible PBIB(2) design for the
six treatments 1,2,3,4,5,6 in six blocks
(1,2,3); (3,4,5); (2,5,6); (1,2,4); (3,4,6); (1,5,6)
The matrix B given by (4.12) is completely specified when specifying U.
this example we get
"
(4.23)
In
13
U
=
21/48
(
3/48
3/48)
21/48
(4.24)
'
The matrix Q for this example is then given by
Q=
"65974
•04737)
( .04737
.65974
From (4.15) we obtain the following confidence intervals
(4.25)
.
ei-ejE[6i-6j+sV 3.3007], if (i,j) are first associates
(4.26)
e -e E[8 -8 ±sv 3.8113], if (i,j) are second associates.
i j
i j
Using Scheffe's S-method on this particular example we get
• ei-ejE[6i-8j+sv 3.8594] for first associates
(4.27)
e.-e.E[8 -e.+s" 4.1684] for second associates.
i J- v
~
J
Using the GTI method of [2] we obtain the conservative intervals
(4.28)
The GT2 method discussed in [2] is based on the I-a percentile of the
studentized maximum modulus distribution with parameters (k',V), where
k'=k(k-l)/2.
In this example k'=15 and
not yet been tabulated.
v=7 and the appropriate constant qas
However, from the discussion in section 4 of [2] it
is clear that we can expect the GTI procedure to give better results than the
GT2 does in this specific example.
5.
On a Modified GT Method
In various designs it is difficult to find a satisfactory Q matrix for
the use of a GT procedure.
This happens in many incomplete block designs
other than the class of FBIB designs and in various models other than the
14
balanced one way ANOVA when fixed covariates are introduced.
In all these cases
pertaining to the general normal setup a conservative procedure is now defined
which involves features of the GT method and the GTl procedure introduced in [2].
We
denote this modified GT procedure by MGT.
8i (a) =
Lj~l,jfila-bijl,
Given a B= «b
ij
»
matrix let
aeIR 1 and B* = Diag«b ii + 8i (a)-a)1/2), (i=l, ••• ,k).
Theorem 5.1
The probability is at least l-a that all estimable linear functions
simultaneously satisfy
(5.1)
Proof.
Fo~lows
from Theorem 2.1 in [2] and the above Theorems in this paper.
Note that various MGT procedures exist corresponding to different values
of a.
For the (itj)th pairwise contrast we get
M(c(' .. )B*) "" max{(b .. +S.(a)-a)1/2 t (b ..+8.(a)-a)1/2}
-
~J
~~
JJ
~
J
(5.2)
A reasonable MGT procedure t i.e. t a reasonable choice for a is the value of a
min
a
max {b .. +S.(a)-a}
~~
~
l<i<k
=
max {bi.+S,(a*)-a*}
l<i<k
~ ~
This optimal choice a* for a is exactly the choice for a in the GTl procedure.
A search procedure for a* is outlined in [2].
(5.3)
e
15
Appendix
Lemma A
Suppose that in an original model of less than full rank two reparametrizations are given
i=1,2
(A.l)
The two resulting classes of GT procedures are equivalent in the sense that
there exists a non-singular A such that a GT based on the second reparametrizabas~d
don with Q2' is the same as a GT
Proof.
on the first with Ql = AQ2'
Let A be the (unique) matrix satisfying
"
(A.2)
Z _.
Since Q Q
2
LZBL
Z it
follows that
Q Q' = AL BL'A' = L BL'
1 1
.Z 2
1 1
Let
&:
(A.3)
e x 1 be the coefficients of a certain linear function in terms of eeZ'
the same parametric function in terms of eel has a vector of coefficients A- l &.
This implies that the argument of the function M(o) when using the first
reparametrization with Ql = AQ Z is &'A-1Ql = &'A- 1AQz = &'Q ' which proves
Z
the Lemma.
Proof of Theorem 2.2
-
The dispersion matrix of e can be expressed as
cr 2 (B+E )
(A.4)
B
B
for some real matrix EB=«e ij »,~,J'=1 , .•• , k'
assun~tions
assumption
It is however clear from the
underlying the general normal setup at an inference stage and the
t e =Ck
of this Theorem that
(A.S)
16
Next we show that it is always possible to construct a matrix R: kxk of the
form R
= 1 ~ hI, h =
(hl, ••• ,~)'EIRk such that
(Q+R)(Q+R)' = B+E B
(A.6)
This is so since '(A.6) together with QQ'=B implies
E
B
= QR'
= I' 0
+ RQ' +RR'
which can always be solved for
h.
Qh + 1 I8l h' Q' + II'
-
~
h 'h
- -
(A. 7)
Following the same line of argument as in
the proof of Theorem 2.1 we have that the probability is l-a that all &E:IR
k
s'imultaneously satisfy
1&'(8-E8)1
~
-< sv q(a)
k,v M(~'(Q+R»
..
The theor~ now follows on realizing that for all
=
£' e
k
and C elR
k
CEC
(A.B)
k
•
-Proof of Theorem 2.3
The condition on Q implies that £'QEC
k
k
for any SEC.
This together with
the equivalence of the events
- vqk(a?)
, v and { I.c'
. (6-E6) I -< s v qk(a)
,v M(c. . ') , V
. {R( 8-E8) < s
and the assumption that
.
"
te
=
Ck imply the theorem.
k
CEC }
....
(A.9)
17
References
[1]
Bose, R. C. (1965).
The general analysis of covariance.
transcribed by Gary G. Koch.
Lecture notes
Available from the Biostatistics Department
of UNC at Chapel Hill, N. C., U.S.A.
[2]
Hochberg, Y. (1973).
Some conservative generalizations of the T-method
in simultaneous inference.
New York, N. Y.
[3]
Contributed paper, Annual IMS meeting
To be published in the J. of Mu1t. Anal., 1974.
John, P. W. M. (1971).
Statistical design and analysis of experiments.
The Macmillan Company, New York.
[4]
Kemptporne,
o.
(1952).
The design and analysis of experiments.
John
Wiley Inc., New York.
[5]
Miller, R. G., Jr. (1966).
Simultaneous statistical inference.
McGraw-
Hill, New York.
[6]
Scheffe, H. (1959).
[7]
Spj~tvo11, E. and Sto1ine, M. R. (1973).
of multiple
The analysis of variance.
comp~rison
John Wiley Inc.
An extension of the T-method
to include the cases with unequal sample sizes.
Contributed paper, IMS meeting, Ithaca, N. Y.
To be published in JASA
1974.
[8]
Tukey, J. (1953).
manuscript.
The problem of multiple comparisons.
Princeton University •
.
~
Unpublished