.
UNIVERSITY OF NORTH CAROLINA
Department of Statistics
Chapel Hill, N. C.
Mathematical Sciences Directorate
Air Force Office of Scientific Research
Wasr.· in gton 25, D. C.
AFOSR Report No.
1418
BAYES RULES FOR A COMMON MULTIPLE COMPARISONS PROBLEM
AND RELATED STUDENT-t PROBLEMS*
by
David B. Duncan
November, 19tiO
Contract No. AF 49(638)-929
Qualified requestors may obtain copies of this report from the
ASTIA Document Service Center, Arlington Hall Station, Arlington
12, Virginia. Department of Defense contractors must be established for ASTIA services, or have their " need-to-know" certified by the cognizant military agency of their project or contract.
Institute of statistics'~)
Mimeograph Series No. 26
1
*
A revision and extension of "A Simple Bayes Solution to a Common Multiple
Comparisons Problem," AFOSR TN 59-492, AD 215 845, April, 1959.
BAYES RULES FOR A COMMON MULTIPLE COMPARISONS PROBLEM
l
AND RELATED STUDENT-t PROBLEMS
by David B. Duncan
University of North Carolina
O.
~~.
The paper is mainly concerned with the f'ollowing
mUltiple comparisons problem in the analysis of variance setting.
a balanced experiment n treatments are to be compared.
In
Each of the
n(n-l)/2 pairwise comparisons is to be made, adjudging each difference
as "positive", "negative", or "not significant"; overall decisions involving intransitivities are barred.
The loss for each difference is
proportional to the error; if a difference is asserted incorrectly the
loss has proportionality constant k + k > k which is the proportionl
o
0
ality constant if "not-significant" is the (incorrect) conclusion.
Total loss for the experiment is taken as the sum of the n(n-l)/2
component losses.
The Bayes rule is shown as a result to consist in
the simultaneous application of Bayes rules to the n(n-l)/2 component
problems.
Each of these in turn is phown similarly to consist in the
simultaneous application of Bayes rules to two subcomponent problems into
which each can be further decomposed.
The subcomponent Bayes rule for
a normal prior density of treatment means is explicitly derived.
The dependencies of the solution on the variance of the prior density,
the degrees of freedom and the loss ratio kl/k are discussed.
o
A principal
1 Research sponsored jointly by the United States Air Force through
the Office of Scientific Research of the Air Research and Develqpment
Command under Contracts No. AF·49(638) .. 929·and No. AF 49{638)-26l, by
the Office of Naval Research under contract mo. Nonr 355(06), and by the
U.S. Public Health Service. Reproduction in whole or in part for any
purpose of the United States Government is permitted.
2
finding is that the Bayes solution for the multiple comparisons
problem corresponds to a tolerated error probability "of the first
kind" for each single difference, which is independent of the number
of treatments being compared.
1. Introductionc
Many procedures have been proposed for the
multiple comparisons problem herein considered.
example, a
£27,
I
least~significant-difference'
These include, for
rule due to 7.:1.sher .•
an 'honest-significant-difference' rule due to "Tukey
[i§.7 and multiple range testing procedures due to :Ne\ftll8n
the author £~7.
,~, \;"~"
[i17, [",".'
[IQ7~ ~d
Some of these have also been described in recent
texts such as Fede,r\er ["~7, Li
and Steel and Torrie
£2.7,
Snedecor [i~7, Scheffe
f.ig7
[i!il. With much help from the recent more
general work of L.ehmann
£17
it has now been possible to solve a
Jeffrey's-like Bayes formulation of the problem.
This is more complete
than any of the previous formulations and leads to a simple solution
with properties which are better defined and which appear to be appropriate to an appreciable class of practical situations.
In the
process, similar Jeffrey's-like Bayes formulations and their solutions
are presented and obtained for two common type of Student..t problems.
These are developed first as problems of separate interest in sections
2 and 3.
The main problem then is fully developed in section 4.
discussion of the more
ap~lied
A
aspects together with further illustra-
tions is planned for a paper to be submitted to Biometrics.
2.
A Two-Decision Student-t Problem:
Given a random observation
t from a non-central t distribution with non-centrality parameter
T
and with v degrees of freedom, as common problem is that of choosing
between the two decisions
3
(2.1)
where D. is some smaJd unspec1:f'1e.dposft1ve I s.d.gnificant I :value. .In the
literature the same problem is at''b=n more loosely regarded as that of
testing H :
o
=0
- ...
T<
with the alternative HI:
T
> 0, the ;dec-is1ons thus
'
(2.2)
d : decide T
o
<=0. and d l : decide
T
>0
Strictly speaking however the null decision does not deny the possibility
of positive though relatively small values for T and some such formulation as (2.1) is more precise.
this problem by itself.
The change is relatively trivial in
It is essential however to our sUbseqe.ent
developments as is brought out shortly after (4.12) ~$.'JL.~a.rJJJ..117
,
tl1s.f;> 1'Q.~\!a".Yi.~ttiAJ:-J:i'r id~.Jl:ti-s:al cbaD8~ .:1.:'.~~~ll;:i.;;;:t;~.i .. · ;.i:::r...:;:::: .
,.
Our first result is a Bayes rule ¢(t) for this problem with
where k and k are positive constants such that k > k ' and with
o
o
1
l
respect to a normal prior density for T,
(2.4)
£(T)
122
= (2~ r2)-2 e- T /2r ,
with mean zero and variance
2
r.
-00
<
T
<
00,
The rule is of the common form
4
where ¢(t)
= 0 or 1 is the usual indicator function for the making of
2
the decision do and d1 respectively, and t* = t*(k, v, 1 ) is a
significant or critical t ratio for which a set of values are given
in Table 1.
the ratio k
The arguments determining the significant value t*, are
= k1 /ko
from the loss-function, the degrees of freedom v for
t and the variance ~ of the prior density for
Table 1:
T.
Minimum-AverliJ,ge-Risk Significant t Values
Log k
1
2
co
.5
1
2
4
6
14
00
0.0
0.0
0.0
0.0
0.0
.375
.413
.434
.443
.451
.457
.807
.860
.884
.891
.898
.902
1.. 353
1.379
1.367
1.356
1.340
1.326
2.102
2.012
1.900
1.848
1.779
1.7a1
0.0
0.0
0.0
0.0
0.0
0.0
.444
.484
.506
.515
.522
.528
1.053
1.060
1.056
1.053
1.047
1.041
2.503
1.926
1.718
1.653
1.582
1.531
4.077
2.623
2.370
2.136
1.987
0.0
0.0
0.0
0.0
0.0
0.0
.572
.610
.629
.637
.642
.646
1.930
1.532
1.395
1.353
1.308
1.275
8.741
2.648
2.303
2.030
1.875
0.0
0.0
0.0
0.0
0.0
0.0
.767
.785
.791
.794
.792
.792
2.292
1.963
1.800
1.653
1.562
1
2
4
6
14
co
1
1
2
4
6
14
00
5
2.0' ,
0.0
00
3
1.0 - . 1.5
v
1
2
4
6
14
co
0
Q)
co
c:>
co
00
8.592
3.980
2.859
.' 2'.433
2.5
3.160
2.. 814
2.502
2.374
2.217
2.091
6.854
5.208
4.010
3.580
3.099
2·759
co
co
co
CD
CD
4.178
3.308
2.724
2.414
00
00
co
13.625
3.891
2.957
(I)
00
co
·co
00
00
00
4.162
2.980
4.685
3.851
3.197
2.948
2.654
2.436
CD
co
00
3.5
co
00
9'.243
3.777
2.693
2.296
3.0
6.670
3.622
9.595
4.732
3..360
2.813
7.706
4.074
3.186
00
co
JX>
00
co
co
5.326
3.445
co
00
7.818
3.902
co
co
co
co
00
co
co
4.219
00
co
00
4.779
5
The rule 0*(t) may be derived as follows.
For the average
risk of any rule ¢(t), we have
coco
(2.6)
A(s, ¢)
=I.J
LLO(T) (l-¢(t»+-Ll (T)¢(t17f(t jT) dt g(T) dll
-eo
- (1).>-
J
00
= constant
+
¢(t)hl (t)dt
-00
where f(tj7) is the non-central t density function and
fell:)
hl (t)
=
JJJ.
1
(T)-Lo (T17 f·S·dT.
-<D
The minimum average risk rule may thus be written
(2.8)
From the inequality hl(t) < 0 we get
00
<
J
koT f·s·d T
o
and hence
(2.10)
>
where
(2.11)
Now, introducing the non-central t density in the form
f(t IT) =
oo _~(ut_T)2
e
1.
o
(2n:)2
J
.
u:1/F(ujiv)du, -co<t< CO,_,
6
where
~(ulv) is the X~related density function of u = X(v)/v,
that is
v/2 v-l _VU2 /2
vue
1/1 (u I)
v = -----=~-=-­
v • 1 '
(~ • 1) ~ i2
(2.13)
= 0,
u>o
otherwise,
and discarding constants in h (T,t) which will cancel and not affect
3
the value of h ( t), we have
2
f e-~[[ut-T)
co
(2.14)
h (T,t) cJ: T
3
Putting ~2
(2.15)
=1
2
2
+ T
2
/rJu1/l(u lv)du •
o
+ 1/~2 this becomes
22
h (T,t) c<. Te- T 13 /2
3
[00eutT- u2 (v+t2 )/2 u d
V lJ.
92/2 00L:
(tT) i...
1/2.
~ (.1-)
(v+J.-l) l
2
i=o
i . v+1t
2
= Te ·"-13
00
Next, putting h (t)
4
J
=
h (T,t) dT and y
= t/t3(v+t 21
)2
o
integrating term by term, we get
(2.16)
h (t)oe
4
co
L:
i=o
fri 2i/2(V+~.1) l J(T/3)i+le -(:rf3) 2 /2d(T~)
00
o
and
7
(2.17)
= ~
h (t)
4
=
y
LY'i=or; yi(V+~-l) ~/(i;1)l7
~ [Yh (y) + yh6(y)]
5
where h (y) is the sum of even terms
5
00
h (y)
5
p =
v/2,
=z
y2i(p+i_i)~/(i~)~ ,
i=o
and h (Y) is the sum of odd terms
6
I
co
h (Y) =.Z y2i+1(p+i)~/(i+l)~
6
J.=O
Working first with h (y) this may be written
5
(2,18)
h (y)
5
= (p-t)~F(p~,
1; 3/2; y2)/(t)~
where F(a, b; c; x) denotes the hypergeometric function
1 +~ +
c
a(a+1~b(b+1)
c c+l)
2
x +
2'"
Applying Euler. 1 s transformation F(a, b; c; x)"
= ( l-x ) c-a-bF ( c-a,
c-b;
h (y) =
5
Cj
.ro
..
x ) and reducing, we get
~ (l-i) -p fil-U2 )P-1du(p-tE) ! / (fi)l
o
Hence
(2,20)
~
J (l-u2)P-.~dE.7.
y
LYh (y17 = (1"'>i/?)-(P+1)[[1- y2 )P+2P
5
o
,,(p-t)!/(t)~
Next, h (y) sums to (p-l) ~[[1-y2fP...!7/y so that
6
d
.
2 -(p+l)
(2,21)
dy [yh6(y17 = (l-y )
2p~y •
,
'
....
8
Treating the denominator of h (t) in the same way and combining
2
results we get
where
2 v/2
;: 2 (v-2}/2
. 2(~H (~H y
g(y) = (l-y)
+ vy J (l-u )
du +
. .
[(v.2)/27~
o
-
From (2.8), (2.10) and (2.22) we have
Jo,
(2.24)
g(y)/g(-y)
<
k
ll, g(y)/g(.y)
>
k
•
Hence, since g(y)/g(.y) is monotone increasing with respect to y
where y*= y*(k,v) is the solution for y in g(y}/g(-y) = k. Finally,
2
2 .!.
putting t*= t*(k, v, r ) for the solution for t in y*= t/~(v+t )2 we
have
(2.26)
as was to be shown.
More specific details of the computation of the
significant t ratios in Table 1 are given in section 6.
EXAMPLE 1:
To illustrate suppose the following:
A standard
treatment is modified in the hopes of producing an increased yield.
An
e~eriment
is run giving r yield observations xll"",xlr and
x "",x for the new and control (standard) treatment respectively.
2l
2r
9
It can be assumed that the respective sets of data are random.
independent samples from normal populations with means III and 112
and with the same, but unknown, variance a2 • It is required to
decide whether the new treatment is signj,iUcantly su:perior (in yield)
0ll:
significantly superior than the standard; whether to generally recommend it as the superior or to withhold such a general recommendation.
Type-I-like errors of recommending a non-superior new
treatment (dl!a ~ 0 , where 5
= Ill -
1;102) are thought to increase in
seriousness in direct proportion to the degree -5 of inferiority
involved.
Type-2-like errors of failing to recommend a superior new
treatment (d /8 > 0) are similarly thought to increase in seriousness
o
in direct proportion to the degree of superiority 8 involved.
any absolute difference 50
treatment with 8
= -8o
= 18/,
F01
recommendation of an inferior new
is considered k' times as seriaus: as; the cor-
responding failure to recommend a superior new treatment with 8
= 80 •
rn the averaging of risks it is desired to weight risks symmetrically
at 8
= t80
for all possible differences 80 ~
° with weights decreasing
1..
2
with respect to 5o as
. given by a normal density for T = 8/(2a /r)2
2 A minimum-average-risIt rule is
WJ.°th mean zero and variance. r.
o
required which would be invariant with respect to any changes of
scale or location in the observations.
Because of sufficiency and invariance considerations the required
rule can be restricted to depend on the observations through only the
t ratio
not
10
where xl and
X2
are the respective sample means and s2 is the pooled
within-sample variance estimate
(2.28)
s
2
=
2 r
~
2
~ (x.. -X ) !2(r-l)
i=lj=l
J.J
i
2
The required rule is then given by (2.26) where t*= t*(k, v,-y ) with
2
2
k = k', v = 2(r-l) and 'I = 'I o •
From the roles they pla.y the parameters k, v, and '1
2
determining
the minimum-average-risk significant t values may be usefully',termed
the
~
or error seriousness ratio, the error degrees of freedom and
the risk-weighting variance ratio respectively.
Before going on it
is of interest to note in Table 1 that a loss ratio of 100 (log k
= 2)
infinite error degrees of freedom (v = 00), and a risk weighting
variance ratio ~f 3 ('12 =3) give a t* of 1.987 close to that 1.960 of
a .025 le¥el test of H :T ~ o.
o
3. A Related Three-Decision Student-t Problem.
Given a
similar observed t value, a problem related to that of section 2 is
one of choosing between the three common decisions
dO: decide
IT I < 6,
d : decide
l
where, as before, Do is some sma,ll
T
~ 6.
and d : decide
2
l.inspec~:t'ied., posit;J.ve
T
~ ,.;;.~
,significant value.
(In the literature the same problem is often very loosely regarded
as that of testing H : T = 0 with the alternative H : T ~ 0, the null
o
l
decision thus being d : decide T = 0, the other two decisions~being
o
llJ%tl;l)ed t(}getm.er as the .one alternative d : decide T .;, 0) ..', I
l
Our second result is a Bayes rule for this problem with respect
to a similar linear loss function
11
t 2 (T)
= t(T J
,0-
ti(T) = t(T,
d )
l
0
(3.2)
d )
=
=
=
\f 0,
T = 0,
l coI T/, T ~ 0,
{"liT!'
0,
{
.
°
Cl
:
T
T
'
T ~
> 0,
< 0,
T-
0,
°
>
='
where Co and c i are positive constants such that c - Co > Co and
l
with respect to the same normal prior density (2.4) for T. The rule is
10
~(t) = (¢~(t) ~i(t) ¢~(t» =
(3.3)
o),lt/ < t*
(0:1 0), t
( (00 ~), t
> t*
<-t*
where the significant t ratio t*= t*(k, v, y2) is the same as that
of the previous section with the loss ratio now given by k
and where ¢~(t)
= ° or
decision d., 1
=0,1,2.
~
~
= cllco
- 1
1 denotes the not making or making of the
This result can be obtained as follows
First the three-decision
subset'system
can be expressed as the restricted product (the full product less
empty intersections, see Lehmann~!7) of two component two-decision
subset systems like that of the previous problem in section 2, namely
+
+
Component system for +T: wo : T < 6, w : T =
>A
l
Component system for -T: w-: - T < 6, w- :-T:> A.
o
1
=
12
'e
Thus
+n -'
(J,)0
=0
(.l)(.l)
, (.l)l = (.l)l+(lI(.l)0- , (.l)"~
0
+n -
The intersection (J,)l
= (.l)0+() (.l)l-
0\ is excluded through bsU1g empty. Put in other;:
words, each of the main decisions is equivalent to two joint component decisions
the joint decision d+ with d - is excluded through having mutually
l
l
incompatible components; dO being the decision T(£ (.l)~; 0 = +, -;
i
i
~
= 0,1.
Second, by putting k = c - c and k = c the losses for ~he
o
0
0
l
l
main decisions can be expressed as the sums of losses for its component
decisions as given by the two-decision loss function (2.3) in the previous section.
Demonstrating this
L (T) + L (-T)
o
0
=
k
o + k o (-T)
= c 0 ITI, T<
0+
=
o
T
0
k1ITI+kO(-T)=
(3.8)
L (T) + L 0
l
L
0
(-
T) =
(T) + L (-T)
1
0
.
+ 0
=
0
0
+ 0
=
0
0
+
k T
>0
\
Ci IT1 , T<O
0
k
T
= cllTI, T = 0
0
klTI+
1
=
T=O
0
= co ITI,
+0
0
1
I--rf = c 1 T
,
= Li(T)
T>O
T
<
0
T= 0
+ k 1 I-TI= c 1 T , T
>
0
2
:; L (T)
2
.
13
Next, any rule
et.2 (t) for the three-decision problem can
also be expressed in terms of two component two-decision rules\ For
this purpose it is convenient to first re-express the two-decision
function ¢(t) in the two-element vector form
where ¢o(t)
=1
= ¢(t).
- ¢(t) and ¢l(t)
In this form, for example,
the Bayes rule ¢~(t) of the previous section appears as
=
~(t)
[
(01),
t<t*
(1 0),
t
> t*
With this vector notation (3.9) for the two-decision function and
(3.7) of the three aecis1ons' 10 mind we cen write
the components
,
\
2
et. (t) = (¢~(t) ¢i(t) ¢~(t»
(3.11)
(¢~(t)¢~(t) ¢~(t)¢~(t) ¢~(t)¢~(t»
where ¢~(t)
ex. ex
d.,
~
= +,
=0
-j
i
or 1 denotes the not making or making of decision
= 0,
1.
Now, working with the average risk of
JJ
([)
2
A(s, rA. )
=
co 2
L:
.
-co -00 ~=O
provided that the condition
L~(T)
~
et.2 (t) we have
¢2(t)
f. dt g. dT
i
14
-co < t < co
is satisfied.
(Following Lehmann
17_7 this
may be termed a com-
patibility condition since if it were not satisfied the component
rules would give incompatible decisions). Assuming pro tern that this
is satisfied, the average risk readily reduces to
where the component average risks
A(f, ~(t», a =
same functions of t and -t as was A(s, ¢)
section 2.
= A(s,
+, -, are the
¢(t»
of t in
To minimize (3.14) it is sufficient to minimize the
component average risks separately subject still of course to the
compatibility condition (3.13).
From the result of the previous
section as expressed in (3 ..10) the component solutions are
[( 1, 0), t < t*
+
t;(t) =
Since ¢~*(t)
=1
(0, 1), t > t*
(l 0), -t < t
and
¢;(t)
=
{ (0 1), -t > t
*
*
only in t > t*, ¢~*(t) = 1 only in -t > t*= t < -t*,
and since t* is positive (from kl > ko ) we do have ¢~*(t)¢~*(t) = 0 for
all t, that is, the solutions are compatible. Thus the required
Bayes rule is given by
2
~(t)
=
=
as was to be shown.
(¢~*'t)¢6*(t) ¢~*(t)¢6*(t) ¢~*(t)¢~*(t»
t
(lO 0), (t < t*)(t.> .t*) -It I < t* '
(0 1 0), (t > t*)(t > -t*) = t > t*,
(0 0 1), (t < t*)(t < -t*) = t < -t* '
15
EXAMPLE 2:
To illustrate suppose:
Two sample of yields have
been observed as in example 1 except that now they are for two
new treatments.
It is required to decide whether the first can
be recommended as the superior (in yield), whether the second can,
or whether to withhold recommendations on both.
Errors of
making a wrong recommendation or of failing to make an appropriate
recommendation are again scorable in direct proportion to the degree
of inferiority or superiority involved respectively.
For any
difference that may exist the error of a wrong recommendation is
considered c' times as serious as that of just failing to make the
right recommendation.
The requirements for weighting of risks and
invariance are the same.
The required rule is then given by \3.16)
where t is as obtained (2.27) in example 1, t*
= c'- 1,
= t*(k, v,
y2 1 with
= 2(r
- 1) and y2= y2. The subtraction of one from thu
o
loss ratio c' will be trivial and unnecessary in most practical sitk
v
uations with c' not small.
The need for it here and not in example 1,
it may be said, comes from the fact that a wrong recommendation now
includes implicitly a failure to make an appropriate recommendation.
4. A Symmetric Multiple
~omparisons
Problem. Given N = n(n-l)/2 t
statistics of the form
(4.1)
t
pq
= (~P- ~ q )/
~
SA ,
~
pq
E
N,
E
N,
with non-centrality parameters of the form
(4.2)
Tpn
"3.
=
(~ - ~ ) /12 a~ ,
P
q
.
pq
where N is used for convenience to denote the set of pairs
{1,2; 1,3; . . . ;(n-l),nl as well as its size, a common multiple
16
comparisons problem is that of choosing between the three decisions
simultaneously for all pq
€
N, where the subsets are of the previous
(3.1) form
1
(J)
(4.4)
pq
The joint density of the t
pq
IS
T
pq ~6.
- ,
(J)2
pq
T
< - 6.
pq =
is that which would result, for
"
" are normal indeexample, from the common assumptions (a) ~l""'~n
pendent variables with means ~l""'~n and the same but unknown
variance
a:
and (b) s" is an estimator of cr" with v degrees of
~
~
~
2
freedom such that u = s fa has the X -related density (2.13)
"
~
"
~
independently of ~l"::'~n' If we put l for any vector of (n-l)
orthogonal normalized comparisons among the estimates "~., that is,
1
"
I
l = A.!:: where AA = In_l"
"
"
Al== 2 (1
peing a vector of ones); and
"I
.!:: = (~1'" ~n)' , then the density of the t pq l s can .!lje represented
conveniently in terms of that of the (n-l)-element t vector
.L
depending on the corresponding (n-l)-element non-centrality parameter
vector
(4.6)
where]
= A.!::
and ~
= (~l"'~n)l.
This can readily be expressed as
17
(4.7)
2,
where w(ulv) is the X -related density (2.13).
Our main result is a Bayes rule for this problem with respect
to generalizations of the linear additive loss functions and normal
prior density used in the previous sections.
More explicitly, the subsets
00 ,
0
wl""'~_l
of the multiple
comparisons problem are the non-empty intersections
j
w.1 =.pq€ N ~pq,
pq
(4.8)
j
pq = 0,1 or 2, i = O, ..• ,M-l,
of the subsets of all the component three-decision problems involved.
The decision system consists of the M corresponding decisions
i = 0, ..• , M-l.
d.
1
For example, thinking of the component subset systems (4.4)
in the form
(4.10)
~ : (I~ _ ~ 1< 61~,
pq
P
q
(Where 6' =6
(4.11)
i2 era)
1
- 61)
pq : (~~-A'
q -p )(l
pq : (~<...~
p=' q'
00
and using the corresponding more graphic notation
(~),
(q, p), (p,q)
012
in place of wpq ' wpq ' wpq respectively, the M = 19
multip~e
subsets in the case n=3 may be developed as in (4.12):
comparisons
18
(1,2)
(~)
(4.12)
(2,1)
0
OJ
(bl.) C~,2)
OJ~= (3,1,2)
(2,3)
(l)5 = (2,1,3)
(J.).e; =
(2,3)
(l) = (j,.gz1)
(l) = (3,2,1)
OJ. ~=
(l) = (3,2,1)
(3,1)(3,2)
(3,1,2)
-
(2,1,3)
= (3,1,2)
11
OJ.
JO
(l)12= (2,3,1)
(2,3)
(M)
ill
8
7
9
2 = (1,3,2)
w'4 = (1,3,2)
1 = (2,3,1)
= (1,2,3)
OJ
(1,2)
(l) =(1,2,3)
~4=
13--
(l) = (1,3,2)
(1,)(3,2)
(2,3)
(1,2,3)
~
('2,1,3)
(l)r,=
I·
(l)16= (1,2,3)
(l)18= (1,2,3)
In general, following ~uncaQ'L:~, the notation (i, j, k, .•. ) may be used
to denote subsets in which the corresponding means
~.,
J.
~.,
J
~k""
are ranked in significantly asoend,ingcwderfrom left to right (1.e. with
differences I~ ~ 6' ).except that subscripts ~derscor.ed by a common line
denote pairs of means for·which thegifference 'is not significant
let
Thus (l)z= (1,3, 2) is~he subset in Whi.ch Jl l ' ~s
significantly. le~s than ~2 but ~l and ~2 each do not differ significantly
(1.e.
from ~3'
<!:i').
The remaining 23_ 19
=~
intersections, e.g. (2,1)(~)(3,2),
of the component subsets are not not included in the multiple comparisons
system through being empty.
Choosing the elements of ! .. as
T2
= (1J.1 + 1J.2 - 21J.3)/~ rr~
T
l = (~l- 1J.2 )/"/2
0'11 for abscissa and
for o;dinate the parameter subsets may be
represented as shown in Figure 2 of
L-i7,
where the vertical lines are
12 =: 6, the lines from top left to bottom right are T13 = ~
T
from bottom left to top right are T
23
= ~ !:i.
!:i and those
19
Referring back for a moment, the need for the definition
(2.1) instead of (2.2) for decisions of our initial subcomponent
problem can now be seen more clearly in the formation of the subsets
If' (2.2) were used the six subsets (1)1' m~/
(4.12).
m,'
(1)l and UJ..3 of the form (i,j,k) would be eliminated.
.
~~
These however
are useful members of the system, hence the need for some such definition
&8".(2.1) to retain them.
The size M of the multiple comparisons subset system increases
rapidly with n the number of means involved.
n
= 4,
In the next case
for example, the numbers of the various forms of subsets,
using the same notation, are
(4.13)
(1,2,3,4)
. 1,
(i,j,k,l)
(W,k,i)
· · .12,
(i,j,k,,o
. (i,j,k,i)
(i,~l£zl)
(i,j,k,1)
making M
= 183
· . 4,
· · .12,
· · .24, (i,j,k,t) · · .12,
-_ · · .24,
· · .12, (i,j,k,l)
· · .24, (i,j,k,,o · · .24,
..
(i,j,k,D
.4
(i,j ,!:d)
.12
n
.12
(i:1,~)
...6
(i, j, ~:,
--;;..~--
in all.
The losses are defined as the sum of the losses (3.2) for each
of the component decisions involved; that is
(4.14)
L~(T) = Ln(T,d
l
-
-
1
)
=E
pq€N
2
L
(T
ji pq
pC!
), ji
pq
= O,Lor
Thus, suppose for example in the case n=3,~/~rr
2;,1. .. O,.,.,M.l,
= (~1~2~3)'/yr2-rr
represents the expected standardized yields of three manurial treatments
on a particular agricultural crop.
•
Then the loss L~4.(::) at
~/vf2rr = (10,12,,8) I incurred by the decision d
' .! e(1,2,3) is
14
20
Li4(!) = L~ (1"12) +
(4.15 )
= L~(-2)
=
+
o
L~( 1"13)
L~(2)
+ 2c
+ L; (1"23)
L; (4)
+
+ 4c
1
0
= 4k0 enters, it may be said, because the
o
has failed to recognize the 4-unit superiority of treatment
The third contribution 4c
decision d
14
two over treatment three. The second contribution 2c
= 2kl + 2ko
l
not only fails to recognize
enters, because, in similar terms, d
14
the 2-unit superiority of treatment one over treatment three,incurring a
ioas of 2ko , v it '.coIll'llits·:the:'add1tional'mo:"e serious error crt ranking ,.treatment three·sbovetreatmen~one·1ncurring.an additional loss of 2k _No loss
l
is incurred by.tlie tirst componerit~dec~sionbe¢~us~ di4 ranks treatment
two correctly above treatment one.
The prior density for averaging risks over all peinte in the
(n-l) dimensional space for 1" is the simple normal density function
-00< 1" < co
(4.16)
This is one
w~1ch
would result, for example, from assuming that the
means J.1 , ... ,J.1 have independent normal prior distributions with the
l
n
same mean Q and same variance ~2 = r2~~ •
1..1
,..
From the additive losses assumption it follows as before that
the average risk for any decision rule ~n(!)
= (¢~(!)"'¢:-l(!»
may be expressed as the sum of average risks for component threedecision rules ~pq(t)
~
-
= (¢pq(t)
¢Plq(t)
.0 -
¢~q(t
~
pq
», provided again that
21
the component rules are compatible.
The steps may be written
(4.17)
dt
.d"l
_. Sn·
0000
If
- -
=
E
pq€N
-00-00
=
•
2
L~(Tpq )
E
j=o J
.
g .d'T
¢l:jq(t) f n • dt
- n
-
E A(g ~ (t »
pq€N
n
pq
The compatibility condition may be written as
J( ¢~q (t)
(4.18)
pqEN
pq -
= 0,
j
pq
= 0,1
or 2,
-00
< -~'.
t <
-
00- .
for all products leading tOincomps.tible·'·decii"s1ons. not in.eluded in
~ di ; i = O, ••• ,M-l} and is required in proceeding from the first to
the second line of
(4.17).
It then follows as before that the Bayes rule
(4.19)
for the multiple comparisons problem is formed by the products
(4.20)
of the elements of the Bayes rules
j
i
pq
= 0,1
or 2
gq(!) minimizing
22
(4.21)
pqeN
provided these are compatible.
The first step in deriving ~~q(!) is practically.1dentical~iththat
(3.12) to
(3.14» in section 3. Thus
(4.22)
where, dropping the superscripts pq+ or pq-, ~(!)
= (¢o*(!)
¢l*(!»
now minimizes a eubcom,pcnent- average risk of the form
0000
(4.23)
A(~ .l!(!) ~ JJi~oLi (Tpq)¢i (!) f n (!I.~) d! ~n(1:) dT
-CD
-co
The elements of ! may be chosen so that Tpq is the first element T
l
of T. The compatibility condition
(4.24)
must again be met.
The work of minimizing (4.23) follows closely that of minimizing (2.6) in section 2 except that now the sample 'and paramet~r;-'spaces
have additional dimensions, (n-2) each.
Dropping the subscript pq from
Tpq the steps follow through with obvious changes till we get to
(4.25)
where the first integration is With respect to !2 defined as the last
n-2 elements of T
= (t !2)'.
This appears in place of h (T,t) as in
3
(2.14) before which may now be denoted by h2 (T, !,
3
with respect to !2 we get
v). On integrating
2 2
2
TJ e -i[(ut- T)2+ T2ly2-n-l
E u t./(l+l 17
i=2
co.
(4.26)
n(T t )...r
h3
,_,v ~
~
o
1
2
.un-luv-le -'2vu du
co
7 , I (v'-l) e _~v'u,2du '
...... TJ e _~[(U't'_T)2+ T jy-uu
2
2
....r
o
where t
=t l
is the first element of t and is thus t
-
(4.27)
and
Vi
u' = u/R, t
l
Vi
/Lv
= v+n-2.
Thus
(4.28)
2
= !it, R =
Je-~gut'-T)2+
co
h;(T,!,v) c£T
= t pq ,
n-l 2
2
+ E t.l(l+l 17,
i=2 ~
2
T /1:7 u1jr(ulv 1 )du
o
Making a direct application of the remaining derivation in section 2
it now follows that
~(t) =
where t*= t*(k,v l
,72 ) is the same significant t value as before except
that its degrees of freedom are now
VI
= v+n-2.
Since t*/R > 0, the compatibility condition (4.24) is met and
applying (4.22) the component average risk (4.21) is minimized by
24
(1 0 0), ( t'
pC! < t*)
(o 1 0), ( t'
pC! > t* )
~C!(~) =
(4·30)
(0 0 1), ( t'
pC! < -t*)
=(
=(
=(
t
pC! < t*/R),
t pC! > t*!, R},
t pC! < .. t*!R),
= RtpC! for all PC!~' Again since t*/R > 0, the preceding
pC!
compatibility conditions (4.18) are met and applying (4.19) the Bayes
where t'
rule for the multiple comparisons problem is given by the simultaneous
application (4.20) of all N
= n(n-l}/2 of the three-decision Bayes
rule s (4 .30) •
EXAMPLE 3:
Suppose: n samples of yields like those of example 2
have been obtained for n new treatments.
For each and every pair (a,b)
of treatments it is reC!uired to decide whether a can be recommended as
the superior, whether b can or whether to withhold recommendations on
both.
Losses are scorable with respect to each pair of treatments as
in example 2, the loss ratio c l being the same for all pairs» and are
additive in giving the losses for each of the joint decisionsto which
they contribute.
Risks are to be averaged with respect to a normal
independent prior density for each of the means
.
2 2/r.
the same mean and same varJ.ance
y o(j
~l""'~n
each with
An invar i an t ru1e is required
as before.
Because of sufficiency and invariance considerations the required
rule can be restricted to depend on the observations through only the
t vector
(4.3l)
~
= A!/(2s2/r}21
where A is as defined before for (4.5), x
= (Xl, ••• ,Xn)'iS
the vector
of sample means and s2 is the pooled within-sample variance estimate
25
2
(4·32)
s
n
r
_ 2
= ~ ~ (xij - xi) /n(r-l)
i=j j=l
with n(r-l) degrees of freedom.
The required decision rule is then given by the simultaneous
application (4.20) of (4.30) where t* = t*(k,v' ,y2) with k
= C'-l,
v' = v+n-2 = n(r-l)+n-2, and y2= y2 and where t'
can be obtained by
o
pq
analysis-of-variance type steps as follow: Put 5 , 5 ,5'
and 5
pq
pq
e
t
for the treatment sum of squares, the sum of squares for the pq
difference, the residual sum of squares for the pq difference and the
error sum of squares
n
- ~ -x·)2/2,.
5t ::::" r.~ i - C, 5pq= r(x'
p
q
J.=l
r 2
n
5'pq= 5t - 5pq ' 5e = (~
~ xiJ-:,··C)
i=l j=l
x
(4.33)
- 5t
n r
2
respectively, where C is the correction term( ~ ~ XOj) /n~.Let
i=lj=l J.
2
s,2
pq denote the pooled estimate ot a obtained as
LS
Then t'
(4.35)
t'
pq
pq
S,2 =
+ 5' /(1+y2'7/ v'
pq
e ~q
~
may be obtained as the square root of the variance ratio
t,2
pq
= 5pq/ s ,2pq
being given the same sign as
xp - xq
A more convenient rule for application can be obtained by
expressing the inequalities t,2 S t*2 in the form d2 S d*2 where
pq
pq
= Xp - x '
d* is a least significant value for the difference d
pq
q
2
2
From t' pq = t* we get
26
(4·36)
t*2
= Spq/6,2pq = v'Spq/IS
LUe
+ (St- Spq )/(1+y2)7
0v'S
pq
But Spq =
2
I'd~q/2, hence this gives dpq
(4·37)
d*
~
= (~t;[Se
+
= d; where
St/(1+y~7/LV'+t;(1+Y~17)!
From this and a check on signs it follows that the multiple comparisons
Bayes rule is given by the simultaneous application of the rules
(q(~) =
(4·38)
(1 0 0), Id l < d.
pq
(0 1 0), dpq > d*
dpq <-d* .
(001),
5. Discussion: 5.1 On the additional error degrees of freedom.
2.l.
The emergence of t' = d /(2rs' )2 with v' = V+n-2 degrees of freedom
pq pq
as the component test statistic in the multiple comparisons solution
may be surprising at first but less so after due consideration.
giving
~l""'~n
In
identical independent normal distributions with
variance
y2a2/r for risk-weighting purposes the residual sum of
o
n-l 2
2 n-l 2
squares between treatments S = r E Yi = rs
E t , in example
pq
i=2
i=2 i
3 for instance, is given the distribution of (1 + r.o2 )rr2X2n- 2' On
this basis
2
s' pq
fQ
2
= LP.
e + Spq/(l+y0 17/v'
becomes the appropriate estimator for rr
result is as might be expected.
2
.
in place of s2
= Se Iv
and the
In practice a user might sometimes be
reluctant to depend on the prior distribution assumption to this extent
however, and might even want, see subsection 5.3, to use S' or better
pq
.....·2
2
St to decide on an appropriate value y
for y.
In such cases it
would seem good sense to use a modified rule based on t
t' .
pq
pq
instead of
This would consist of simultaneous applications of
~pq{j)
(5.2)
-
(1 0 0),
d
pq < d*
(0 1 0),
dpq > d*
(0 0 1),
d
pq < -d*
where d* is the least significant difference
with t*
. . .2
= t*(k,v,y
) based on v degrees of freedom.
5.2. On the independence of the least significant difference and
n.
By far the most striking feature of the multiple comparisons Bayes
rule is the practically complate lack of dependence of the least significant difference d* on n, the number of means involved.
(The
dependence of d* on n via the estimation of error as discussed in the
previous subsection is a relatively trivial side issue here and decreases
to 0 as v increases to
00).
This is a direct consequence of the
additive losses assumption similar results of which have also been
treated by Duncan ~17 and Thompson
iIi7
form and context, by Lehmann ~17.
In the past a rule of this type,
and,in a far more general
with d* not increasing with n, has been considered more or less unacceptable. \ The main basis of objection has been the rapid increase
in its so-called n-treatment s1gnificance level (Duncan ~17 and ~L7)
or its experimentwise error rate (Tukey
= PLrejecting
with respect to n.
[i17
J,
Hn IHn
and
[f.§.7)
28
To illustrate, a non-increasing least significant difference
of d* = 1.960
"f2',
(J,.
IJ.
= 2. 77
(J~
IJ.
in the case v
= 00
gives the experiment-
wise error rates (found as upper-tail probabilities P~> 2.717 of the
range
(5.5)
Clu)
a2
= .0500, a3 = .1223, a4 = .2034, ... ,
a20
= .9183.
The possibility in this case of wrongly rejecting the homogeneity
hypothesis for 20 means, for example, with a probability of 91.83
per cent, may at first appear to be unacceptably high.
As a result
procedures have been proposed with increasing significant differences
aimed at suppressing the rapid increases in a.
n
These have varied
considerably from rapidly increasing significant differences such
as, (in comparable cases, dropping the factor (J,.)
IJ.
(5.6)
2.77, 3.32, 3.63, . . . , 5.01,
to more slowly increasing ones such as
(5.7)
2.77,2.92,3.02, . . . , 3.47,
depending on the relative importance attached to experimentwise
error rates and the degree to which they should be suppressed.
The
differences (5.6) termed honest significant differences by Tukey ~117'
~1§.7 suppress all of the experimentwise error rates to
.0500.
The
differences (5.7) proposed by the author ~l7 and ~~7 suppress them
to much less conservative so-called levels based on degrees of freedom
obtained as a~
=1
- (1-a )n-l. A less conservative procedure yet
2
is the one due to Fisher ~~ which uses the same least significant
difference for all n (2.77 in the case above) provided that first the
homogeneity hypothesis H can be rejected by an F ratio test.
n
29
Now it appears that in a Bayes sense, provided the losses are
additive and other things (e.g. k and l2) are equal, the same least
significant difference is optimum no matter how large the number n
of treatments involved.
(From Lehmann's work ~17 it is clear that
this would also apply under other optimality criteria such as, for
example minimax.)
The high experimentwise error-rate of 91.83 per cent
in the case quoted might well be worth tolerating for instance,
because, it might be said, of the relatively low prior probability of
the hypothesis H involved and its relative unimportance among so
20
many others.
The inverse form of dependence of the least significant difference
2
d* on the risk-weighting variance ratio.l may do much to reconcile
its independence of n with at least some of the common almost instintive urge to make it increase with n.
directly proportional to
values of v.
(1+1/l2)~ and
In
the case v
= 00
it is
approximately so for smaller
If, in the conduct of a large experiment the treatments
under study have a lower anticipated heterogeneity than those which
would have been studied in a more limited experiment with fewer
populations, a lower risk-weighting variance ratio l2 would be appropriate and hence would be a larger least significant difference.
Such
a situation could often arise in practice, and, if l2 is varying in an
interval of small values this could make a substantial increase in
the significant difference with an increase in n.
however, the reverse situation could also arise.
On the other hand,
In selection
experiments, for example, the treatments under consideration maybe.
the top n performers as assessed by experiences in previous trials.
Here, the larger the number of treatments it has been possible to
30
include in an experiment, the larger will be the appropriate
r2
and
hence, the smaller the least significant difference.
!
5.3. A practical adaptation of the Bayes rule. In the complete
absence of prior criteria for choosing r 2 , the user might sensibly,it
would seem, obtain an estimate of it from the variance ratio
1
F
= -=-1
n
n-l 2
1::
i=l
2
Yi/s "
1 St / s 2 in example 3),
(= n--l
IJ
employed in the preliminary F test of Fisher's least-significant-difference procedure. Since the ratio of the corresponding expectations is
2
~
2 "2
l+r he might for example put y = F-l,enter Table 1 with r = y. , and
use the simple direct rule as given (5.2) in subsection 5.1.
It is
of considerable interest to note the closeness of Fisher's earlier
procedure to this type of adaptation.
Both rules depend on a pre-
liminary inspection of the F ratio.- In
F1she~
rule a big or small
F ratio leads to the uee of an independently chosen least significant
difference on a go no-go basis.
In the new rule a big or small F ratio
leads to the use ofa small or big least significant difference on
a continuously related basis.
5.4. Concluding remarks. As presented the model is limited
to
a class of symmetric problems in which the loss and prior probabilities
are invariant with respect to all
n~
permutations of the means involved.
Many problems in practice- however are naturally taken to be symmetric in
this way.
Within this class, the assumptions of linear lesses and
normal prior densities would seem directly appropriate for some
problems and useful at least as good approximations for many others.
From the given development it is fairly clear that similar rules
for a wider class of less symmetrical problems can be obtained leading
31
to ;the use of different significant t ratios for each of the component
or even each of the sUboomponent problems involved.
Development and
discussion of these are deferred to a further paper.
A most interesting point is one raised (in private correspondence) by Professor F. J. Anscombe following out of the type of
discussion
in subsection 5.3.
In addition to providing an estimate of
the prior variance y2 and therefore a means of of rejecting an
the data may provide evidence for reasonably reassumed value y2,
o
jecting other assumed features of the prior density.
In such a
situation a change may be required to another prior density such as
one for example with one or more different variances.
In conclusion, it is worth repeating, the most important result
dtscueeedin subsection 5.2 namely the independence of the leastsignificant-difference d* and the number of components problems
involved, depends only on the additivity assumption for the losses.
It is independent of the form of the component loss functions and of
prior density assumed.
It appears further that the same result would
follow even if the class' of component problems were extended to
include all contrasts among the means as considered by Scheffe
LIll.
Thus in a symmetric situation, for example, the same significant t
ratio would be appropriate whether it be desired to test just one
comparison chosen ~ priori,the set of all n(n-I)/2 comparisons in the
multiple comparisons problem or the set
o~
all contrasts.
The addi-
tivity of losses assumption on which this critically depends appears
to be a reasonable one, and appropriate to many practical situations.
6.
Computation of significant t ratios.
In computing the values
in Table 1 the ratios g{y)/g{-y) in (2.22) may be simplified first to
gl{y)/gl{-y) where
~ + Y sin -1y ~ ny/ 2
( 1 - Y2 )2
,
v = 1,
v
= 2,
2{1_~)3/2+ 3y2{1_y2)~+ 3ysin- l y + 3ny/2,
v
= 3,
(l + y)3/{3 + y),
v
= 4,
(l + y)4/{5 + 4y + y2),
v = 6,
{1+y)8/{429+1384Y~20~;2+1776y3+9l5y4+264y5+33y6),V~14,
v =
00,
and f and F are the standard normal density and cumulative distribution
functions respectively.
(6.2) in the case v
These follow readily from (2.23) except for
= 00
which can be obtained as follows.
2 1
2
2
Putting y = z/{v+z )2 and thus (l-y ) = l/(l+z Iv) and
dy
= dZ/~V~{1+z2/v)3/:?
g{y)
in g{y) in (2.23) we get
1
= {1+z2/v )V/2
+
2z 1
(l+z /V)2
l
z
du
(l+u Iv)
-~2~"'(-V"":"'+1=-')~/r:::'2
0
1
+
(nv)2Ltv-2)/g7~ z
Lrv-l)/g7~ 2{1+z2/v)~
Recalling that the probability density function of the Student t
distribution is
33
(6.4)
h(tlv)
!JV-l)/g]~
1
=-----
(1+t2/v) (v+l)/2
(nv) jJv-2)/?7~
we may write
2
1
ZLH(zlv)-l_7
I
g(y) cC (l+z /v)~ h(z Iv) +
2
I
+ .2.L~
I
(l+z /V)2
(l+z /V)2
~ (1+Z2/V ) h(zlv) + zH(zlv)
where H(zlv) is the cumulative distribution
= g3(z)
say,
z
h(tlv)dt.
J
Treating
-00
g(-y) in the same way we reach the result
(6.6)
which reduces to g2(z)/g2(-z) as v';;' co •
. Next., y*(k,v) or z*(k) is found as the solution of
gl(y)/gl(-Y)
=k
or i (z)/g2(-z)
2
as the positive square root of
=k
respectively.
Finally t* is found
(6.7)
Acknowledgments. Help from Professor Harold Hotelling and other
colleagues is gratefully acknowledged.
REFERENCES
L!7
David B. Duncan, "A significance test for differences between
ranked treatments in an analysis of variance," Virginia J.
Sci.. Vol. :2 (1951), pp. 171-189,
e
L?7
David B. Duncan, "On the properties of the multiple comparisons
test," Virginia J. Sci., Vol.3(1952), pp. 50-67.
[)] David B. Duncan, IIMultiple range and multiple F tests,1I
Biometrics, Vol. 11 (1955), pp. 1-42.
l"!±7
Walter T. Federer, Experimental design, MacMillan, New York, (1955).
L-17
R. A. Fisher, The design of experiments, six eds. Oliver and Boyd,
London (1935-1951).
l"§.7
E. L. Lehmann, IISome principles of the theory of testing hypotheses,1I
~ Math. ~, Vol. 21 (1950), pp. 1-26.
l"V
E. L. Lehmann, IIA theory of some multiple decision problems, I,ll
~
l"§.7
Math. Stat., Vol. 28 (1957), pp. 1-25.
E. L. Lehmann,
~ ~
l"27
II
A theory of some multiple decision problems, II,"
Stat., Vol. 28, (1957), pp. 547-572.
Jerome C. R. Li, Introduction to statistical inference, Edwards
Bros., Ann Arbor (1957).
[f.Q7 D. Newman, "The distribution of the range in samples from a normal
population, expressed in terms of an independent estimate of
the standard deviation," Biometrika, Vol. 31 (1939), pp. 20-30.
LI17
Henry Scheffe, 'A method for judging all contrasts in the analysis
of variance', Biometrika, Vol. 40 (1953), pp. 87-104.
[ig7
Henry Scheffe, "The analysis of variance", John Wiley and Sons,
New York (1959).
ilL7
G. W. Snedecor, Statistical methods, 5th ed., Iowa State College
Press, Ames (1957).
[i!J:.7
R. G. D. Steel and J. H. Torrie, "Principles and procedures of
statistics," McGraw-Hill, New York (1960).
e
517 w.
A. Thompson, 'A note on ranking means,' Ann. Math. Stat.,
Abstract 13, Vol. 27 (1956).
35
[i§.7
J.
w.
Tukey, "Comparing individual means in the analysis of
variance /' Biometrics, Vol. 5 (1949), pp. 99-114.
[f.17
J. W. Tukey, "Quick and dirty methods in statistics, Part II,
Simple analyses for standard designs," Proc. Fifth Annual
Convention, Am. Soc. for Quality Control,(195l), pp. 189-197.
fJ..Fi!
J. W. Tukey,
II
The problem of multiple comparisons," unpublished
dittoed notes, 396 pp., Princeton University (1953).