•
.L\NALYSIS OF PAIRED COMPARISOH DESIGNS
~TITH
INCOMPLETE REPETITIONS
by
John W. 1,IJilkinson
University of North Carolina
This research was jointly supportod by the
United States Air Force throug,h the Air Force
Office of Scientific Research of the Air Research and Development Command under contract
No. AF 18(600)-~ and the National Research
Council of Canada
Institute of Statistics
Mimeograph Series No. 147
.:..pri1, 1956
·e
ii
ACKNOWLEDGMENTS
I am deeply indebted to Professor R. C. Bose at whose suggestion
this research was initiated.
His continued encouragement and advice
have contributed substantially to its completion, and I welcome this
opportunity to indicate my gratitude.
I wish to acknowledge my appreciation for the financial assistance received from the United states Air Force, the National Research
Council of Canada, and the Institute of st:Jtidics of the University
of North Carolina.
'.
In addition, I wish to thank my Wife, Joan, whose patience, understanding, and assistance considerably aided my completion of this research.
.\1so, I am sincerely
appreciati~3
of the excellent coopera-
tion received from Miss Martha Jordan, Mrs. Mary Ann Taylor, and Mrs.
Hilda Kattsoff in their careful typing of the manuscript.
iii
TABLE OF CONTENTS
PtllGE
CHAPrER
[I.CYJWHLEDGMENTS
. . . . . . . ..
...
·...... . ....
INTRODUCTION
I.
II.
III.
IV.
•
DESCRIPTION OF THE PnIRED Ca1PllitISON DESIGNS
.....
SOME TEST PROCEDURES FOR EXAMINING THE RELATIVE RATING
OF t OBJECTS • • • • • • • • • • • • • • • • • • •
S01E LARGE-SlJiPLE PROPERTIES OF THE TESTS AND SOME
EXAMPLES TO CLARIFY THE TEST PROCEDURES • • • • •
iv
1
...
7
...
38
AN ANALOGUE TO KENDALL'S COEFFICIENT OF 11.GREEMENT FOR
THE CASE OF INCOMPLETE REPETITIONS • • • • • • • • • ••
56
.. . .. . ·. . • •
64
APPENDIX • • • •
BIBLIOGRAPHY • •
• •
• • •
·.....
80
iv
INTRODUCTION
1.nlhen quantitative measurement of treatment effects is not possible,
or is possible but is not practicable, a method of paired comparison of
treatments is frequently used.
Experiments involving paired comparisons
have mainly concerned the situation where each observer compares all
possible pairs of treatmonts or objects.
In certain types of experi-
ments, this may requiro an excessive numbor of comparisons to be made
by any observor.
For this reason, GxperimentQl designs, symmetrical
with respect to both observers and to objects, which do not require each
observer to make all possible comparisons will bo useful •
••
Historically, various procedures for roducing the numbor of comparisons requirod by each observer have boen proposod (theso procedures
usually boine sOMowhat dependent on advance knowledge of the troatments)
but apparently the first research on the construction of such designs
was carried out by Kendall
-/-9~.
-
His procedure was to assign pairs of
objects to obsorvers in accordance with tours around a ''PrGforence''
polygon whoso t vortices depict tho t objects considered in the oxperimont.
Later, Bose
-;-1-7 constructed
throe series of paired comparison
designs by making certain correspondoncos vnth known balanced incomplato block desiGns.
one of those sarios produced the samo serios of
designs as the tours around the preforence polygon considered by Kendall
-/-9-7.
Howevor, neithor
BOSG
nor Kandnll proposed any procedures
lThe numbers in squ1re brackets refor to tho bibliography.
·e
v
of analysis in thoir respective papers.
Tho main purpose of -this study is to consider thG problom of analysis of tho paired comparison designs developed by Bose and Kendall.
In
addition, further designs are obtained by constructing tho complementary
dosigns corresponding to the designs given by
BOSG
L-l_7.
The analysis of experiments involving pGired comparisons, for the
situation whore each observGr compares all possible pairs of treatments,
has been considored by savsral authors.
Thurstone L-16_7 proposed a
method of paired comparisons based on a model ,{hich is essentially dependent on the assumption that judgments concerning characteristics of
••
an object arc normally distrihutod.
- -
Mostellar /-13 7 has summarizod the
notions undorlying Thurstone's method and has extended its scope.
Ul thoir considoration of a mothod of pairod comparisons, introduced a
cocfficient of agrooment and a coefficiGnt of consistenco, and proposed
tests of certain null hypotheses based on those statistics.
Bradley and Torry
~2_7
considored a mGthod of pairod comparisons
based on a m::tthomatical model which is a generalization of the binomial
modol.
Using this model, thGY obtained maximum-likelihood estimatos of
tho true objact rntings and likelihood ratio tosts for cortain classes
of hypothosos.
They dorived tho distributions of certain of their pro-
posed test stJtistics and provided tables (lJtcr extGnded by Bradley
-/-3-7>
of these distributions when the nlmmor of objects is small.
L-15_7,
and BrIJdloy
Addi-
tional doscription of the mathod is givGn by Torry, Bradley, and Davis
L-S_7 has
discussed tho appropriateness of this model
vi
/-4 7, Bradley investigated the reliability
of the estimators of the true object ratings and the large-sample oval-
for paired compnrisons.
In
uation of the power functions of his original tests, comparing thom with
the analysis of vDriance and with a multi-binomial procedure.
In this study, an annlysis of the paired comparison designs dovoloped by Bose and Kendall is proposed.
Using the Bradley-Terry model,
likelihood ratio tests of sevoral classes of hypotheses are proposed,
and in certain situotions it is shown that significance levels for these
tests may be obtained from available tablos.
The construction and do-
scription of these tests are displayed in Chaptor II, While, in Chapter
•
III, large-samplo properties of the tests are discussed and somo examples
are presentod to clarify the test procedures.
In addition, in Chapter
IV, an analogue to Kendall's coefficient of agreement is given for the
case of incomploto repetitions nnd under certain assumptions is shown
to provido an
alt~rnDtivo
test for a specific class of hypotheses.
Chap-
ter I is devoted to describing the paired comparison designs and discussing tho procedure of constructing the complementary designs which are
listed in the
~ppondix.
CHAPTER I
DESCRIPTION OF THE PAIRED COMPARISON
1.1.
D~SIGNS
Introduction.
-------Since
B
large portion of this investigation is devoted to con-
structing and describing tests usable in connection with certain
paired comparison designs given by Bose ~1_7, it is deemed advisable
to present a clear description of these designs.
The designs have a
high degree of symmetry as they are balanced by numbers of comparisons,
objects compared, and numbers of observers on given comparisons.
Since
it is this symmetry that enables much of the ensuing mathematical manipulation, the following detailed description is quite pertinent.
1.2.
tit
Definitio~~f
paired comparison designs.
Suppose we have t objects which we desire to compare according to a
certain characteristic, and we have v judges available to perform the
comparisons.
(i)
The procedure of comparison will be as follows:
Each judge compares r pairs of objects (1 < r ~ t(t-l)/2),
and, for each pair compared, expresseS his preference for one
or the other object of the pair. (It may be desirable to allow
the judge to express no preference with respect to either of
the objects
fo~ing
the pair.
Hcwever, this possibility will
not be entertained for the main part
of our discussion.)
(i1) The pairs compared by any Judge are all different.
(11i)Among the r pairs compared by each judge, each object appears
equally often, say a times.
(iv) Each pair is compared by k judges, (1 < k :: v) .
(v)
Given any two judges, there are exactly
compared by both judges.
f
pairs which are
2
Designs satisfying the above conditions are
referred to by Bose
as linked paired comparison desisns.
1.).
Relationships among the parameters.
Using the usual design of experiments terminology, suppose we con-
sider each Judge to correspond to a treatment and each pair of objects
to a block.
Then if a pair is compared by a Judge, the block corres-
ponding to the pair may be considered to contain the treatment corresponding to the judge.
Therefore, if a paired comparison design as de-
fined in Section 1.2 exists, there must exist a corresponding balanced
incom~lete
block d6sign with v treatments, b
=
t(t-l)/2 blocks, such
that each block contains k treatments, each treatment occurs in
•
I'
blocks, and two given treatments occur together in ;, blocks. However,
the eXistence of the balanced incomplete block design does not in
general ensure the existence of the corresponding linked paired comparison design.
From the relationship among the parameters of balanced incomplete
block desjgnB,and from the definition of the paired comparison designs
given in Section 1.2, we can obtain the folloWing relationships among
the parameters of a paired comparison design:
e
(1,3.1)
I'
(1.3 .2)
b = t( t-l) /2,
(1.3 .,)
bk =
VI',
(1.3.4)
vcY,
k(t-l),
(1.3.5)
b > v
-
( 1.3 .6)
tc/ ? 2k,
(1.3.7)
I' ~
(1.3.8)
I'
to/2,
=
=--
or
,,( v-l) = r(k-l) ,
I'
> k,
A,
.?: 0:(0:+1)/2,
A ~ 0:(0:+1)/2.
3
·e
1.4.
Three series of paired comparison
d~s1ee!.
Series 1.
Bose obtains one group of designs by considering the case
cv,
= 2.
When this is the case, the parameters of the paired comparison design
can be written in terms of t -- the number of objects to be compared -and, in fact, these parameters are
(1.4.1)
V
= (t-1 )(t-2}, b
2
= t(2t - 1 ),
r : :; t, k
= t-2,
k :::;
2, a = 2.
The existence of (1.4.1) implies the eXistence of the ba.lanced incomplete
block design with parameters
(1.4.2)
v
= (t-l)(t- 21 b _ t(t-l) , r : :; t, k = t-2,
2
'
2
~
= 2,
which can be derived by writtng down a solution of the symmetrical
•
balanced incomplete block design
(1.4.3)
v
t't-1)
= b = ~---
+
1,
r
=k
:::; t,
~:::;
2,
and then deleting one block and all the treatments in this block.
Con-
versely, Bose has shown that we can obtain a solution for the paired
comparison design (1.4.1) for any value of t for which a solution of the
symmetrical balanced incomplete block design (1.4.3) is known.
four designs with
(i)
the folloWing parameters have been obtained:
t = 4,
v
= 3,
= 6,
b = 6,
r :::; 4,
k : :; 2,
A
= 2,
a:::; 2j
a :::; 2',
(11) t
:::;
= '5, k = 3,
A
= 2,
(11i)t
= 6, v = 10, b = 1'5, r = 6, k = 4,
1\
= 2, a = 2'
(iv) t
= 9, v = 28, b
(1.4.4)
5,
v
b
=
:;:
10, r
36, r
= 9,
k
= 7,
The designs for these have been listed in Table
e
ThUS,
A :;:
)
2,
0:
= 2.
1 of 11_7. For
facilitating reference at a later stage, designs (i) and (11 ) of
(1.4.4) will be listed here.
If we denote the objects to be compared
4
-e
by the natural munbel's 1, 2, ... , t, and denote the judges by J , J ,
2
1
... , J , then we can present the designs as in Table 1.4.1.
v
TABLE 1.4.1
----DESIGN NUMBlm
PARAMETIl:RS
JUDGE
--
i),
(1.4.4)
t ::: 4,
b :::
6,
k ::: 2,
------- a
( 11), (1.4.4)
t
b
:::
k :::
•
v ::: 3
J
1
(1,4), (1,3), (2,4), (2,3)
=4
J
2
(1,3), (2,4), (1,2), (3,4)
A ::: 2
J
I'
3
::: 2
:::
Dl!SIGN
PAIRS ASSIGNED TO A -Jffi5G-r---
(1,4), (1,2), (2,3), (3,4)
-
5,
v :::
6
J
1
(3,5), (2,4), (1,3), (1,4), (2,5)
10,
I' :::
')
J2
(2,3), {:3,4), (1,4) , (1,5), (2,'5)
3,
I'
=
2
J
(2,3), (3,5), (1,2), (4,5), ( 1,4)
Q; ::: 2
J
J
J
3
4 (3,5), (1,2) , (3,4), (2,4), (1,'5)
5
6
(1,2), (3,4), (4,5), (1,3), (2,5)
(2,3), (4,5), (2,4), (1,3), (1,5)
--
-
Series 2.
Bose obtains another series of paired comparison designs for the
situation where the number of objects t is even, say t ::: 2z.
When
this is the case, the z(2z - 1) pairs can be divided into 2z - 1 sets of
z pairs each, such that each object occur-s exactly once among the pairs
of a set.
Then, considering a balanced incomplete block design with
v* treatments, b* ::: 2z - 1 blocks, r * replications, block size k* , and
in which every pair of treatments occurs together in the same block
A* times, where each block corresponds to one set and each treatment to
one judge, a paired comparison design can be obtained by assigning to
each judge the sets of pairs corresponding to all blocks in which the
'5
treatment corresponding to the judge occurs.
This procedure ytelde
paired cmmparison designs with parameters
t : 2z, v ::
*
b
V ,
= z(2z - 1),
r
= zr * ,
k
= k*,
~
= z~* ,
a = r*.
Designs obtained in this manner and with the following parameters are
listed in Table 2 of ~1_7:
b _. 6,
r
= 4,
k
r
28, r
= 12,
= 12,
28, r
= 16,
v :: 3,
t
= 4,
= 6,
v
= 5,
b
= 15,
(iii) t
= 8,
v
= 7,
b
::
t
= 8,
v
= 7,
b
::
(i)
t
( ii)
(1.4.5)
(iv)
= 2,
= 2',
k
= 2,
= 4, r. = 9,
a =
4',
k
= 3,
1\
= 4,
a=
3;
k
= 4,
~
= 8,
a
/I
a
= 4.
It is of interest to note that design (i) of (1.4.5) is the same as design (i) of (1.4.4).
Series :3.
•
In a. somewhat similar manner, another series of paired comparison
designs is obtained With parameters
t
= 2z
+
1, v:: v', b
~.
where
b'
VI,
= z,
r', k l ,
complete block design.
t
:: (2z
~'
= z(2z
+ l)~',
+
1), r
= (2z
+ l)r', k
= k',
a = 2r',
are the parameters of a known balanced in-
The design with parameters
= 7, v = 3,
b::
21,
r
= 14,
k::
2,
~
= 7,
Q
= 4,
is given in Table 2 of ~1_7. It should be noted that the paired comparison designs obtained in this menner are the ssme as those considered
1.5. Some further paired
compar~E....E..eslgns.
For experimentation using paired comparison methods, the usefulness
of these designs would increase with increasing number of objects tj
hence,some merit would seem to be attaohed to the listing of further
designs for larger values of t.
Twenty-seven additional designs, with
6
4
<:.
t
<:
25, are listed in Appendix A.
To determine the pairs of objects,
or sets of pairs of objects, to be assigned to each judge, field plans
for corresponding balanced incomplete block designs displayed in
Cochran and Cox
£6_7,
and in Fisher and Yates
£8_7
were utilized.
Several of these new designs were obtained in the following manner:
Consider a known paired comparison design D with parameters t, v, b, r,
k, A,
G.
Then from this design we can construct a complementary de-
sign
D by
the b - r
taking as the pairs to be compared by judge u, (u = 1, ... ,v),
=
t(t - 1)/2 - r pairs not compared by judge u in the
original design D.
If we consider D and D together, then each judge
would compare b pairs and each pair would be compared by v judges;
hence, in D each pair will be compared by v - k judges.
b
= t(t
Since in the
- 1)/2 pairs, each object appears (t - 1) times, then in the
pairs compared by each judge in
D ea.ch
object appears t - 1 - a: times.
In D, any two Judges compare exactly r - )\. different pairs and I\. common
pairs, and hence b - 2(r - A) - A
compared by either Judge.
Therefore, in
exactly b - 2r + A common pairs.
mentary design
tl
D in
- 2r
+ A
pairs are not
D any two Judges will compare
Thus the parameters of the comple-
terms of the original design parameters, are
= t, VI = v,
A
l
=b
=b
bl
-
= b,
rl
2r + A, a l
= b-r,
= t
kl
=v
- I -
ex.
- k,
CHAPTER II
SOME TEST PROCEDURES FOR EXAMINING THE RELATIVE RATING OF t OBJECTS
2.1.
Introduction.
In this chapter several tests are developed for examining the rela-
tive rating of objects when a method of paired comparisons is employed.
The experimental procedure used is that prescribed by the paired comparison designs obtained by Bose ~1_7.
The tests are constructed us-
ing the Bradley-Terry model ~2_7 and the method of maximum likelihood.
2.2.
Experimental_ procedure and notation.
Consider a set of t objects (a , a 2 , ... , at)'
l
We wish to examine
these objects with respect to a prescribed characteristic x, and we desire to express opinions concerning the relevant status of the objects
with respect to each other.
Suppose we have available v judges, and we require each of these
judges to compare, according to a certain characteristic x, r pairs of
the t(t - 1}/2 !,ossible pairs of the set of objects (a , a 2 , ... , at)'
l
(1 < r
~
t(t - 1)/2}.
The r pairs to be compared by each judge will
be specified by the field plan of the appropriate paired comparison design employed.
Presently available designs are outlined in Chapter I
and are given in detail in ~1_7 and in Appendix A.
We shall refer to
one set of r pairs compared by a specific judge as an incomplete repe. tition or a complete repetition, depending on whether or not
r < t(t - 1}/2 or r
=
t(t - 1}/2.
When r
=
t(t - 1}/2, the situation
becomes that which is considered in detail by Bradley and Terry ~2_7,
Bradley ~3, 4,
5_7,
and Terry, Bradley, and Davis
f 15_7.
For this
reason, we will strive to employ similar notation to that used by the
above authors.
8
For each pair of objects compared by judge u, (u
= 1,
judge expresses a preference for one object or the other,
... , v), the
assi~ing
the
rank one to the object Judged superior a.ccord1ng to the characteristic
x, and the rank two to the object judged inferior.
Let r
denote the rank assigned by judge u to the 1-th object
iju
when it 1s compared with the j-th object.
Let
\a
the preference indicated by judge u for object a
(i ., j, i, j = 1, ... , t
j
= 1,
u
(
(2.2.1)
r iju
=
and
i
observed that
8
j
i over object a J'
Then
... , v).
~
1,
if
\ ai
2,
if
{a1 ~ aj I u},
)
t,
if objects a
~ ajl u~ denote
i
From (2.2.1) it 1s readily
are compared by judge u.
(2.2.2)
(i ., j
J
i, j
= 1,
... , t
j
u = 1J
••• ,
v) ,
when objects a
and a j are compared by Judge u. To handle the complicai
tion created by the fact that each Judge does not compare all possible
pairs of objects, we shall define
1,
if
8
i
0,
if
8
i
and a
J
are compared by judge u,
(2.2.3)
and a
J
are not compared by
Judge u,
(i
IJ,
1,J'c
l~,..... ,tjU =
n l1u = 0,
l, ... ,v), and we shall conventionally take
(i = 1,
"'J
tj
u
= 1,
"'J
Hence, for each Judge u we can define an ~~nce matrix Nu
v).
=
(n )
iJ
which is symmetric, has l's or o's for elements, and is specified by
the field plan of the paired comparison design used.
For example, for
design (i) of (1.4.4) we have the following three incidence matrices:
9
·e
N1
=
0
0
1
1 \
i
I 0
0
1
1
1
1
0
0
1
1
0
0
\
,
N2 =
0
1
1
o'
f 1
0
0
1
1
0
0
1
0
1
1
0
\\
,
I
I
(2.2.4)
N =
3
0
1
0
1
1
0
1
0
0
1
0
1
1
0
1
0
\
)
I
From the highly symn:etrical nature of the original paired comparison designs, as indicated in Section 1.3, these
incidence
ma.trices
possess the following interesting properties:
t
t
E
i=l
t
(2.2.5)
E
n iju
t
E
i=l j=l
t
E
i=l
t
E
j==1
=
2:
j==l
n
iju
= 0:,
n iju = ta == 2r,
v
E n iju
u=1
= 2rv
(u= 1, ... , v),
= kt(t - 1).
If, in addition, we conventione.lly define
NO ::: kI,
where I is the (t x t) identity matrix, then
(2.2.6)
v
E
u=O
N == kE,
u
where E is a (t x t) matrix, each element of which is unity.
To simplify notation, we will introduce the following conventions:
E, E, E, E and TI, TL TI, TI will indicate) respectively, single sums and
ijum
i
j
u m
10
·e
products with respect to the depleted quantity over its range, whero
= 1,
u
1, ... , v;
m
... , t;
denote
E for some i, where this i will be clear from the context.
j=l
jfi
j
... , t;
=
i :::: 1,
==
1, ... , g.
I: I
j
wi 11
t
E
i < j
=
i
will 1ndicatejrespectivel~dmuble sums and products,
and)1
i <:: j
1, ... , j-l;
j
=
2,
... , t.
t
t
E
E
t
and
i=1fi=1
11
and
E
ilj
will indicate)respective}Y,
.ifj
t
n
TI
Any departures from these conventions will be
i=lill=l
specified when the departure is inclz·red.
In addition, log and In will
be used to denote common and natural logarithmS, respectively.
.e
2.~.
Mathematical model (Bradley-Terry model) .
Suppose there exist numbers
(u = 1,
.... ,
1C]]l'
••• ,
rf
tu
corresponding to judge u,
v), where
rf
(2.~.1)
> 0,
i u-
E
i
1C
iu
(1
= 1,
= 1, ... , t·,
(u = 1,
... ,
u = 1,
... ,
v) ,
v) ,
and such that
(2.) .2)
= _rf.-j,,-u_ _
rf
(i
1 J,
i, j ::; 1, ... , t;
The conditions in
(2.~.1)
u
=
iu +
rf
ju
1, ... , v).
are imposed partially for convenience and
partia.lly to ensure determinacy in
certain systems of equations which
will be encountered.
Tt
These numbers
1u ' ... ,
1C
tu ' will be considered as
true ra.tings (or preferences) of the t objects a , ... , at corresponding
l
11
to the judge u.
Estimates of 1l'1u' ... , 1(tu will be denoted by Plu ' ... ,
Ptu ' respectively.
2.4.
The likelihood function.
From (2.).2), the probability of judge u assigning rank r iju to
object
8
i
and rank r
=
Jiu
3 -
r
iju
to object a
upon comparison of the
j
pair of objects is
I
1l'
2-r
\ 2-rijU
\
jiu
;
iu
\ 1l'.lU +1Cj U "I
\
(2.4.1)
I
Provided that the pair of objects is compared by Judge u, we observe that
\
ajl uj
,then r iju
n
and if
\ ai
~
= 1,
rr iu
iju
a J ' uJ
ll'iu +
,
1C ju
then r iju
r jiu
=2
and (2.4.1) becomes
=P
{a i --")
= 2,
r
Jiu
=1
aj
I u} ,
and (2.4.1) becomes
If we assume probability independence between pa.irs of objects when
compared by judge u, we obtain the likelihood function for the set of
comparisons made by judge u,
12
L
11
=TI
R
2-r iju 2-r jiu
1(
1(iu
ju
n
iju
1(iu + 1( ju
(2.4.2)
2a -
TI
:1
=
1(iu
TI
R
~
j
n
iju
r
iju
(u = 1, ... , v),
nijU(niU +
J(
j)
where R denotes the set of numbers R.: \ i, j: i <: j, i .: 1, ... , j-l;
j .: 2, ..• , tj
between
n j ju
ex:rer~"r3x_s
=1
J.
If we assUIIle probability independence
.ie:d'ormeJ. by different jUfg38, then from (2.4.2) we
obtain the likelihood function for the complete set of comparisons made
by the v judges,
(2.4.3 )
.:
")(
U
L ,
u
and hence if we set
we have
(2.4.4)
2.5.
Test I.
In this and succeeding sections, we will be examining hypotheses
concerning the true ratings of the t objects, a , ... , at' under various
1
assumptions. These assumptions wHl usually be depicted and taken care
of by the alternative hypothesis considered.
To start, we shall consider
the situation where each of the v judges compsres only one set of
the set being specified by the pa,ired comparison design used.
I'
pairs,
We desire
13
information about the differences, if any, among the true ratinrs of the
t objects.
In this section, we are willing to make the a,ssumption that
the v judges are consistent a,s a group;
that is, we desire to test the
null hypothesis,
HO: 1£1u = lIt,
against the alternative hypothesis,
(1:::: l, ... ,tj u = 1, ... ,v),
where the
1£1'6
are not all equal.
When the alternative hypothesis HI 1s true, (2.4.4) becomes
:::: ~ a* jnl£ - k ~
iii
1< j
Kn(l£i + l£j)'
where the simplification of the latter term is a.ided by the properties of
the incidence matrices listed in (2.2.5) and (2.2.6).
* defined in (2.4.4), can be rewritten, using (1.3.4),
We note that ai'
as
a*i : : 2k(t - 1) - ~ ~ n iju r 1ju '
When r
= t(t - 1)/2, then n iju = 1 for all i, j, u, (i I j), k : : v, and
a*
i
= 2v(t - 1) -
~' ~
j
u
r
iju'
and corresponds to a of 12_7 with n replaced by v.
1
Now maximizing (2.5.2), subject to the constraint E 1£1 :::: 1, yields
i
the set of equations
(1 :::: 1, ... , t),
14
·e
Summing (2.5.3) with respect to i yields
(2at - 4r) v + IJ
which implies IJ
= 0,
= 0,
since fram (1.3.1), r
= ta/2.
likelihood estimates P1' ... , Pt of 1£1' ... ,
1f
Hence the maximum-
t will be obtained from
system of equations
*
a1
Pi
-kE'(p +p
J
i
)-1
j
=0,
1, ... , t),
(i
(2.5.4)
When the null hypothesis H is true, (2.4.4) becomes
O
[ dnL
l" 1
IH0"i,.
- (2rvut
-
u,v
=-
vr
3
-
rv
"n t1
) (/
-
kt(t-l)
2
-
"n t2
d
In 2,
using (2.2.2), (2.2.5), and (2.2.6).
Thus the likelihood ratio test of the null hypothesis H against the
O
alternative hypothesis H , as specified in (2.5.1), will be in terms of
l
the likelihood ratio A , where, if we define
l
where Pl' •.. , Pt are solutions of (2.5.4), then
(n"-l
When r
=
t(t - 1)/2, then k
= - { vrfn2
=
1
- B( ) (n 10 }
v, and B(l) corresponds with B of ~2_7.
1
It will also be convenient at this stage to define the statistic
15
(2.5.8)
T(l)
= - 2 fn ~l = 2 vrfn 2 - 2 B(l) fn 10.
For the likelihood ra.tio in (2.5.7) to be useful to test the hypothesis stated in (2.5.1), it would be desirable to have some knowledge
of the distribution of B(l) -- at least when the null hypothesis H is
O
true.
It is possible to generate all combinations of the object sums of
ranks.
Then under the null hypothesis of equality of true object ratin@3
for each judge, the probability of each combination of rank sums is attainable.
This information would be sufficient to obtain the distribu-
tion of B(1) under H '
O
However, a direct computation of the probabilities of these various
combinations of rank sums would be extremely tedious, even for sma.n
values of the design parameters.
The fact that every permutation of the
rank sums corresponding to each judge is not possible, since each Judge
does not compare all possible t(t - 1)/2 pa.irs, is the cause of the
complication.
However, we can utHize the symmetry of the designs to
circumvent this particular difficulty.
Let Ai' (i
= 1, ... , t), denote the set of v (.:
= k(t - 1) elements
which are the k( t - 1) ranks a.ssigned by all the v judges to the object
a1 when compared with the other (t - 1) objects.
That there are
vex;:; k(t - 1) such ranks follows from the fact tha.t each of the v
judges compares a i with exa.ctlyex of the remaining (t - 1) objects.
Also,from the design properties depicted by (1.3.1) - (1.3.4), (2.2.5)
and (2.2.6), it follows that these k(t - 1) ranks are those for a i a.fter
comparison with each of the remaining (t - 1) objects exactly k times.
Hence, we can subdivide Ai into k disjoint subsets Aiu " with each Aiu '
16
containing (t - 1) elements which are the ranks for a i after comparison
with ea ch of the rema.ining (t - 1) ob jects exactly once.
denote the (t - 1) elements of Aiu' by r 1 ju " (i
I
jj
Thus if we
j = 1, ... , t),
then the sum of elements of Aiu ' is
and the sum of the elements of Ai is
k
E E' r iju '
u'=l j
= E E n iju
u j
r iju '
Hence the set of rank sums
(~ ~ nlJu r lju'
. .,
,
~ ~ ntJu r tju )
k
= ( E E' r lju ' ,
u '=1 j
(2.5.11)
:;::
k
E (Slu"
u'=l
... ,
... ,
k
E' r tju 1)
u'=l j
E
Stu') ,
could be considered as being the set of rank sums obtained from k complete repetitions of all t(t - 1)/2 pa.irs.
It is these rank sums which
determine B(l), and hence its distribution can be determined from the
distribution of the different combina.tions of the rank sums.
The
possibility of putting the rank sums in the form. of (2.5.11) enables
us to obtain the dlptrlbution of B(l), under the null hypothesis HO of
equality of true object ra.tings, in a manner described by Bra.dley and
Terry
Z-2_7.
This procedure will be described briefly below, but for
more deta.il the reference given should be consulted.
Consider the design (i) of (1.4.4) with parameters t :;:: 4, v :;:: 3,
b :;:: 6, r :;:: 4, k
sums, (Slu"
= 2,
A :;:: 2, a
= 2.
The possible sets of revised rank
... , Stu')' of (2.5.11) are
17
e
(i)
(3, 4, 5, 6),
( ii)
(:3, 5, 5, 5),
(11i)
(4, 4, 4, 6),
(iv)
( 4, 4, 5, 5) .
(2.5.12)
Each of the twenty-four permuta.tions of the elements of (i) has proba.bility 1/64, ea.ch of the four permutations of the elements of (11) and
the four permutations of the elements of (iii) ha.s probability 2/64,
while each of the six permutations of the elements of (iv) has proba.bility
4/64.
The possible rank sums in (2.5.1l) for k
=2
are obtained by add-
ing the sets of elements of (i), (ii), (iii), (iv), in turn,to the
corresponding elements of the sets of all possible permuta.tions of the
elements of these sets.
The probability of a given permutation is ob-
tained by multiplying the probability of the combination involved from
(2.5.12) and the permutation used to produce the given permutation.
To
......
obtain the probability of a given new combination of rank sums, the
probabil1ties for ea.ch permutation of the elements of the combination
are added.
By successive repetition of this process a desired number of
times, the combinations of rank sums (2.5.ll) and their corresponding
probabilities under HO can be determined for a.ny given k and t. Of
course, for large k and t the procedure would be extremely la.borious.
By this process, the combinations of rank sums, their corresponding
proba.bllities under H ' their corresponding va.lues of B(l) and its
O
probabilities under HO' for t = 4, k = 2, have been determined and are
tabulated in (2.5.13).
18
·e
(2.5.13 )
2
E (51 "" .,5 4 ,)
,u ' :::1 u
u
P{
r
1
e
i
u ' =l
(5 1ur '" .,5 4u ')
}!
IHO
T
B( l)~ P {B( )
l
.~
IHO}I
I
I
6 8, 10, 12
,
6, 8, 11, 11
24/4096
6, 9, 9, 12
24/4096
7, 7, 10, 12
24/4096
7, 7, 11, 11
24/4096
6, 9, 10, 11
144/4096
7, 8, 9, 12
144/4096
o
24/4096
}
24/4096'
!
0.602
72/4096
1.204
24/4096
j
1.498
288/4096
]
1.806
80/4096
)
6, 10, 10, 10
40/4096
8, 8, 8, 12
40/4096
7, 8, 10, 11
432/4096
2.359
43 2/4096
7, 9, 9, 11
336/4096
2.631
336/4096
7, 9, 10, 10
528 / 40 96
2.898
528 / 40 96
1056/ 40 96
8, 8, 9, 11
8, 8, 10, 10
408/4096
3.158
408/4096
8, 9, 9, 10
1224/4096
3.389
1224/4096
9, 9, 9, 9
152 /4096
3.612
152/4096
}
From the previous discussion, it then becomes clear tha.t the
tables of 12_7 and 13_7may be used to prOVide us with the distribution of B(l) under the null hypothesis HO '
The ta.bles are a.vailab1e
= 1,
for design parameters t and k in the following range:
t = 3, k
... ,10; t = 4, k
These ta.b1es
=
1, ... ,8;
t = 5, k = 1, ...,5.
also list the estima.tes, Pl' ..• , Pt' corresponding to the rank sum
combination (2.5.11).
19
·e
Thus a test procedure for (2.5.1), for t and k in the indicated
range, is as follows:
Determine the rank sums (2.5.11).
Then from
tables in ["2_7 and ["3_7 obtain the corresponding values for PI' ... ,
1
Pt ' B( ), and the probability P tha t B( 1 ) will not be exceeded i f the
null hypothesis HO is true.
If either t or k is outside the range for the tables, the above test
procedure cannot be used.
However, if only k is outside the indicated
range, it is possible to use the available tables to obtain the estimates
PI' ... , Pt' or at least a good first approximation to them, depending
on whether or not there exists an integer c which divides the rank sums
and k evenly and which is such that k/c is within the above indicated
This technique is discussed in detail in ~3_7 and ~t5_7,
range for k.
and the example given in Section 3.7, involVing its use, will describe
it further.
If t > 5, the available tables will not be of assistance even in
obtaining approximations to Pl' ... , Pt except in special ca.ses.
Hence,
to obtain estimates, Pl' ... , P , in this cBse, it will be necessary to
t
solve equations (2.5.4).
Some methods aiding in the solution of equa-
["7_7, and Bradley and
tions of this form are suggested by Dykstra
( 1)
Once Pl' ... , Pt have been obta.ined, B
can be eva.luated from
(2.5.6), but the significance level of B(l) can only be approximately
determined.
2.6.
A discussion of this will be given in Chapter III.
Test II.
For the situation and assumptions used in Section 2.5, we now de-
sire to test the null hypothesis H aga.inst the alternative that the
O
20
1{1'6
under H are split into two groups, the elements within each grcl.l"tl
l
being equal.
Tha.t is, we desire to test the null hypothesis,
1t'iu
HO:
= lIt,
(i = 1, ... ,t; u
= 1, ... ,v),
a.gainst the alternathe hypothesis,
(2.6.1)
{"
11:
iu
(1
= 1, .. _,sj
=
\
~=:1{, (1
If we let p denote an estimate of
1t,
= s+l, ... ,t),(u=l, ... ,v).
then the equa.tions (2.5.4)
become
*
ai
f s-l
(t_s)2}
p - k \ 2p·· + 1+p(t-2s)
(2.6.2a)
*
(2.6.2b)
(i = l, ... ,s),
= 0,
si(t-s)
k f
s(t-a~ + (t-apt-s-l)
l-sp
\ l+p{ t-2a
2 l-ap}
)
!
= 0, (i=s+l, ... ,t).
Summing the equations in (2.6.2a), with respect to i from 1 to s,yields
(2.6.3)
1 a
*
f a-l
(t_S)2}
P i~l a1 - ks 1.. 2p + 1+p(t-2s)
=
0,
from which we obtain
s
*
2 E si - ks(s-l)
p = - - - - - . . . ;i=l
;....;;;..------------
2ks(t-a)2 - (t-2s) {2
~ a~
- kS(S-l)}
i=l
Upon substituting for a * from (2.4.4), and using the parameter re1a.tioni
ahips of (1.3.1) - (1.3.4), we obtain
21
8
ks(4t - s - 3) - 2 E E E n ij r 1j
i=l J u
u
u
p =
s
2
ks(5st - 2t - 6s + 3t) - 2(2s - t) E E E n ij r ij
1=1 j u
u
u
(2.6.4)
Summing
e~uations
1n (2.6.2b),w1th respect to i from s
+ 1
to t,yie1ds
the same result, (2.6.4) -- as is expected.
Now, by proceeding in a manner ana.1agous to that given by Bradley 1n
.r4_7,
we will obtain a. test for (2.6.1).
Let X denote the number of
times an object of the first group of s objects ranks above an object of
the second group of (t - s) objects.
Then,
and upon SUbstitution in (2.6.4) we obtain
X
p=---.."....::.;:...---ks(t-s)2 + (2s-t)X
(2.6.6)
From the model of Section 2.3, we obtain the probabillty that any object
of the first group is ranked a.bove any object of the second group,
1f
---~l--a-1f1f + t="B"""
= 1f(t-s)
1 + (t-2s)1f '
Which under the null hypothesis H equals 1/2. Now,from this discussion
O
and from the observation that there are ks(t - s) comparisons of objects
of the first group with objects of the second group, it becomes clear
that the binomial or sign test will provide an appropria.te test procedure
for (2.6.1).
situation.
The example given in Section 3.7 will further clarify the
22
·e
2 .7.
Test III.
Suppose we have g groups with v judges in each group.
We are in-
terested in investigating the equality of the true treatment ratings of
the t objects under the assumption of within-group judge consistency,
but not necessarily assuming between-group judge consistency.
It should
be observed that these g groups could contain the same v judges, but due
to some additional feature, such as significant time lag between repetitions, or training gained from continued experimentation, we are unwi11ing to assume between-group judge consistency.
Let
m
~lu'
... ,
m
~tu
represent the true ratings of the t objects
corresponding to the u-th judge in the m-th group, (u = I, ... , Vj
m = 1, ... , g).
Then,we wish to test the null hypothesis,
m
J(iu
=
lit,
against the a1 ternative hypothesis,
(2.7.1)
(i = 1, ... ,t; u
I
1, ... ,Vj
m=l, ..• ,g).
m
Let n
iju
=
1 or 0 depending on whether or not the i-th and j-th
objects are compared by the u-th judge of the m-th group, (i
...
f
Jj
...
...
, Vj m = 1, , g) , and conventionally
, tj u = 1,
i, J = 1,
m
take n
= 0, (i = 1, ... , t; u = 1,
, Vj m = 1, ... , g) • It is
11u
clear that the corresponding incidence matrices
...
(m
= 1,
... , Sj
u = 1, ... , v),
will have anala.gous properties to those given in (2.2.5) and (2.2.6).
m
be the rank assigned to the i-th object if compared with
iJu
the j-th object by the u-th Judge of the m-th group. Then, corresponding
Let r
23
·e
hav~
to (2.2.2), we
(m
= 1,
... , gj
U
= 1,
... , v),
when thei-th and J-th objects are compared by the u-th judge of the
m-th group.
Now, under the assumption of probability independence between
pairs of objects, judges, and groups of judges, and by defining
we obtain, by procedures used in Sections 2.4 and 2.5,
{in L(3) I~l
(2.7.5)
(2.7.6)
LC~) IHo}
(rn
=; {~ .;m fn"~ - \ ~ P("~ "~)
+
= -v
r
g
}
,
fn 2,
where L(3) is the likelihood function, and {fn L(3)
IHi }
denotes the
na.tural logarithm of the likelihood function when the hypothesis Hi is
true, (i
= 0,
Then, if we denote the likelihood r~tio by ~3 and
3).
define our test statistic to be T(3) = -2 fn A , and define
3
m
m
E j log(P1 + Pj) - E
i
i <.
*
8 im
log Pi'
m
m
m
m
where Pl' •. "Pt' the maximum likelihood estimates of J"Cl , ... , ret' are
solutions of the system of equations
k E'
j
(p~ + p~)
-1
= 0,
(i = 1, ... , t),
(2.7.8)
m
E P
i
i
= 1,
=
(m=l, .•. ,g),
2 v r fn 2 - 2 B(l)
fn 10,
m
24
·e
we obtain,
~
the methods of Section 2.5,
Define
B(3)
(2.7.11)
=E
m
B(l)
m
For BO ) or T(3) to provide a. test for (2.7.1), some knowledge about
the distribution of B(3) under the null hypothesis H 1s needed. Clearly,
O
the distribution of B~l) under H is the same as the distribution of
O
B(l) under H ' Hence, the probability of a. specified value B of B(3)
O
can be obta.ined by determining the probabilities of the joint occurrences
of all combinations of Bl(l), ... , B(~) such that E B(l) = B, and then
g
m
summing these probabilities. Thus the distributi~n of B(3) under H
O
can be obtained from the distribution of B(l) under H '
O
For example, consider the case t = 4, k
tion B(3) = Bil) +
= 2,
g
= 2. In this situa-
B~l), and the distributions of Bi l ) and B~l) under
H are given in (2.5.13) and in £2_7. One possible value of B(3) is
O
1.204. This value can be obtained 1n the following ways:
(i)
B( 1)
1
= 0.602,
B(l) = 0.602;
2
( 11)
B(·l)
1
=0
B(l) = 1.204;
2
(iii) B(l) = 1.204,
1
B(l) - 0
2 -
In (1), three different sets of rank sums (6, 8, 11, 11), (6, 9, 9, 12),
(7, 7, 10, 12), each w1th probability of occurrence 24/4096 when HO is
true, can produce the value 0.602 for Bi 1 ) or B~l). Hence the pro2
babilityof occurrence of event (i) under H is 3 (24/4096)(24/4096)
O
= 5184/(4096)~. In (ii), Bi l ) is zero only when the set of rank sums
·e
(6, 8, 10, 12) occura, ~~l)
(7, 7, 11, 11) occurs.
= 1.204 only when the set of rank sums
Hence the probability of occurrence of event
( 11) under HO is 576/( 4096) 2 • 'Similarly, the probabUity of occurrence
of event (iii) under HO is 576/(4096)2. Thus}
P { B(3)
= 1.2041HO } = (5184
+ 576 + 576)
I( 4096) 2 = 0.033777.
By proceeding in this manner, the distribution of B(;) under H can be
O
obtained.
For reference, the distribution of B(3) under HO,for the above
example,has been tabulated in Appendix B.
For additional discussion
and tables for t = "), g,k = 2, ... , 5, consult ["2_7.
We are now able to introduce the following test procedure for
(2.7.1):
Compute the rank sums
m
m
~ ~ niju r iju '
(i = 1, ... , t),
corresponding to the m-th group of Judges, (m
design parameters are within the range t
k
=
B~l)
1, ... , 8;
t
=
k
= 1,
from the available tables in £2_7 and £,_7.
= 4,
g,k
= 2,
... , 10;
t;;; 4,
5, k ,.; 1, ... , 5, determine the corresponding value
with respect to m, thus obtaining B(3).
t
="
= 1, ... , g). Then, if the
Sum these
If t :: 3, g,k
B~l) 's
= 2, .•. , 5;
the exact significance level of B(3) is given in 12_7
or in Appendix B.
Otherwise, without further extension of these
tables, an exact test of' (2.7.1) using B(3) will not be available.
However, the significance level of B(3) can be approximately determined
and will be discussed in Chapter III.
For large t, g, k, it will be necessary to solve the equations in
m
( 2.7. 8) for Pl'
)
(3)
... , Pm
t , (m = 1, .. " g, and then to evaluate B
using (2.7.7) and (2.7.11).
26
It should 'be noted that estimates of the true object ratings bp.;,;;eo
on ell the gv judges will not be available.
It will only be possible to
obtain estimates of the true object ra.tings corresponding to a. specific
group of v judges.
This is entirely reasonable, though, since we were
not originally willing to make the aesum.pt.ion of equality of the true
ratings from group to group.
2.8. Test IV.
It is conceivable that the g groups of v judges considered in
Section 2.7 could have had the S8me judges in each group.
Then we
would have v judges each making g incomplete repetitions of r pairs,
the specific r pairs included in each repetition being specified by
(2.7.2) which depends on the paired comparison design used. We desire
to test for the equality of the true object ratings.
The situation where
we are unwilling to assume group consistency of jUdges from one repetition to the next is handled in Section 2.7, with the hypotheses being
stated in (2.7.1).
If we are willing to a.ssume consistency of judges
from one repetition to the next,we may wish to test the null hypothesis,
1f~.u
(2.8.1)
=
lit,
aga.inst the alternative hypothesis,
m
l!iu
= 1f i ,
(1
=
1, ..• ,t; u
= 1, ... ,v;
m = 1, ... , g).
If we define
(2.8.2)
then, under the assumption of probability independence between pa.irs of
objects, judges, and groups of judges, and by procedures used in
Sections 2.4 and 2.5, we obtain
{in
(2.8.3)
L(4)
IH 4}
=
{in L ( 4) IHo}
as in (2.7.6),
== -
ai/,n 1(i - gk
vr
th~ simp11fioationa
and (1.3.1) - (1.3.4).
B(4)
(2.8.4)
~
=
g
fn
i
~
j
"n(rri + 1(,J)
2,
b&ing enabled by (2.7.2), (2.2.5),
Define
g k
E
J
i <
log (Pi + Pj ) -
~ a 1 log Pi'
i
where a1 is defined in (2.8.2), and where Pl ' ... , Pt , the maximum-likelihood estima.tes of 1£1' ... , 1(t' are solutions of the sys'cem of equations
g k E' ( Pi + P )
J
j
-1
==
(1 = 1, ... , t),
0,
which has been obtained by maximizing (2.8.3) subject to the constraint E 1£i
=
1.
Then, i f we denote the likelihood ratio by A.4' and
i
if we define the test statistic to be T( 4)
=
-2 dn l>'4' we obtain from
(2.7.6), (2.8.3), and (2.8.4),
(2.8.6)
T(4)
=
2 v r g fn 2 - 2 B(4)
in
10.
Upon examination of (2.5.2) and (2.5.6), we note the similarity
between
8
1 and
a:, and between B(4) and B(l), with k repla.ced by gk.
Hence, for t and gk, within the range of the ta.bles in 12_7 and £3_7,
a test for (2.8.1) is prOVided by computing the rank sums
(i = 1, ... , t),
and, corresponding to this set of rank sums, determining :P l , ... , :Pt ,
28
(4)
B
,and its significance level
from these available tables.
If either t or gk is outside the range of the presently avajlable
tables, the above test procedure cannot be used.
For this situation,
we can determine Pl' ... , Pt by direct solution of equa,tiona (2.8.5),
or, if t
5
5, we can use the m&thod mentioned in Section 2.5.
We can
then evaluate B(4) end T(4) from (2.8.4) and (2.8.6), respectively, but
the significance level can only be approximately determined.
This will
be discussed in Chapter III.
It should be noted that there are many ways in which g sets of r
pairs could be assigned to each of the judges.
No matter how the sets
of pairs are assigned, the statistic B(4) will have the same distribution under the null hypothesis H of (2.8.1).
O
'e
This would suggest that
simplicity of assignment of the g sets to the v judges would be the
popular criterion.
However, it is felt that it would be better to ha.ve
each judge compare as many different sets of r pa,irs as possible, as
this may provide greater connectivity.
The investigation of an exact
criterion is desirable, but will not be conSidered at this time.
2.9.
Test V.
Consider the situation of Section 2.7 where we had g groups of v
judges, and where we assumed within-group judge consistency.
we wish to test for judge consistency between groups;
This time
that is, we wish
to test the null hypothesis,
against the alternative hypothesis,
H :
3
mu = 1£i'
m (i = 1, ... , t; u = 1, ... , v; m = 1, ... , g ) •
1£i
Under the assumptions of probability jndependsnce of
kave
{in
LC~) IH3 )
, given in (2.7.5), and
{in L(3)
;'.7, we
Sec~jcn
IH4 }
, given in
Hence, if we denote the likelihood ratio by A. , and if we de5
fine our test statistic to be T(5) ~ -2 fn ~5' we obtain
(2.8.3).
T(5) = 2(B(4) - B(3))fn 10,
(2.9.2)
where B(;)iS defined in (2.7.11), and B(4) in (2.8.4).
From the dis-
cussion of B(;) and B(4) in Sections 2.7 and 2.8, and hence from the
correspondence we can make of B(;) and B(4) with statistics which are
tabulated in ["2_7 and 13_7, we can evaluate T(5) by using these tables,
provided that our parameters 11e within the range of the tables.
When
the parameters a.re outSide this range, we must first solve equations
-e
(2.7.8) and (2.8.5), and then we must calculate B(;) and B(4) from
(2.7.11) and (2.8.4), respectively.
However, the distribution of T(4) under the null hypothesis H4 of
(2.9.1) will depend on nl , ... , nt' and, hence, will not give us an
exa.ct parameter-free test. An a.pproxima.te test will be discussed in
Chapter III.
2 .10 . Test VI.
Suppose we have v judges each comparing g sets of r pa.irs of objects,
a1' •.. , at' where the sets compared by ea.ch judge are specified by
(2.7.2).
Under the a.ssumption of judge consistency from one repetition
to the next, we wish to test for equality of the true object ra.tings;
that is, we desire to test the null hypothesis,
30
(2.10.1)
HO: 'lf iu = lit,
a.ga.inat the a.lterna.tive hypothesis,
R :
6
No
iu is a,seumed equal to any 'If jW'
(i I j, u Iw; i,j = l, ... ,t; u,w
'lf
=
l.- .•. ,v).
Then, under the assumptions of proba.bili ty independence of Section
2.8,if we define
(2.10.2)
we obta.in
(2.10.3)
and
Then, if we denote the likelihood raxl0 by
test statistic to be ±(6)= - 2 Vn
(2 .10 .4)
T( 6)
=~ {2
~6'
~6'
and if we define the
we obta.in
grin 2 - 2 Bu
in
10
J
where
(2.10.5)
and where P1u ' ... , Ptu ' the maximum-likelihood estimates of
'lf
tu ' are solutions of the equa.tions
'lf
lu ' ••• ,
(i = 1, ... , t),
(2.10.6)
1: Pi = 1,
i
u
which were obta.ined by maximizing (2.10.3) SUbject to the constraint
1:
i
1f
iu
= 1, (u = 1,
... , v).
31
·e
Obae~ve
that if g :: cv,
C ~
1, such that eeen judge Compares each
of all the POssible v different sets of
!'
pairs c times, the situation
reduces to the equivalent of each judge making the comparison of c k
sets of all PQBsib1e t(t - 1) /2 pairs.
~-2_7. Thus,
Th.is situation is dealt with in
if this method of repetition is employed, for g :: cv, T(6)
will provide an approximate test for (2.10.1) as described in Section 10
of
L2_7
and Section 7.1 of 14_7.
It would be possible to provide an exact teet for (2.10.1) by using
the statistic E B defined in (2.10.5).
u
u
u
under the null hypothesis
similar to that for B(l).
p~ired
The distribution of Band E B
u
u
He
of (2.10.1) could be obta.ined in a manner
However, the distribution would depend on the
comparison design being used, the number g of repetitions made by
ea.ch judge, and the criterion for a.ssigning the g sets of r pa.irs to each
judge.
The assignment of' these sets is, in itself, a design problem,
and it is conceivable that with the assignment of these sets to sa.tisfy
certain additional symmetry conditions, ta.bulation for small values of
t would be practicable.
2.11.
Test VII.
Consider the situa.tion of Section 2.10 where v judges compare g sets
of r pa.irs of objects 81' ... , at' and where we a.ssume judge consistency
from one repetition to the next.
among the judges;
HI:
(2.11.1)
This time we wish to test for agreement
that is, we desire to test the null hypothesis,
1'(iu = rr i ,
against the a.lternative hypothesis,
H6 :
No :J(iu is assumed equal to any rr jw '
(i
I
j, u
I
w; i,j:: l, ... ,t; u,w:: l, ... ,v).
32
·e
If ve denote the likelihood ratio by
statistic to be T(7)
1\7' and if we define the
(;sst
=-
2fn A , then, under the probability independence
7
assumpt.ions of Section 2.10, and from (2.10.3), (2.10.5), (2.8.3), and
(2.8.4), we obtain
(2.11.2)
T(7)
=
2(B( 4) - E :B )in 10.
u
u
As in Section 2.10, it 1s observed that if g
=
cv, c
~
1, such that
each judge compares each of all the possible v different sets of r pairs
c times, the situotion becomes equivalent to each Judge ma.king the
comparison of ck sets of all possible t(t-l)!2 pairs.
For this situa-
tion, which is dealt with in 12_7 and 14_7, T(7) will provide an ap:proxima.te teet for (2.11.1).
Any exact test for (2.11.1) provided by
T(7)Will not be a parameter-free test, since, under the null hypothesis
Hl , it will depend on T[l' ... ,
1'C
t .
For the situations encountered here and in Section 2.10, it would
be of coneiderable int....,rest to investigate how good the apprOXimate
tests prOVided by T(1} and T(6) actually are for small S, when g 1s
not some multiple of v but 1s comprised of as many different sets of r
pairs as possible.
2.12.
Extreme sets of rank sums.
For the situation of Section 2.5, the extreme values which the
sum of ranks corresponding to the i-th object may have are
(i
= 1,
... , t).
When either of these cases arise for any of the obJects, equa.t10ns in
(2.5.4), which provide our maximum-likelihood estimates for the parameters, become somewhat valueless.
In order for this rank sum to be
2 a: v (or a v), it is necessary that the i-th object be JudgeCl jnfedvr
(or superior) in 8.11 comparisons ma.de b.r all Judges.
Thus, when these
situa.tions arise, it would seem re<l,soJ)801e to estimate 1[i by 0 (or 1);
then, drop this object from the .anaJ..,vsis and consider only the rema.ining t - 1 obJects.
Let us consider the case where the t-th object has the extreme sum
of ranks
Clearly, there is no loss in generality in selecting the t-th object
for convenience in notation.
omitting this object from consideration,
we will have a reduced set of (t - 1) rank sums, namely,
(2.12.1)
Now, from the symmetry of the paired comparison designs as displa,yed
in (2.2.5) and (2.2.6), we obta.in the logarithm of the likelihood function L(l) under the alternative hypothesis Hl of (2.5.1) to be
{in
L(l)
IH1}
= tiE=-ll { 2(av-k) -
t~l
J=l
1:: n
u
:1,Ju
r
1Ju
}
in 1(1
(2.12.2)
t-l
k
fn( 1(i +
E
i
<
1(
J) •
j
It also follows easily that the system of equations, which prOVides
maximum-likelihood estimates of
:lt
l
, .•. ,
:lt
- , and the test statistic
t l
34
B(l) will be the same a.s in (2.5.4) and (2.5.6), respectively, with t
replaced by t - 1.
Hence, the test procedure for (2.5.1) will be the same as that outlined a.t the end of Section 2.5 with t replaced by t - 1, and with the
original set of rank sums replaced by the reduced set (2.12.1).
For the case where the t-th objeot ha.s the extreme rank sum
the omission of this object from considera.tion will result in the 1"educed set of (t -
1)
rank sums
(2.12.3)
Simila.rly, the test procedure for (2.5.1), in this case, is the same a.s
that outlined in Section 2.5, with t repla.ced by (t -
1),
and with the
origina.l set of rank sums repla,ced by the reduced set (2.12.3).
In Test IV of Section 3.8, for the case where the t-th object has
the extreme rank sum
(2.12.4)
m
m
E E E n
r
tju
m j u tju
=2 a
g
v or a g v,
the test procedure for (2.8.1) is the same as described in Section 2.8,
with t repla.ced by (t - 1), and with the origina.l set of rank sums repla.ced. by
EEE m
m
k
EEE m
m
k
m J u ~Ju r lju - g, ..• , m j u nt-1,ju rt-l,Ju - g ,
EEE
mJu
m
~J
m
u
r 1j
m
m
- 2gk, •.• , E E E nt _1 j rt-1,Ju - 2 gk,
u
mJu
,u
depending on whether or not the t-th object rank sum is 2
respectively.
Q;
g v or a g v,
35
·e
For Tests III and V, of Sections 2.7 and 2.9, respectively, if an
extreme rank sum occurs for the t-th object of the m-th group, namely,
E E
j u
m
m
n
r
tJu tju
== 2 a
v or
a
v,
the test procedures described in the respective sections can be employed
with t replaced by (t - 1), and with the original Bet of rank sums replaced by the reduced set of rank sums,
m
2: 2: ~j
j u
m
m
m
r lju - k, ... ,2: 2: nt-l,ju rt-l,ju - k,
u
j u
or
~j ~
~
rl!'~ju
m
m
r m - 2k
Iju
, ... , ~j ~" nt-l,ju r t-l,ju - 2k.
Such an a1tera.tion to the teet procedure can be extended when more than
one of the g groups has an extreme rank sum.
For Teet V, if the t-th
object in all g groupe hae an extreme rank Bum as in (2.12.4), in order
to evaluate B(4) we will need to follow the procedure of Section 2.8,
with t repla.ced by t - 1, and to employ the reduced eet of rank sums of
If, in Test VI of Section 2.10, the t-th object hae an extreme rank
eum corresponding to the v-th Judge, namely,
m
nt j
2: 2:
m j
,v
m
r tj
v
==
2 a g or a g,
then,
{ in L(6)
(2.12.6)
v-I
}
IH6 =
+
m
E 2: n
fn(rri +
{ 2:i b i u Inrc i u- m
i<J iJ u
u
u==l
2:
t-1
2:
1=1 m
t-l
{
E 2(a-
t-l
n~tv)- E n~jV r~jV
j=l
m
2:
2: nij fn(rci + rr j )
i< jm
v
v
v
}
lt
ju
fn T(iv
)}
36
·e
Hence, the test statistic T(6} of (2.10.4) will become
(6)
=2
T
v-l
gvr fn 2 - 2( E Bu + B'}fn
10,
v
u= 1
where
(2.12.8)
BI
v
=
t-l
E
E
f 2(a _ nm
i=l m \
itv
m
)- t-l
E n
j=l
ijv
r
m )
ijv
t-l
E nm
i E
< j m
ijv 1og ( Piv + Pjv ) '
where P1v ' ... , Pt - l ,v are solutions of
~
{ 2(" -
'e
= 0,
(i
= 1,
... , t - 1),
t-l
E
i=l
P
iv
= 1.
The manner of extension of this procedure to the situa.t.ion where other
objects ha.ve extreme rank sums corresponding to certain judges is clear.
For Test VII of Section 2.11, the above procedure can also be employed if the t-th object has an extreme rank sum corresponding to the
v-th judge.
8S
If, in addition, the t-th object ha.s an extreme rank sum,
in (2.12.4), corresponding to each judge, the test statistic T(7} of
(2.11.2) becomes
(2.12.10)
i'
'.
37
where B l is defined in (2.12.8), and where B(4)' is obtained from B(4)
u
of (2.8.4) by replacing t by (t - 1), and by using the reduced set of
rank sums of (2.12.5).
CHAPTER III
·e
SOME LARGE-SAMPLE PROPERTIES OF THE TESTS
AND SOME EXAMPLES TO CLARIFY THE TEST PROCEDURES
3.1.
Introduction.
The exact tests discussed in Chapter II are restricted by the
limited range of available tables.
In this chapter, approximate tests
for these situations will be described.
To obtain these approximate
tests, we will redefine the parameters in such a way that we may make
use of the results obtained by Bradley ~4_7.
Also, since discussion
with the use of incomplete repetitions may be somewhat obscure, some
examples will be included to help clarify the test procedures.
5.2.
Large-sample distribution of T(l) and T(4).
Let
(3.2.1)
Yi ;;; t(Pi -
1
i)'
(1;;; 1, ... , t),
where Pl , "', Pt are the maximum-likelihood estimates of
given 1n (2.5.1).
Now observe that
;;; 3kt(t-l)
L. E L.
1 j u
2
hence,
0.2.2)
'- a.* ;;;
i
1.
=
2avt _ )kt(t-l)
2
kt(t-l)
2
~l'
.•• , n t
where a * 1s defined in (~.4.4). Now, upon substitution in (2.5.6) and
i
(2.5.8), we obtain
and
respectively.
Substitution of (3.2.1) in (2.5.4) yields
E { 1 + ~(y i + YJ) }
-1
,
j
"-
and hence, upon substitution for a * in (3.2.4), we obtain
i
£n(l+Yi) E' { 1 +
J
+ 2k
E
£n { 1 +
i<J
~(Y1
~(y i
-1
+ Y
J
)}
-1
+ YJ) }
•
T(l), as expressed in (3.2.5), can be put in the form
where R(Y i ) depends on higher powers of Yi than the second power.
in the manner followed by Bradley
~4_7,
Then,
if we redefine
(i = 1, ... , t),
where 0ik 1s a sequence of constants converging to 01 as k
--->
00,
it
40
·e
can be shown that R(y.) converges to zero in probability as k
1
-->
00,
and
k
that T(1) has the same limiting distribution as
k t L y~. This 1imit2
i
ing distribution is, under H , a non-central X -distribution with (t - 1)
l
degrees of freedom and parameter of non-centrality
0. 2 .8)
which, for large k, can be approximated by
Under the null hyPOthesis HO' ~i
Therefore A(l)
=0
= lit,
and thus, under H
0'
0ik
= 0i = 0,
(i
= 1, .•• ,t).
T(l) has a limiting central
2
"-
X -distribution with (t - 1) degrees of freedom.
Similarly, T(4) has the same limiting distribution as gk
Thus, approximate tests for
-->
00 •
(2.5.1) and (2.8.1) are provided by
T(l) and T(4), respectively, with approximate significance levels being
2
obtained from tables for the central X -distribution with (t - 1) degrees
of freedom.
3.3. Large-sample distribution of T(3).
If, in (2.7.l),we redefine
(3.).1)
m
~i
1
=t
+
m
°ik
jk
,
m = 1, ... , t),
(i = 1, .•• , t;
where O~k is a sequence of constants converging to o~ as k
from section 3.2 and from
r +- 7,
-
1
-->
00,
then,
i t follows that T(l), which is defined
m
in (2.7.9), has a limiting distribution, under H , which is a non-central
3
2
X -distribution with (t - 1) degrees of freedom and parameter of
41
-e
non-centrality
which, for large k, can be approximated by
2
Since T(3) = L T(l)
it follows from the additivity property of x
T(3) has a limiting
d~stribution, under H , which is a non-central
m m '
that
3
X -distJ:"J.tution with g(t-l) degrees of freedom and parameter of non2
centrality
(3.3. 4)
which, for large k, can be approximated by
m I
m
Uneer the null hypothesis HO' ~i = t' 0ik
= 0im = 0,
(1
= l, ••• ,tj
m = l, ••• ,g), and A(3) = O. Hence, under H ' T(3) has a limiting central
O
2
X -distribution with g(t-l) aegrees of freedom and therefore provides an
approximate test for (2.7.1).
3.4. Larg8-s~ple distribution of T(5) •
From (2.9.2) we have
(3.4.1)
= 2vrgQn
= TO)
2 - 2B(3)2n 10 - ~2Vrgin 2 _ 2B(4)in 10}
_ T(4) •
4t:.,-.,
·e
Then, with the parameters of (2.9.1) redefirled as in (3.2.7) and (3.).1),
it follows, from sections 5.2 and 3.3 and from
~4_7,
that T(5) has a
."0
li~iting distribution, under H , which is a non-central XC-distribution
;.. i L;
3
(1',-1) (t-l) degrees of freedom and parameter of non-centrality
which, for large k, can be approximated by
Unaer the null hypothesis H ,
4
:rt
m
i
::
:rt
i
, (i :: 1, ... , tj m :: 1, ••• , g),
~(5) = 0, and hence T(5) has a limiting central X2-distribution with
(g-1)(t-l) degrees of freedom.
Thus an approximate test for (2.9.1) is
provided by T(5), the approximate significance level being obtainable from
2
the table for the X -distribution with (g-l)(t-l) degrees of freedom.
3.5. Large-sample distribution for T(6).
From (2.10.4) we have
T(6) :: L (2grin 2 - 2B
u
u
in 10)
:: L T
u
u
If g
= cV
J
(c
~
1), and the g incomplete repetitions are assigned to each
judge in a manner such that he compares all possible v different sets of
r pairs a total of c times, and if we redefine the parameters
43
(1
= 1,
... , tj
U
= 1,
••• , v),
where 0iuk is a sequence of constants converging to 0iu as kc
-->
00,
it follows, from Section 2.10, Section 3.2, and ~4_7, that Tu ' (u
= 1,
••• , v), has a limiting distribution, under H6 , which is a non-central
2
X -distr1bution with (t-l) degrees of freedom and paramter of non-central1ty
},
(6)
u
which, for large kc, can be approximated by
(3.5. 4)
~(6)
u
=
~ kct 3 E (~
~
i
Hence, from the additivity property of
iu
__tl )2.
x2 ,
it follows that T(6) has a
2
limiting distribution, under H6, which is a non-central X -distribution
with v(t-l) degrees of freedom and parameter of non-centrality
which, for large kc, can be approximated by
Under the null hypothesis H ' ~iu
O
= lit,
(i
= 1, ••• ,t;
u
= l, ••• ,v),
~(6) = 0, and hence T(6) has a limiting distribution, under H ' which is
O
2
a central X -distribution with v(t-l) degrees of freedom.
Hence an
approximate test for (2.10.1) is provided by T(6), the approximate
44
2
significance level being obtainable from a table for the X -distribution
with v(t-l) degrees of freedom.
3.6. Large-sample distribution for T(7) •
From (2.11.2) we have
= 2(B(4)
T(7)
- L B )£n 10
u
(3.6.1)
= L.
(2 grin 2
u
2B In 10) - (2grvfn 2 - 2B(4)2n 10)
u
u
Hence, if g
0_
= cv,
(c
~
1), and, if the repetitions are performed as
prescribed in section 3.5, then, from Sections 3.2 and 3.5, and from
~4_7, it follows that T(7) has a limiting distribution, under H6,
2
which is a non-central X -distrlbution with (v-l)(t-l) degrees of free-
dom and parameter of non-centrality
(3.6.2)
which, for large kc, can be approximated by
~(7) =
i kc
Under the null hypothesis H ,
l
~(7)
= 0,
t
3EE
i u
~iu
= ~i'
(i
= 1,
••• , tj
u
= 1,
••• , v),
and hence T(7) has a limiting distribution, under H , which is
l
2
a central X -distribution with (v-l)(t-l) degrees of freedom.
Thus an
approximate test for (2.11.1) is provided by T(7), the approximate signi2
ficance level being obtainable from a table for the X -distribution with
(v-l)(t-l) degrees of freedom.
45
'3.7.
Examples of test procedures.
l
Example 1.
For the situation of Section 2.5, consider t
= 5 specimens
of hand-
writing compared pairwiElf:3 by v '" 6 judges, with each JUdge recording a
preferonce for one object of each pair which he compares.
'vie will refer
to the factors which influence judgment as chara.cteristic x.
compared by the u··th Judge, (u
= 1,
The
I'
pairs
... ,6), will be dictated by the
field plan of design (ii) of (1.4.4) displayed in Table 1.4.1.
The in-
cidence matrices for this design, obtainable from Table 1.4.1, are:
1
1
o\
l
0
o
0
1
1
(:
\:
o
o
1
1
o
o
1
o
1
o
o
o
1
o
1
0
1
o
1
o
o
o
1
o
1
o
o
1
1
o
o
1
1
0
o
o /
0
1
0
1
0
/ 0
1
0
0
1
0
1
1
0
0
1
0
0
0
0
1
1
0
1
1
0
0
1
0
1
0
0
N '" I
1
i
I
N; =
o
1
1
1
0
0
\
!
0
1
0
0
1
1
0
0
0
1
0
0
1
1.
0
,
J
I
N
4
~
\
/
1. The data used in these examples is a small portion of that
collected for an experiment conducted a.t the Psychometric Labore,tory of
the University of North Caroline.
46
·e
!
i
I
I
1
0
1
0
, 0
0
/
=1
I
1
0
1
0
0
1
1
0
1
1
0
0
0
1
0
0
0
1
1
0
0
1
0
0
0
1
0
1
0
1
0
0
1
0
1
0
1
0
1
0
0
1
0
I
N
5
0
,
N6
=
FrOID (3.7.1) it .1a easily seen. that
(u= 1, ... ,6),
:; 1(5 x 5) + E Nu (5 x 5)
==
3 E(5 x 5).
u
Now, for Judge u, (u
matrix Ru
{ 8
=
==
1,
... ,
(r ij ) , where r
iju
y), we construct a (t x t) preference
==
i ~a J lu } , respectively.
1 or 2 if tai
If objects a
--4
i
8
j
lu} or
and a j are not com-
pared by judge u, we will conventionally take the elements r
r 111
. of R11 to be zero.
J
iju
and
Thus, the preference matrix R will have l's and
u
2's as elements corresponding to the unit elements of N , and zeros
u
corresponding to the zero elements of N .
u
From this definition of R , we
u
observe that the sums of the rows of R yield a (t x 1) vector, the
u
elements of which are the rank sums of the t objects corresponding to
judge u.
E (r
j
1j1
For exemple, the preference matrix R end the rank sums vector
1
, ... , r
tj1
) corresponding to judge J
1
for this experiment are
47
·e
I
i
0
0
2
1
0
0
0
0
1
1
1
0
0
0
,:)
?
2
0
0
0
0
2
1
0
0
I
I
f
I
R1
==
\
c.
J
~ n Sjl r 5j1
where 1 is a unit column vector of a.ppropriate dimension.
By this
procedure we obtain for each judge the following rank sums vectors:
J1:
(3, 2, 3, 4, 3)
Jr"\:
c.
(2, 2,
4, 3, 4)
J :
(2, 4,
3, 4, 2)
J 4:
(2, 3, 3, 3, 4)
JO):
( 4,
"-,
6:
( 2,
2, 4, 4,
3
(3.7.2)
J
r)
2,
4, 3)
3) .
Summing these with respect to judges yields
(3.7.3)
e
Hence,
(~ : n1ju r 1ju' ... , ~ ~ n Sju r 5Ju )=(15,
15, 19, 22, 19) .
48
,e
where a*1 is defined in (2.4.4).
The rank sums vector in (3.7.3) is
what we require in order to use the tables of 13_7.
tables for t = 5 and for k
(p l '
... , pS)
= (.38,
= ;
complete repetitions, we obtain
.38, .10, .03, .10), B(l)
P \ B(l)
~
Entering these
= 6.686,
6.686/H } :: 0.0404.
O
Hence,we would conclude, at the 0.0404 level of significance, that the
five handwriting specimens are different with respect to the characteristic x undor the assumption that the judges are consistent as a group.
It is of interest to note, from (2.5.8), that
T(l)
=2
v Kn
I'
2
- 2 B(l)
Kn
10
10 .80
x (4)
from Section 3.2.
From the large-sample properties of T(l), we obtain the a.pproximate
significance level to be
p{
X(4)~10.80IHO}
=0.028.
Thus, although the approximate test will give too many significant
results, the approximate significance level obtained is reasonably
close to the exact significance level, even for k as small
BS
3.
,1j;xample 2.
We will now consider an example for the situation portrayed in
Section 2.8.
Suppose that the six judges of Example 1 make two in-
complete repetitions with
I'
pairs in each repetition.
The
I'
pairs
.e
compared by the u-th Judge in the first repetition are those indicated
in
<:~. 7.1),
and, hence, the incidence matrices for the first repatiticn
are
(u
= 1,
... , 6).
The r pairs compared by the u-th judge in the second repetition are indicated by the inoidence matrices
The rank sums vectors corresponding to each Judge for the first repetition are given in (3.7.2).
By an ana1agou8 prooedure to that
used in Example 1, we compute the rank sums vectors corresponding to
0_
each Judge on the second repetition:
J :
l
(2, 3, 2 , 4, 4)
J :
(3, 2, 3, 3, 4)
2
J
(3.7.5)
3
:
(2,
3, 3, 4, 3)
4: (2, 3, 3, 3, 4)
J : (3 , 2, 3, 4, 3)
J
S
J :
6
(2, 2, 4, 4, 3) .
Summing these with respect to Judges, we obtain
CL7.6)
2
2
2
2
(~~ ~ju r 1ju " "'~ ~ nSju r SJu )
= (14,
15, 18, 22, 21).
Hence,
(8
*12 ,
... , a *
52) =- (10, 9, 6, 2, 3),
where a*i2 is defined in (2.7.7).
(3 .7 .3), we obtain
By summing the vectors in (3.7.6) 8.nd
.e
(29, iO, 37, 44, 40).
Hence, we obta,in
(ai, ... ,
("3.7.9)
8
5) = (19,
where ai- is defined :1.n (2.8.3).
18, 11, 4, 8),
Tho result in (3.7.9) can pe checked
by adding vectors in (3.7.7) and (3.7.4).
The elements of the vector
(3.7.8) are the rank sums which enter into the statistic B(4) of (2.8.4).
Here t = 5 and gk
=
6 -- the corresponding number of complete repeti-
tiona -- are outside the range of available tables.
Hence, we are
unable to obtaln either the value of B(4) or its significance level
directly from the tables.
outlined in
'e
1.-3_7
However, since t
£15_7
and
=
5, we can use a procedure
as a means to assist us to eva.luate B (4) .
DiViding gk and the elements of (3.7.8) by two, we obtain
(3.7.10)
gk/2
= 3,
(3.7.8)/2
=
(14.5, 15, 18.5,22,20).
Then, for rank sums vectors:
(14, 15, 19,
we enter the tables of
~)2,
20) and (15, 15, 18, 22, 20),
£3_7
for t = 5 and for 3 complete repetitions
and obtain corresponding estimates
(.51, .33, .08, .02, .05) and (.38, .38, .14, .03, .07),
respectively.
Then, by linear interpolation, we obtain an a.pproxima.-
tion to the estimates, Pl' ... , P ' corresponding to the elements of
S
(3 .7.10) to be
(.445, .355, .110, .025, .060),
which, a.fter adjustment to add to 1, is
(.447, .357, .111, .025, .060).
Starting with these approximations and using the iterative formula
given in
.r 2..] , we obta.in
.e
51
(:7).7.n)
(PI'
"0' P5) ;;: (.441, .361, .106, .029, .063).
SUbstituting these values in (2.8.4), we obtain
B( 4) = 6 {log .802 + log .547 + log .470 + log .504
+
log .467 + log .390 + log .424
+ log .135 + log .169
+
log .092
J
- 19 log .441 - 18 log .361 - 11 log .106 - 4 log .029
- 8 log .063
This value sUDetituted in (2.8.6) yields T(4) ;;: 25.353.
From Section
3.2,it follows that
a.pproxima.te1y.
Hence, under the assumption tha.t the judges are con-
sistent as a group and consistent from one repetition to the next, we
would conclude, at less than the 0.0005 level of significance (a.pproximately), that the five handwriting specimens are different with
respect to the characteristic x.
Example 3.
Suppose that in the previous example we were unwilling to a.ssume
Judge consistency from one repetition to the next.
becomes that of Section 2.7.
The situation then
Here we have g = 2 groups of v = 6 Judges
compa.ring t objects according to a chara.cteristic x, and a.ccording to
this characteristic the judges in each group would not be considered
the same.
~
')2
..
Using the data of Examples 1 and 2 for each repetition separate1,y,
we obtain
(Pl.'
_'
(1) _
6
(1)_
•.. , P ) - (.38, .38, .10, .03, .10), B
- 6.68 , Tl
- 10.800,
l
5
(Pl'
... , Pc)'" (.51,
.33, .10, .02, .03),
B~l)=
5.598,
T~l)
= 15.809.
Hence, from (2.7.10),
T(3)
=
10.800 + 15.809
=
26.609.
From Section 3.3, TO) has, under H ' approximately a
O
X 2-distr1but1on
with g(t - 1) "" 8 degrees of freedom, and therefore an approximate
significance level for Test (2.7.1) is
'e
Hence, under the assumption of within-group Judge consistency only, we
conclude, at approximately the 0.001 level of significance, that the
fi ve handwriting specimens are different with respect to the characteristic x.
]xample 4.
Suppose for the experiment used in Examples 2 and ? we now desire
to test for the consistency from one group of repetitions to the next.
Then, the test given in Section 2.9 is the one we need.
From (2.9.2) we have the test statistic
T(5) '" 2(B(4) - B(3))fn 10
::. T(3) _ T(4)
= 26.609 - 25.353
= 1.2'56.
, ~-... vm
(~.J. 4 • 1) ,
from Examples 3 and 2,
.e
53
From Section 3.4, und6r H of (2.9.1), T(5} has approx.:fmately a
4
;2
X -distribution with (g - l}(t - 1)
=4
degrees of freedom.
Therefore,
the approximate significance level for Test (2. 0 .1) in this case is
Hence, under the assumption of within-group judge consistency, we have
no evidence to doubt the hypothesis that the group judging criterion is
the same from one repetition to the next.
Example 5 ~
Suppose that in Exemple 1 we were only j.nterested in testing the
null hypothesis -- tha.t the true object ratings are equa.l -- against
the alternative that thte true ra.tings are in two groups, within which
they are equal, but between which they are not necessarily equal.
test for this situation is given in Section 2.6.
A
For the data. given in
Example 1, we wish to test the null hypothesis,
a.gainst the alterna.tive hypothesis,
TC
iu
In our example s :: 2, t
=
{
= 3.
(i :: 1, ... , s),
TC,
1 - s]"(
t - s
, (i
= 6+ 1, . . ., t j u = 1, ... ,v).
To determine X, the number of times an
object of the first group of s = 2 objects is ranked above an object of
the second group of t - s:::5
objects, recode the objects in such a
way that a1' ... , as are in the first group and a s + l ' ... , at are in
the second group.
Then for judge u, (u
= 1,
... ,
v), construct a new
preference matrix corresponding to R of Example 1, but this time use
u
54
the notetion
J
if objects a 1 and a j are compared by judge u.
of the elements of the v, (s x
Then X will be the sum
t::li) sub-matrices conta.ined in the
upper right-hand corner of the new t x t preference matrices correFor the da ta of our example, the 6, (2 x 3)
sponding to each judge.
Bub-ma.trices are:
(:
:)
1
1
o
1
,
C ~ ) (:
1
,
0
o
~)
o
1(
0
0
0
)
o
,
1
Summing the elements of these, we obtain X '" 15.
an estimate of
1
Hence, from (2.6.6),
is
P
=-
X
ks(t - 8)2+ (2s - t)X
= 0.385.
If' we consider the preference of an object in the first group over
an object in the second group as a success, we have X = 15 successes 1n
ks(t - s)
= 18
trials.
The probability of such an occurrence, under
the hypothesis that the probability 1s one-half that an object of the
f1rst group is ranked over an object of the sscond group, is
55
In addition,
(These probabilities have been obtained from
114-7).
Hence, under the
aSBumption that the judges are consistent as a group, we would reject
the null hypothesis of equality of true object ratings in favor of the
alternative hypothesis H2 , at approximately the 0.01 level of significance.
CHAPTER IV
AN ANALOGUE TO KENDALL'S COEFFICIENT OF
AGREEMENT FOR THE CASE OF INCOMPLErE REPETITIONS
4.1.
Introduction.
Ivl.
G.
Kendall and Babington Smith Ll:?7,snd Kendall L10,
11_.7,
introduced a coefficient of agreement for the case where v observers
each compare all possible t(t - 1)/2 pairs.
In this chapter, an ana-
la.gous statistic for the case where each judge compares only r pairs is
constructed, and some of its implications are discussed.
Under certain
assumptions, it is shown to provide an alternative test for the situation described in (2.5.1) .
0_
4.2.
Some definitions and notation.
Corresponding to judge u, (u
= 1,
... , v), we define a (t x t)
matrix
(4.2.1)
where
,if
,if
if objects
8.
i
and
8
j are compared by Judge u, and where Piju 1s always
o if a i and Sj are not compared by judge u. From the design properties
exemplified by the incidence matrices (2.2.3) corresponding to each
Judge, we observe that
={
l,or
if n
0,
iju = 1,
PiJu
o
,
if n iju
= 0,.
57
and, hence, r elements of P will be unity and the remaining t
2
u
The
elements will be zero.
~~x
- r
Pu will, be qefined to ba the pre-
ference matrix corresponding to judge u.
Now we shell define a combined preference matrix P as
(4.2.2)
= E Pu (t
p{t x t)
x t) .
u
If we let I'ij' (i,j ::; 1, ... , t), denote the element in cell (i,j) of
P, then
:::: 0
(4.2 .3)
(4.2.4)
--
l
ji
= k - f'ij ,
(1 :::: 1, ... , t),
(1
tj,
i,j = 1, ... , t).
These follow from the properties of the incidence matr1ces, with (4.2.4)
specifically following from the fact that each pair of objects is
compared by exactly k judges.
y
i,J
ij
=v
Also,
r :::: kt{t - 1)
2
We also observe that i f all the judges agree perfectly on their a11ocation of preferences, P will have t{t - 1)/2 elements k and t{t+l)2
elements 0.· Poorest agreement among the Judges will be reflected in P
when
if k 1s even,
k + 1
= --2--(i
I
4.3.
J
j
i,J = 1, ... , t).
A coefflcient of agreement.
Define
, if k is odd,
.e
58
(4.3.1)
Physically,
~
can be interpreted as the sum of the number Qf agreements
between pa irs of judges.
The maximum value of E is
(c..~t) (k2)
'.'
and is attained when the judges are in perfect agreement in the a11ocation of their preferences.
The minimum value for L is
, if k is even,
, if k is odd,
and is a.ttained when the judges are in poorest agreement.
Now define
(4.3.2)
Then, from the above discussion, u has a maximum value of 1 and a
minimum value of
1
-k=l
, if k is even,
, i f k is odd.
The magnitude of u gives a. mea.sure of the agreement among the judges.
When each Judge compares all t(t - 1)/2 pairs of objects, the u of
(4.3.2) is equivalent to Kendall's coefficient of agreement in~lO_7.
.e
59
4.4. Tests based on E.
A calculated value of
~,
without any knowledge of the 1mplic.a.tions
of this value, would be somewhat meaningless.
question:
We are interested in the
If we are Willing to use the Bradley-Terry model, would E
provide a useful test for the situation outlined in (2.l1.1)?
In
pursuing an answer to this question, we insert the following discussion:
From (4.2.4) we obsorve- that the contribution to E from the pair of
cells (i,j), (j,i) of P is
The probability that the contribution from this pair of cells to E 1s
is equal to the probablli ty that 11j = 'I, Which, under the null
...
(1 ::: 1,
hypothesis H : r'f
, t; u ::: 1, ... , v) , and under the
iu = :lt i ,
l
assumption of probability independence between judges, is
Hence, the probability generating function for the contribution to E from
the above pair of cells is
(4.4.2)
/
60
Therefore, under the assumption of probability independence between
pairs of objects, the probahility generating function for E becomes
(4.4.3)
f(t) =
i
n
<
fiJ(t) .
j
From (4.4.3) we can obta.in
P \E
= x}
= the coefficient of t
X
,
and
=
i
(the coefficients of t ) .
E
i ~x
Thus, the distribution of E under the null hypothesis H depends on
l
'11:
1'
..• ,
1f
t
"'lence, E will not prOVide an exa.ct parameter-free test far
;
(2.11.1) .
However, as will be indicated, E is useful to test the null
hypothesis H that the preferences are alloca.ted a,t random.
O
Under H '
O
the probability generating function for the contribution to E from the
pair of cells (i,j), (J,i), which is given in (4.4.2), becomes
(4.4.4)
g(t)
The proba.bility generating function for E, under HO' becomes
h(t) = [g(t)
Jk (k-l)/2
The probability generating function given in (4.4.5) is identica.1 with
tha.t for the situation where thE'! combined preference matrix is constructed by summing the k preference ma.trices corresponding to k
judges who compare all possible t(t - 1)/2 pairs.
considered by Kendall.
This is the case
Thus we may a.vail ourselves of his ta,bles of the
61
pX'C)hal:>ill ty thut·) certain value of E will bo attained or exceeded.
These tables are available for parameters in the following range:
k ~ 3, t - 2,
k
= 6, t
... , 8;
k
= 4, t = 2,
..• , 6;
= 5,
k
t
=
2,
..
0' 5;
4. Further tables, useful for some of the presently
2, ... ,
available pa.ired comparison designs, are included in Appendix C.
An exact test ¢ of H for parameters in the above range can, thereO
fore, be defined as
,
(4.4.6)
,
/
ifE <: c
a:
where a and Co: are such that
¢0:) IHQ }
E {
where 0 <
Q
= 0:,
< 1.
For larger values of k and t, Kendall has indicated that
{
~: _ ~ (
c.
t )
(
k) 7k=2'
(k-3»)
")
~
r:1
c.
\K.-c:.j
4
k-2
hae approximately a X2-distribution with
C)
v
k(k-D
:.=
(k_2)2
degrees of freedom, this approximation being reasonably good for parameters outside the above indicated range.
If we assume the v JUdges to be consistent as a group, (4.4.6)
could be considered as a test for the equality of' true object ratings
8S
outlined in (2.5.1).
It is conjectured that ¢(E) is less powerful
than the test based on B(l), say \U(B(l», given in Section 2.5.
A proof
62
of this for the general case is somewhat intractable, but is supported
= 3,
by ca.ses for v
Case (i)
v
==
t
3, t
= 2,and
==
v
= 3,
t
=3
as indicated below:
2.
Consider
1
9'(1:)
==
\
1
1: ~ 3
y.'(B(l»
0
t.:
< 3
==
{
, B (1) < .602
B(l)
1/6
0
,
==
.602
,
B(l) > .602
,
since 1\.2
~
O.
Similarly we can show that, for these values of v and t,
the above property obtains for tests ¢(1:) and y..t (B(l»
Case (ii)
Consider
of any equal size.
63
¢U.:)
\
=
where E { ¢(E)
A
3
then,
"IT:
=
1
E
> 9
3/4
E
=9
E
< 9
a
IHO }
J
= E { y;(B(l))
IRa}
l
= 0.0117.
1)3332333233
4rT (1f
1 2 - J'f 3 ) + J'f2 (J'f1 - J'f 3 ) + J'f3 (1f l
~ ~(B(l)'Hl:
rfl'1l2,1t3}
= \
Let
-
f 1t~(1t~ + lr~) + 1l~()[~ + 1l~)
Ai \
J
since
~
:::
o.
APPENDIX A
FURTHER PAIRED COMPARISON DESIGNS FOR 4 ~ t ~ 25
The following tables list the pairs of objects to be assigned to
each judge for twenty-seven new paired comparison designs.
For Series
2 and Series 3 designs, the pairs assigned to each judge are assigned
in sets.
For Series 2, t = 2z, and we divide the z(2z-1) pairs into
2z-1 sets of z pairs each, such that each object occurs exactly once
among the pairs of a set.
The 2z-1 sets are obtained by developing,
mod (2z-1), the initial set
(1, 2z-2), (2, 2z-3), ... , (z-l, z), (0, 00),
where 0, 1, 2, ••• , 2z-1, co denote the 2z objects, and the object 00
remains unchanged in the development.
The initial sets for even values
of t from 4 through 24 are tabulated in Table Al.
For Series 3 designs, t
= 2z+1,
and we divide the z(2z+1) pairs
into z sets of (2z+l) pairs each, such that each object occurs exactly
twice among the pairs of a set.
The z sets are obtained by developing,
mod (2z+l), the second elements of each pair of the initial set
(0, 1), (1, 2), (2, 3), ... , (2z-l, 2z), (2z, 0),
leaVing the first elements of each pair unchanged.
The initial sets
for odd values of t from 5 through 19 are tabulated in Table A2.
For certain Series 2 and
eries 3 designs, the sets of pairs to be
assigned to each judge can be obtained by developing an initial set with
respect to a certain modulus.
These designs are listed in Table A3, in
which Roman numerals are used to denote the sets of pairs obtained from
the initial set I.
Series 2 and Series 3 designs which cannot be listed
65
in this way are presented in Table A4, the sets of pairs for each judge
being completely indicated.
In Table A5, two Series 1 designs, which
are complements of designs given in Table 1 of
~1_7,
are listed.
In
this case,the actual pairs of objects to be compared by each judge are
presented.
TABLE A1
t
Initial Sets of Pairs
-
4
(1,2),(0,00)
6
(1,4),(2,3),(0,00 )
mod 5
8
(1,6),(2,5),(3,4),(0,co )
mod 7
10
(1,8),(2,7),(3,6),(4,5),(0,00)
mod 9
12
(1,10),(2,9),(3,8),(4,7),(5,6),(0,00)
mod 11
14
(1,12),(2,11),(3,10),(4,9),(5,8),(6,7),(0,00)
mod 13
16
(1,14),(2,13),(3,12),(4,11),(5,10),(6,9),(7,8),(0,00)
mod 15
18
(1,16),(2,15),(3,14),(4,13),(5,12),(6,11),(7,10),(8,9),
mod 3
mod 17
(0,00 )
20
(1,18),(2,17),(3,16),(4,15),(5,14),(6,13),(7,12),(8,11),
mod 19
(9,10),(0,00)
22
(1,20),(2,19),(3,18),(4,17),(5,16),(6,15),(7,14),(8,13),
mod 21
(9,12),(10,11),(0,00)
24
(1,22),(2,21),(3,20),(4,19),(5,18),(6,17),(7,16),(8,15),
(9,14),(10,13),(11,12),(0,00)
mod 23
66
TABLE A2
t
Initial Sets of Pairs
5
(0,1),(1,2),(2,3),(3,4),(4,0)
mod 5
7
(0,1),(1,2),(2,3),(3,4),(4,5),(5,6),(6,0)
mod 7
9
(0,1),(1,2),(2,3),(3,4),(4,5),(5,6),(6,7),(7,8),(8,0)
mod 9
11
(0,1),(1,2),(2,3),(3,4),(4,5),(5,6),(6,7),(7,8),(8,9),
mod 11
(9,10),(10,0)
13
15
(0,1),(1,2),(2)3),(3,4),(4,5),(5,6),(6,7),(7,8),(8,9),
(9,10),(10,11),(11,12),(12,0)
(0,1),(1,2),(2,3),(3,4),(4,5),(5,6),(6,7),(7,8),(8,9),
(9,10),(10,11),(11,12),(12,13),(13,14),(14,0)
mod 13
mod 15
(0,1),(1,2),(2,3),(3,4),(4,5),(5,6),(6,7),(7,8),(8,9),
17
(9,10),(10,11),(11,12),(12,13),(13,14),(14,15),(15,16),
mod 17
(16,0)
(0,1),(1,2),(2,3),(3,4),(4,5),(5,6),(6,7),(7,8),(8,9),
19
(9,10),(10,11),(11,12),(12,13),(13,14),(14,15),(15,16),
(16,17),(17,18),(18,0)
mod 19
67
.e
TABLE A3
Design
number
Paramoters
= 8,
= 28,
v
r
k = 6,
A,
t
b
1-
=7
= 24
= 20
Sets of
pairs
Sets of pairs assigned to initial Judge
t(8)
I,II,III,IV,V,VI
mod 7
t(9)
l,II,III
mod 4
t (10)
I,II,IV,V,VII,VIII
mod 9
t(lO)
I,II,III,IV,V,VI,VII,VIIl
mod 9
= 44
= 33
tell)
I,II,III,IV
mod 5
12, v = 11
:= 66, r = 30
k = 5, >.. = 12
t(12)
I,IV,VI,VII,VIII
mod 11
t(12)
II,III,V,lX,X,Xl
(Complement of number 6)
mod 11
=6
t = 9,
b = 36,
0:
2.
:=
Q
b
k
5.
= le,
= 45,
v
r
= 2,
>..
Q
!.~
b
•
:=
= 30
:=
15
8,
k
:=
Ct
=8
=9
= 40
A,.. = 35
v
r
t = 11, v
I b = 55, r
I
k := 4, >..
Q := 8
!
5·
=3
6
= 10,
= 45,
t
r
=4
= 27
3, >.. = 18
=6
k
t
v
=5
,
t
:b
,
6.
ex
7.
:=
=5
. t = 12, v
,b := 66, r
k = 6, A.
a := 6
i
= 111
= 36
= 18
68
.e
TABLE A3 (continued)
Design
number
.. Parameters
Sets of
pairs
= 13, v = 3
= 78, r = 52
t
b
Sets of pairs assigned to initial Judge
t(13}
I,II,IV,V
mod 6
t(14)
I, V,XI ,XIII
mod 13
t(14}
II,III,IV,VI,VII,VIII,IX,X,XII mod 13
(Complement of number 9)
t(15)
I, V, VII
mod 7
12.
= 15, v = 7
b = 105, r = 60
k = 4,
30
'" =
ex = 8
t(15}
II,III,IV,VI
(Complement of number 11)
mod 7
13.
t = 16, v
b = 120, r
k = 7, '"
a = 7
= 15
= 56
= 2).j.
t(16}
I,VI,VIII,XI,XII,XIV,XV
mod 15
8.
= 2, '" = 26
Ct = 8
t = 14, v = 13
k
9.
b
= 91, r = 28
k = 4; '" = 7
a = 4
= 14,
= 13
10.
v
b = 91, r
k = 9, '"
a =9
=7
1l.
t = 15, v
b,= 105, r
k = 3, '"
a = 6
t
= 63
= 42
= 45
I
I
I
= 15
t
69
.e
TABLE A; (continued)
Design
number
t
14.
b
t
15·
b
Parameters
Sets of
pairs
= 16, v = 15
= 120, r = 64
~ := 8, ~ = 32
0: = 8
t(16)
II,III,IV,V,VII,IX,X,XIII
(Complement of number 13)
mod 15
= 20, v = 19
= 190, r = 90
It = 9, >.. = 40
0; = 9
t(20)
I,VI,VII,X,XIII,XIV,XV,XVI,
XVIII
mod 19
t(20)
II,III,IV,V,VIII,IX,XI,XII,
XVII,XIX
(Complement of number 15)
mod 19
= 20, v = 19
b = 190, r = 100
k = 10, A = 50
Sets of pairs assigned to initial judge
t
16.
Ct =
10
TABLE A4
Design
number
Parameters
Sets of
pairs
t=13,v,.,4
Judge
Sets of pairs assigned to each
judge
J1
I,III,V
J?
I,IV,VI
k=?,>..=l?
J
II,III,VI
0:",6
J4
b:78,r:39
t(13)
1.
3
II,IV,V
70
.e
TABLE A4 (Continued)
Design
numbet'
Parameters
sets of
pairs
Sets of pairs assigned to each
Judge
J1
I,IV,VII,X,XIII
J2
I, V, VIII,XI,XIV
k=2,;\=8
J)
II,IV,IX,XII,XIV
a=5
J4
II,VI,VII,XI,XV
J
III,V,IX,X,XV
t=16,v=6
b=120,r=40
2.
Judge
t(16)
5
J6
III,VI,VIII,XII,XIII
J1
II,III,V,V~,VIII,IX,XI,XII,XIV,XV
J2
II,IIIjIV,VI;VII,IX,X,XII,XIII,XV
k=4,;\=48
J
I,III,V,yI,VfI,VIII,X,XI,XIII,XV.
a=lO
J4
IjIII,IV;V,VIII,IX,X,XII,XIII,XIV
J
I,II,IV,VljVII,VIII,XI,XII,XIII,XIv
I
\
3.
t=16,v=6
I b=120,r=aO
t
I
t(16)
\
I
5
J6
1
I
I,II,IV,V,VII,I~,X;XI,XIV,XV
J1
(Complement of number 2)
I,II,III,IV,V,VI
J2
I,II,VII,VIII,IX,X
k=4,A=16
J
I,III,VII,XI,XII,XIII
0'=6
J4
I,IV,VIII,XIII,XIV,XV
J
II,V,IX,XI,XIII,XIV
I
t=16,v=10
4.
3
b=120,r=48
t(16)
3
5
J6
II,VI,VII,XII,XIV,XV
J
III,V,VIII,X,XII,XIV
7
Ja
III,VI,IX,X,XIII,XV
J
IV,V,VII,X,XI,XV
9
J 10
IV,VI,VIII,IX,XI,XII
71
.e
TABLE A4 (Continued)
Design
number
Parameters
Sets of pairs assigned to each
judge
I,IV,VII,VIII,IX,XI,XII,XIV,XV
J2
I,III,IV,VI,VII,IX,X,XII,XIII
k=6,~=40
J~
V,VI,VIII,IX,X,XII,XIII,XIV,XV
0:=9
J4
I,III,IV,V,VI,X,XI,XIV,XV
J
I,II,III,VIII,X,XI,XII,XIII,XIV
b=120,r=72
.,
t(16)
(
\
5
I
J6
11,111, 1V,V1,V11,VIII,XIII,XIV,XV
J
II,IV,V,VII,X,XI,XII,XIII,XV
I
I
.-
Judge
Jl
t=16,v=lO
5.
Sets of
pairs
I'
t
7
Ja
1,I1,IV,V,VI,V1II,IX,XI,XIII
J
1,II,III,V,VI1,VI1I,IX,X,XV
9
J lO
II,II1,V,VI,VII,1X,X1,X11,X1V
(Complement of number 4)
J1
1,1I,III,IV,V,V1
J2
I,VII,VIII,IX,X,XI
k=2,~=11
J
II,VI1,X11,X1II,X1V,XV
et=6
J4
III,VIII,XII,XVI,XVII,XVIII
J
IV,IX,XII1,XV1,X1X,XX
i
I
I
6.
t=22,v=7
I b=23l,r=66
I
I
t(22)
I
I'
I
I
I
i
I
3
5
J6
V,X,XIV,XVII,XIX,XXI
J
V1,XI,XV,XV111,XX,XXI
i
,
7
t=22,v=7
7.
b=231,r=165
k=5,A=110
0:=15
t (2:?)
Complement of number 6
72
·e
TABLE A4 (Continued)
Design
number
Parameters
I Sets
of I Judge
pairs
Ii
.~~22,v=21
8.
J1
I,V,IX,XIII,XVII
J2
I,VI,X,XIV,XVIII
k=5,)o..=11
J
I, VII,XI,XV,XIX
0:=5
J4
I,VIII,XII,XVI,XX
J
II,V,X,XV,XX
b=231,r=55
t(22)
3
5
J6
J
.-
7
II,VI,IX,XVI,XIX
I II,VII,XII,XIII,XVIII
J8
II,VIII,XI,XIV,XVII
J
III,V,XI,XVI,XVIII
.
9
J 10
III,VI,XII,XV,XVII
J II
III,VII, IX,XIV,XX
J I2
III,VIII,X,XIII,XIX
J
IV,V,XII,XIV,XIX
13
J 14
IV,VI,XI,XIII,XX
J
IV,VII,X,XVI,XVII
I5
J I6
J
I7
IV,VIII,IX,XV,XVIII
I I,II,III,IV,XXI
J I8
V,VI,VII,VIII,XXI
J
IX,X,XI,XII,XXI
19
J 20
J 21
e
Sets of pairs assigned to each
judge
XIII,XIV,XV,XVI,XXI
IXVII,XVIII,XIX,XX,XXI
73
.e
TABLE A4 (Continued)
Design
Number
Sets of Judge Sets of pairs assigned to each
judge
pairs
Parameters
t=22,v=21
9·
I
b=231,r=176
I Complement
I
t(22)
k=16,"-=132
0=16
of number 8
I
I
I
I
i
TABLE A5
Design
number
"-
Pairs assigned to each judge
Parameters
Judge
t::=6,v=10
J1
(1,4),(1,6),(2,6),(1,5),(2,4),
(3,4),(2,5),(3,6),(3,5)
b=15,r=9
J2
(1,6),(4,6),(2,4),(1,3),(3,4),
(2,5),(1,2),(3,5),(5,6)
k=6, "--==5
J
3
(4,6),(1,6),(2,3),(1,5),(2,4),
(3,6),(1,2),(4,5),(3,5)
0=3
J
4
(4,6),(2,3),(1,5),(2,4),(1,3),
(2,5),(3,6),(5,6),(1,4)
5
(2,6),(1,5),(2,4),(1,3),(3,4),
(3,6),(1,2),(4,5),(5,6)
6
(1,4),(1,6),(2,3),(3,4),(2,5),
(3,6),(1,2),(4,5),(5,6)
7
(1,4),(1,6),(2,3),(2,6),(2,4),
(1,3),(4,5),(3,5),(5,6)
a
(1,4),(4,6),(2,6),(1,3),(2,5),
(3,6),(1,2),(3,5),(4,5)
9
(1,4),(4,6),(2,3),(2,6),(1,5),
(3,4),(1,2),(3,5),(5,6)
1.
J
J
J
J
J
J
10
I
(4,6),(1,6),(2,3),(2,6),(1,5),
(1,3),(3,4),(2,5),(4,5)
Complement of number (iii) of (1.4.4)
74
.e
TABLE A5 (Continued)
Design
number
Parameters
Judge
Pairs assigned to each judge
t=9,v=28
b=36,r=27
Complement of number (iv) of (1.4.4)
2.
k=21,~=20
a=6
'e
75
.e
APPENDIX B
This table gives the distribution of B(3) defined in (2.7.11) for
case
= 4,
g
= k = 2.
P denotes the probability of B(3) attaining
l
the indicated value, whereas P2 denotes the probability that the indit
cated value of B(3) will not be exceeded.
B(3)
P1
P2
0
.ot~3433
0.602
.032060
4
.0 3433
.032403
1.204
.o~
1. 498
.038240
1.806
.0 34349
2
.0 2472
2
.0 1236
~
2.100
2.359
.037210
.039613
2.408
2.631
.038240
2
.0 3021
2
.0 3708
2.702
2.898
2.961
.024 944
.032289
2.996
3.010
3.158
3.233
I
3.304
I
I
3.389
.0 36180
.021442
3777
2
.0 1167
2
.0 2884
2
.0 2747
2
.0 3502
I
B(3)
3.500
II
3.563
P2
1
2
.0 9064
.021236
.03936
.04059
.044 91
3.835
.0 38163
2
.0 3502
.039613
3.857
.01483
.06070
3.991
.01051
2
.0 3021
.07121
3.612
I
P
3.760
2
.0 1877
2
.0 4349
2
.0 5585
2
.0 6306
2
.0 7267
2
.0 8091
4.165
.01111
4.214
.01482
4.362
.01976
4.396
.01999
4.437
.03625
.023204
.02116
4.593
2
.0 3502
4.102
4.129
.04141
.0 4587
.07423
.0115 4
.024120
.08577
.08989
2
.0 1305
2
.0 1167
.09119
.09236
.1286
.1318
I
I
.1353
.02404
4.656
.01401
.02679
4.704
.01007
.1594
4.718
.01112
.1705
.03029
,I
1
.1 493
76
-e
------1----
B(3)
I
I
BO)
Pl
P2
4.816
.034349
.1710
4.887
.04202
2
.0 3891
.2130
.2342
5·110
.01730
2
.0 5219
.239 4
5.195
.01167
.2511
6.287
5.257
.305 4
5.418
.05438
2
.0 6729
2
.0 1450
.3136
5·517
.02101
.3346
5.529
.04230
.3769
5.748
.06303
.4400
5.789
.01634
.4563
4.964
4.990
5.262
.2169
.3122
Pl
P
.5228
5.971
.06647
.027828
I
6.020
.04903
.5796
I
I
6.056
.05136
.026088
.6310
I
I
I
I
5.796
6.243
2
.5306
.6371
.7912
6.316
.1541
.029922
I
6.510
.01913
.8202
I
I
I
I
6.5 47
.8798
6.770
.05953
2
.0 7393
.8871
6.778
.08930
.9764
7.001
.02218
2
.0 1377
·9986
I
I
I
7.224
.8011
1.0000
77
.e
APPENDIX C
In the following table,the
probabilit~P,that a
attained or exceeded is tabulated for k
k -- 2
-4
-
t -
,
-
t
= 4,
I
.•• ,
10j k
- ,
k - 2
= 4,
t
6
t -
r.
P
1.0000
0
1.0000
1
.9990
1
.9999
.90G3
2
.9893
2
.9995
.6565
3
.9453
3
.9963
4
.3438
4
.8281
4
.9824
5
.1094
5
.6230
5
.9408
6
.01563
f
.3770
6
.8491
'"(
.1719
7
.6964
8
.05 469
8
·5000
9
.01074
2
.0 9766
9
.3036
10
.1509
11
.05923
I
12
,
!
13
r.
p
r.
p
0
1.0000
0
1
.9844
2
3
"-
,
= 2,
value L will be
I
II
I
I
I
I
!
10
iI
14
.01758
2
.0 3693
.0 34883
i
15
4
.0 3052
I
I
!
= 8.
·e
k
= 2,
L.
=7
t
:;
4
.9993
5
,
I
==
=8
2, t
t
P
I,
I .9998
6
k
r.
P
= 2,
t
=9
r.
p
8
.9998
23
.06625
6
I .9995
9
.9994
24
.03262
.9981
10
.9980
25
.01441
.9937
11
.9943
26
.Og5666
2
.0 1967
.035966
I
7
.9867
I
8
I
7
.9608
9
.9822
12
.9856
27
8
.9054
10
.9564
13
.9674
28
9
.8083
11
.9075
14
.9338
29
10
.6682
12
.8275
15
.8785
30
11
·5000
13
.7142
16
.7975
31
.57 47
17
.6911
32
II
p
.9999
.9964
I
I
5
I
I
k
I
I
.031563
4
! .0 3480
I
1'.0°:64970857
12
.3318
11-!-
13
.1917
15
.4253
18
.5660
33
14
.09462
Ie
.2858
19
.4340
34
15
.03918
17
.1725
20
.3089
35
.0 61136
8
.0 9706
.095384
.01330
2
.0 3600
18
.09247
21
.2025
36
10
.0 1455
19
.04358
22
.1215
.0 37448
20
.0\106
4
.0 1049
6
.0 4768
21
22
.01785
2
.0 6270
.021860
23
•031~561
24
.0 9000
25
.0 1372
26
.051516
.0 61080
It:
:
,
17
18
j
I
i
,
19
!
20
I!
21
i
,
!
I
:
I
II
27
Ii
,
I
4
i
i
I
4
Ii
!
I
II
I
I
I
I
, 28
_.. _.
._ _
~
_
__'_
...:..._.
.l___ _
~
_
79
.e
k
r.
I
= 2,
t
p
= 10
k
r.
p
r.
= 4,
p
t
=8
I
I
r.
p
11
.9999
29
.03623
94
.0780
12
.9996
30
95
.0599
13
.9988
31
.01785
.028047
110 ! .0322
111
.0316
96
.0454
112
14
.9967
32
97
.0339
113
15
.9920
33
98
.0249
114
16
.9822
34
99
.0181
115
17
.9638
35
100
.0129
120
4
.0 24
.0516
18
.9324
36
101
.0090
125
.0 783
.8837
37
102
.0061
130
20
.8144
38
103
.0041
21
.7243
39
104
.0026
22
.6170
105
.0016
23
.5000
II
41
106
24
.3830
I
42
.0010
.0365
I
I
43
19
II
I
I
2
.0 3304
2
.0 1229
.03'+120
.031235
4
.0 :;287
.057687
.051561
6
.0 2709
.073939
!~o
8
.0 4667
.0 94327
I
25
.2757
26
.1856
27
28
--
:
I
,
.0102944
11
.0 1307
.013 2842
44
.1163
I
.06758
I
I
!
!~5
,
:
107
108
109
.0344
.0331
~
I
II
I
.0310
4
.0 65
.0440
8
.0 29
10
135 j .0 80
11
140
.0 15
.01320
145
.01516
.01711
150
155
21
.0 97
.02552
160
168
,
!
80
.e
BIBLIOGRAPHY
Bose, R. C., "Paired Comparison Designs for Testing Concordance
Between Judges," to appear in Biometrika, 43 (1956).
Bradley, R. A. and Terry, M. E., "The Rank Analysis of Incomplete
Block Designs. I. The Method of Paired Comparisons,"
Biometrika, 39 (195 2 ), 324-345.
f ,_7
Bradley,
1{.
A., "The Rank Analysis of Incomplete Block Designs.
II. Additional Tables for the Method of Paired
Comparisons," Biometrika, 41 (195 4 ), 502-537.
A., "The Rank Analysis of Incomplete Block Designs.
III. Some Large-Sample Results on Estimation and
Power for a Method of Paired Comparisons,"
Biom~trika,
42 (1955), 450- 470.
£5_7
Bradley, R. A., "Incomplete Block Rank Analysis: On Appropriateness of the Model for a Method of Paired Comparisons,"
Biometrics, 10 (195 4), 375-390.
£6_7
Cochran, W. G. and Cox, G. !vi., Experimental Designs, John Wiley
and Sons, New York, 1950.
£7_7
Dykstra, otto, Jr., "A Note on the Rank Analysis of Incomplete
Block Designs; Applications Beyond the Scope of
Existing Tables," Unpublished manuscript, 1956.
£8_7
Ii'isher, H. A. and Yates, F., Statistical Tables for Biological,
Agricultural and Medical Research, Hafner Publishing
Company Inc., New York, 1949.
Kendall, M. G., "Further Contributions to the Theory of Paired
Comparisons," Biometrics, 11 (1955), 43-62.
Kendall, M. G., Advanced Theory of Statistics, Vol. I, Charles
Griffin and Company Limited, London, 1952.
Kendall, M. G., Rank Correlation Methods, Charlee Griffin and
Company Limited, London, 1948.
Kendall, M. G. and Babington smith, B., "On the Method of Paired
Comparisons," Biometrika, 31 (1940), 324-345.
£15_7
Mosteller, !,'rederick, "Remarks on the Method of Paired Comparisons:
I. The Least Squares Solution Assuming Equal
Standard Deviations and Equal Correlations,"
Psychometrika, 16 (1951), 3-9.
81
~14_7
Tables of the Binomial Probability Distribution, Department of
Commerce, National Bureau of Standards, Applied
Mathematics Series 6, Washington, D. C., 1950.
£15_7 Terry, M. E., Bradley, R. A. and Da.vis, L. L., "New Designs
and Techniques for Organoleptic Testing," Food
Technology, 6 (1952),No. 7, 250-254.
---£16_7 Thurstone, L. L., "Psychophysical Analysis," American Journal
of Psychology, 38 (1927), 368-389.
·,e