RESEARCH
REPORTS
I
October, 1966 No. 15
CAN COMPUTERS WRITE
COLLEGE ADMISSIONS TESTS?
Jam es M. Richards, J r .
Published by Research and Development Division • American College Testing Program • P. 0. Box 168, Iowa City, Iowa 52240
Sum m ary
F o r m an y y e a r s p s y c h o l o g i c a l t e s t s have been s c o r e d by m a c h in e s ,
and r e c e n t ly c o m p u t e r s have a s s e m b l e d e x is t in g it e m s into t e s t s and have
s c o r e d e s s a y e x a m in a t io n s .
T h is study g o e s beyond t h e s e e a r l i e r t e c h ­
n iq u e s and e x p l o r e s the p o s s i b i l i t y of c o m p u te r ite m w ritin g .
A co m p u te r
p r o c e d u r e fo r w ritin g V e r b a l C o m p r e h e n s io n i t e m s w a s d e v e lo p e d and u s e d
to w rite a 7 2 - i t e m t e s t .
T h is t e s t , to g e th e r with the Wide R a n g e V o c a b u ­
l a r y T e s t , w a s a d m i n i s t e r e d to U n i v e r s i t y of Iow a f r e s h m e n .
The t e s t
in te r c o r r e l a t i o n s , r e l i a b i l i t i e s , and c o r r e l a t i o n s with g r a d e s s u g g e s t
that, in p r i n c i p l e , c o m p u t e r s can w rite c o l l e g e a d m i s s i o n s t e s t s .
s ib le o b je c t io n s to c o m p u te r w ritte n t e s t s a r e c o n s id e r e d .
P os­
Can C o m p u t e r s W rite C o lle g e A d m i s s i o n s T e s t s ?
J a m e s M. R i c h a r d s , J r .
F o r m an y y e a r s , o b je c t iv e t e s t s and m a c h in e p r o c e d u r e s have
been re d u c in g the r o l e of hum an b e in g s in m e a s u r i n g the ap titu de and
a c h ie v e m e n t of s tu d e n ts .
T he f i r s t l a r g e s c a l e a p p lic a tio n of m a c h in e s
in t e s t in g , of c o u r s e , w as to the s c o r i n g of t e s t s .
With su b s e q u e n t a d ­
v a n c e m e n t in the tech n olo g y of s c o r i n g m a c h in e s , it i s now p o s s i b l e to
s c o r e in two o r t h r e e d a y s the t e s t s of s e v e r a l hundred th o u san d stu d e n ts
who took a nation w ide e x a m in a tio n .
In l a t e r a p p lic a t io n s of m a c h in e s ,
R o c k (1965) h a s shown th at c o m p u t e r s can s i m u l a t e the b e h a v io r of human
t e s t d e v e l o p e r s in a s s e m b l i n g a t e s t with s p e c i f ie d p r o p e r t i e s f r o m an
it e m f i l e ; P a g e (1966) h a s shown that c o m p u t e r s can s c o r e s o m e a s p e c t s
of e s s a y e x a m i n a t io n s ; and O sb u rn (1966) h a s d e v e lo p e d a c o m p u te r p r o ­
c e d u r e fo r w ritin g s t a t i s t i c s p r o b l e m s .
R e c e n t ly , l a r g e , s u p p le m e n t a r y r a n d o m a c c e s s m e m o r y d e v i c e s
have been d e v e lo p e d f o r c o m p u t e r s .
T he c h a r a c t e r i s t i c s of su ch d e v i c e s
a l s o s u g g e s t the p o s s i b i l i t y of w ritin g m u ltip le ch oice t e s t it e m s on a
com puter.
T h e p u r p o s e of the p r e s e n t study, t h e r e f o r e , i s to d e t e r m in e
w h eth er, in p r in c ip le , t e s t s fo r s c r e e n in g c o lle g e a p p lic a n ts can be
w ritten by c o m p u t e r s .
The t e s t f a c t o r m o s t u s e fu l fo r p r e d ic t in g a c a d e m i c s u c c e s s in
c o lle g e , p ro b a b ly , i s V e r b a l C o m p r e h e n s io n .
I atte m p te d , t h e r e f o r e , to
d e v e lo p a p r o c e d u r e that a c o m p u te r could u s e to w rite v e r b a l c o m p r e ­
h en sion i t e m s .
S p e c i f i c a l l y , a p r o c e d u r e f o r w ritin g sy n o n y m s i t e m s w as
-2d e v elo p ed .
A sy n o n y m s t e s t p r o v i d e s a good m e a s u r e of v e r b a l c o m p r e ­
h e n sio n , and the u s e of the Wide R a n g e V o c a b u la r y T e s t f r o m the E T S
F a c t o r Kit m a d e it p o s s i b l e to c o m p a r e a m a c h in e w ritten sy n o n y m s t e s t
with a f a c t o r i a l l y p u re sy n o n y m s t e s t w ritten by h u m a n s.
C o m p u te r P r o c e d u r e
It is r e l a t i v e l y e a s y to d e v e lo p a p r o c e d u r e fo r c h o o sin g a s t e m
and a c o r r e c t a l t e r n a t iv e .
a s u p p le m e n ta r y
One m e r e l y s t o r e s a d ic t io n a r y of sy n o n y m s in
m e m o r y d e v ic e , g e n e r a t e s a ra n d o m n u m b e r, and
u s e s th is n u m b e r to s e l e c t a w ord to be the s t e m .
One then g e n e r a t e s
an o th er r a n d o m n u m b e r and u s e s it to s e l e c t one of the sy n o n y m s f o r the
s t e m a s the c o r r e c t a l t e r n a t iv e .
If the s t e m w ord i s m o r e than one p a r t
of s p e e c h , it i s n e c e s s a r y to c h o o se one p a r t at r a n d o m f o r u s e in the
i te m b e fo r e ch o o sin g the c o r r e c t a l t e r n a t iv e .
J u s t th is p r o c e d u r e w a s u s e d to p ic k s t e m w o r d s and c o r r e c t a l ­
t e r n a t i v e s in the p r e s e n t study.
S in c e no c o m p u te r with a l a r g e s u p p l e ­
m e n t a r y m e m o r y d e v ic e w a s a v a il a b l e , the a c tu a l o p e r a tio n of the c o m ­
p u te r w as sim u la te d .
T he sim u la tio n w a s r i g o r o u s , h ow ev er, and the
i t e m s c o r r e s p o n d e x a c tly to what would be w ritte n by a c o m p u t e r .
T h is
s im u la tio n u s e d one of the o l d e s t r a n d o m a c c e s s s u p p le m e n ta r y m e m o r y
d e v i c e s - -n a m e ly , a book.
A ft e r a s t e m and c o r r e c t a l t e r n a t iv e w e re ch o se n , o b v io u sly the
next s te p w a s to c h o o se " d i s t r a c t o r " a l t e r n a t i v e s ( i . e . , w rong a n s w e r s ) .
It soon b e c a m e c l e a r that d e v e lo p in g a s e n s i b l e p r o c e d u r e fo r c h o o sin g
d i s t r a c t o r s i s the m o s t d iffic u lt p r o b le m in w ritin g t e s t s on a c o m p u te r.
Indeed, a p e r u s a l of the l i t e r a t u r e , a f t e r th is p r o b le m b e c a m e o b v io u s ,
-3s u g g e s t e d th at d i s t r a c t o r a l t e r n a t i v e s a r e little i n v e s t ig a t e d and little
u n d e r s to o d and a r e p e r h a p s the m o s t i m p o r t a n t n e g le c t e d p r o b l e m in
the c o n s t r u c t io n of m u ltip le ch o ice t e s t s .
M o s t w r i t e r s on t e s t d e v e l o p ­
m e n t m e r e l y s u g g e s t that the ite m w r i t e r m u s t u s e h is ju d g m e n t and
in gen uity in ch o o sin g d i s t r a c t o r s.
S in ce c o m p u t e r s have n e it h e r j u d g ­
m e n t n o r ingenu ity, such a d v ic e w a s u s e l e s s fo r the p r e s e n t study.
T he p r o b l e m of d i s t r a c t o r s fo r sy n o n y m s it e m s w a s so lv e d in the
fo llo w in g way.
R o g e t 's T h e s a u r u s p r o v i d e s a c l a s s i f i c a t i o n s c h e m e fo r
w ord m e a n i n g s in which e a c h w ord i s c a t e g o r i z e d into one or m o r e c a t e ­
g o rie s.
N u m b e r s a r e a s s i g n e d to t h e se c a t e g o r i e s of m e a n in g in a c c o r ­
d a n ce with an o v e r a l l c o n c e p tu a l s c h e m e .
In the p r e s e n t study, the
b a s i c p r o c e d u r e fo r p ick in g d i s t r a c t o r s w a s to c h o o se r a n d o m ly f r o m
w o r d s in a d ja c e n t c a t e g o r i e s in the T h e s a u r u s s c h e m e .
M o r e s p e c i f i c a l l y , the n u m e r i c a l code o r , in o th er w o r d s , the
c a t e g o r y of m e a n in g , s h a r e d by the s t e m and the c o r r e c t a l t e r n a t iv e w as
d e t e r m in e d .
If m o r e than one n u m e r i c a l code p e r t a in e d , a r a n d o m n u m ­
b e r w a s g e n e r a t e d and u s e d to s e l e c t one of the c o d e s .
T he p r o c e d u r e
f o r m o d ify in g the n u m e r i c a l code a s a st e p in p ic k in g d i s t r a c t o r s f r o m
a d ja c e n t c a t e g o r i e s c o n s i s t e d of adding J_ to or su b t r a c tin g
co d e n u m b e r f o r the s t e m and c o r r e c t a lt e r n a t iv e .
f r o m the
A random num ber
d e t e r m i n e d the ch o ice betw een ad dition o r su b t r a c tio n .
Then the d i s ­
t r a c t o r w a s ch o se n at r a n d o m f r o m w o r d s with the m o d ifie d code n u m b e r
and of the a p p r o p r i a t e p a r t of s p e e c h .
A ch e c k w a s then m a d e to d e t e r ­
m in e w hether the d i s t r a c t o r had any code n u m b e r s in co m m o n with e ith e r
the s t e m o r the c o r r e c t a l t e r n a t iv e .
If so , a new d i s t r a c t o r w a s c h o sen
-4 a t ra n d o m .
A d d itio n al d i s t r a c t o r s w e re ch o sen by r e p e a t i n g the p r o c e s s
of m o d ify in g the code n u m b e r and ch o o sin g a d i s t r a c t o r at ra n d o m .
T h u s,
the f i r s t d i s t r a c t o r w a s ch o se n f r o m w o r d s one c a t e g o r y aw ay f r o m the
s t e m , the se c o n d d i s t r a c t o r f r o m w o r d s two c a t e g o r i e s aw ay, e tc.
In
a few i n s t a n c e s , o f c o u r s e , a l l the w o r d s in a c a t e g o r y would s h a r e a
co m m o n m e a n in g with the s t e m .
In su ch c a s e s , the p r o c e d u r e w a s s i m ­
ply to m od ify the code n u m b e r and c h o o se a d i s t r a c t o r f r o m the n ex t c a t e ­
gory.
The lo c a t io n s of the c o r r e c t a l t e r n a t iv e and the v a r i o u s d i s t r a c t o r
a l t e r n a t i v e s in the ite m w e re a l s o d e t e r m in e d by ra n d o m n u m b e r s .
In t h is w ay, a 7 2 - it e m c o m p u te r w ritten (or at l e a s t w r ita b le ) s y n ­
o n y m s t e s t with f o u r - a l t e r n a t i v e it e m s w a s d e v e lo p e d .
t e s t s have s e v e r a l in t e r e s t in g p r o p e r t i e s .
Such i t e m s and
F i r s t of a ll, su ch i t e m s a r e
d e fin ite ly a ra n d o m s a m p l e f r o m a s p e c i f ie d p o p u latio n of i t e m s .
T h u s,
/
a t e s t c o m p o s e d of th e m would c o n fo r m r i g o r o u s l y to an a s s u m p t i o n fo r
s e v e r a l w ay s of e s t im a t i n g r e l i a b i li t y .
S eco n d , while the p o p u latio n of
i t e m s i s v e r y l a r g e , it i s c e r t a i n l y fin ite.
S i m i l a r l y , while the n u m b e r
of a l t e r n a t iv e t e s t f o r m s of a giv en length that can be a s s e m b l e d f r o m
su ch i t e m s i s v e r y , v e r y l a r g e , it a l s o i s fin ite .
It a p p e a r s , m o r e o v e r ,
that the fin ite qu ality r e s u l t s p r i m a r i l y f r o m the c o m p le te s p e c i f ic a t i o n
of the o p e r a t io n s fo r w ritin g i t e m s and m a y be co m m o n to m an y , if not
m o s t , i t e m p o p u la tio n s.
S in ce the popu lation of i t e m s and t e s t s i s v e r y
l a r g e , t h is h a s few p r a c t i c a l i m p l i c a t i o n s .
p l i c a t io n s , h ow ev er.
It m a y h av e t h e o r e t i c a l i m ­
F o r e x a m p le , tru e s c o r e s a r e u s u a l l y d efin ed in
t e r m s of an infinite n u m b e r of t e s t s .
M o r e o v e r , it a p p e a r s that s i m i l a r p r o c e d u r e s can be u s e d to w rite
-5 m a n y o th er k in ds of m u ltip le ch oice ap titu de t e s t i t e m s .
With only m in o r
m o d if ic a t io n s the p r o c e d u r e u s e d in the p r e s e n t study could be u s e d to
w rite a n a l o g i e s i t e m s .
In addition , no in su r m o u n ta b le d i f f ic u l t i e s should
be e n c o u n te re d in p r o g r a m m i n g a c o m p u te r to w rite m a t h e m a t i c s i t e m s ,
alth ou gh d e v e lo p in g e ffic ie n t d i s t r a c t o r s m igh t be a p r o b le m . * T he w ork
of O sb u rn (1966) i s a p r o m i s i n g s t e p to w ard c o m p u te r w ritten m ath i t e m s .
E v a lu a t io n of the T e s t
The c o m p u te r w ritten t e s t , to g e th e r with the Wide R a n g e V o c a b u la r y
T e s t w a s a d m i n i s t e r e d in the f a l l of 1965 to e n te rin g f r e s h m e n at the U n i­
v e r s i t y of Iow a.
A g e n e r o u s tim e lim it p e r m it t e d a l l stu d e n ts to c o m p le te
the c o m p u te r t e s t .
At the end of the f i r s t s e m e s t e r , the g r a d e point a v e r -
a g e w a s d e t e r m in e d f o r t h e s e s tu d e n ts.
The m e a n s , s t a n d a r d d e v ia t io n s,
K - R 21 r e l i a b i l i t i e s , p r e d ic t iv e v a l i d i t i e s , and in te r c o r r e l a t i o n of the
two t e s t s ^ f o r 599 m a l e and 613 f e m a l e f r e s h m e n a r e shown in T a b le 1.
T h e s e r e s u l t s a r e p a r t l y bad and p a r t l y good.
T he i t e m s in the
c o m p u te r w ritten t e s t , on the a v e r a g e , a r e e a s i e r than th o se in the Wide
R a n g e t e s t ( c o m p a r e the m e a n to the length fo r both t e s t s ) , and the r e l i ­
a b i l i t i e s a r e lo w e r f o r the c o m p u te r t e s t .
T h is i s d isa p p o in tin g b e c a u s e
the g r e a t e r length of the c o m p u te r t e s t should p ro d u c e h igh er r e l i a b i li t y .
l i t i s l e s s c l e a r that a l l p o s s i b l e t y p e s of i t e m s can be w ritten by
c o m p u t e r s . Jo h n H olland ( p e r s o n a l c o m m u n ic a tio n s) h a s s u g g e s t e d a
m o d e l fo r i t e m s in which a l t e r n a t i v e s a r e p o in ts e q u i- d is t a n t in a r e ­
s p o n s e s p a c e . It i s m o s t u n lik ely that e x is t in g c o m p u t e r s can w rite such
ite m s.
^1 want to thank T e d M c C a r r e l , C h a r l e s B. S t a t l e r , and W illard L.
B oyd f o r m a k in g it p o s s i b l e fo r m e to obtain t h e se d a ta .
3D ata a n a l y s i s fo r th is study c a r r i e d out at the U n iv e r s it y of Utah
c o m p u te r c e n te r .
-6On the o th er hand, the v a l i d i t i e s a r e c o m p a r a b l e to th o s e fo r the Wide
R a n g e t e s t , and the in te r c o r r e la tio n of the two t e s t s i s not too f a r f r o m
the l i m i t s se t by r e l i a b i li t y .
On the whole, t h e s e r e s u l t s a r e e n c o u r a g ­
ing enough to j u s t i f y fu r t h e r a n a l y s i s .
T ab le 1
M ean , S ta n d a r d D ev iatio n , R e l ia b i l i t y , V a lid ity , and In te r c o r r e l a t i o n
of Wide R a n g e V o c a b u la r y T e s t and the C o m p u te r W ritten T e s t
M ales
(N = 599)
F em ale s
( N = 6 13)
48
48
M ean
S ta n d a r d D e v iatio n
23. 74
6 .4 4
25. 16
6. 42
K - R 21 R e l ia b i l i t y
. 73
. 73
C o r r e l a t i o n with f i r s t s e m e s t e r G P A
. 30
. 30
Wide R a n g e V o c a b u la r y T e s t
N u m b e r of I t e m s
C o m p u te r W ritten T e s t
N u m b er of I t e m s
72
72
M ean
S ta n d a r d D e v iatio n
53. 88
5. 76
55. 53
5. 41
K - R 21 R e l ia b i l i t y
. 60
. 57
C o r r e l a t i o n with f i r s t s e m e s t e r G P A
. 30
. 32
I n te r c o r r e l a t i o n of two t e s t s
. 62
. 64
I n te r c o r r e la t i o n c o r r e c t e d f o r u n r e lia b ilit y
of both t e s t s
. 94
. 98
S in c e the i t e m s in the Wide R a n g e V o c a b u la r y T e s t w e r e c a r e f u l l y
s e l e c t e d , while the c o m p u te r t e s t c o n s i s t e d of e n t i r e l y u n s e le c t e d i t e m s
the e ff e c t of i t e m s e le c t io n sh ould be e v a lu a te d . ^ The Wide R a n g e t e s t
c o n s i s t s of two p a r t s with 24 i t e m s e a c h .
A c c o r d in g ly , 24 i t e m t e s t s
w e r e s e l e c t e d f r o m the c o m p u te r t e s t by t h r e e d if fe r e n t p r o c e d u r e s .
T he i t e m s e le c t io n w a s b a s e d only on d a ta fo r m a l e s , so th a t the f e m a l e
s a m p l e would p r o v id e a c r o s s - v a l i d a t i o n gro u p .
The f i r s t p r o c e d u r e ,
d e s i g n e d to m a x i m i z e v a lid ity , w a s s im p ly to p ic k the 24 i t e m s with the
h ig h e s t c o r r e l a t i o n with f i r s t s e m e s t e r g r a d e s .
The se c o n d p r o c e d u r e ,
d e s i g n e d to m a x i m i z e h o m o g e n e ity , w a s to p ic k the 24 i t e m s with the
h ig h e s t c o r r e l a t i o n with the C o m p u te r T e s t to ta l.
The th ird p r o c e d u r e ,
d e s i g n e d f o r high d i s c r i m i n a t io n am o n g stu d e n ts t e s t e d , w a s to e lim in a te
i t e m s a n s w e r e d c o r r e c t l y by m o r e than 90% or fe w e r than 10% and then
c h o o s e the 24 i t e m s with the h ig h e st c o r r e l a t i o n with the C o m p u te r T e s t
t o t a l.
E a c h of the p r o c e d u r e s , of c o u r s e , could e a s i l y be in c o r p o r a t e d
in a c o m p u te r p r o g r a m fo r a s s e m b l i n g t e s t f o r m s .
D a ta f o r the two p a r t s of the Wide R a n g e V o c a b u la r y T e s t and the
t h r e e d if f e r e n t t e s t s c o m p o s e d of s e l e c t e d i t e m s f r o m the C o m p u te r
W ritten T e s t a r e s u m m a r i z e d in T a b le 2.
It sh ould be r e m e m b e r e d that
s o m e v a l u e s fo r m a l e s a r e in fla te d b e c a u s e the c o r r e l a t i o n s and ite m
s e l e c t io n a r e b a s e d on the s a m e g ro u p .
T h e r e i s a l s o c o n s i d e r a b le
i t e m o v e r l a p in the th r e e c o m p u te r t e s t s , which p r o d u c e s m u c h in fla te d
in t e r c o r r e la tio n s.
While the c o m p u te r t e s t i s s t i l l so m e w h a t e a s i e r , t h e s e d a ta c o n ­
f i r m without q u a lific a tio n th at sy n o n y m s t e s t s can be s u c c e s s f u l l y w ritten
4 lt sh ould be noted, h o w e v e r, that s e l e c t e d i t e m s a r e no lo n g e r a
ra n d o m s a m p l e f r o m the s p e c i f ie d p o p u latio n .
T a b le 2
C o m p a r is o n of Two P a r t s of Wide R a n g e T e s t with T e s t s
C o m p o s e d of S e le c t e d C o m p u te r W ritten I t e m s
1
2
M a l e s (N=599)
4
3
5
Wide R a n g e T e s t
1. P a r t 1
2. P a r t 2
. 51
C o m p u te r
3. I t e m s
4. I te m s
5. I te m s
. 62
. 61
. 64
. 43
. 41
. 45
. 93
. 89
. 88
. 56
K - R 21 R e lia b ilit y
C o r r e l a t i o n with f i r s t s e m e s t e r G P A
. 29
24
N u m b e r of i t e m s
12. 32
M ean
S ta n d a rd D ev iation
3. 59
. 71
. 21
. 61
. 41
. 60
. 35
. 63
. 35
24
11. 54
4. 34
24
18. 63
3. 17
24
19. 03
3. 08
24
14. 94
3. 78
T est
s e l e c t e d fo r v a lid ity
s e l e c t e d fo r h om ogen eity
s e l e c t e d fo r d i s c r i m i n a t io n
F e m a l e s (N= 613)
1
2
Wide R a n g e T e s t
1. P a r t 1
2. P a r t 2
. 50
--
C o m p u te r
3. I t e m s
4. I t e m s
5. I t e m s
. 52
. 45
. 51
. 68
K - R 21 R e l ia b i l i t y
C o r r e l a t i o n with f i r s t s e m e s t e r G P A
. 28
24
N u m b e r of i t e m s
13. 57
M ean
4. 14
S ta n d a rd D e v iation
T est
s e l e c t e d fo r v a lid ity
s e l e c t e d fo r h o m o ge n eity
s e l e c t e d fo r d i s c r i m i n a t io n
3
4
5
. 56
. 56
. 58
-.9 2
. 89
. 88
. 41
. 22
. 55
. 30
. 53
. 29
. 61
. 29
24
11. 58
3. 14
24
19. 30
2. 82
24
19. 66
2. 69
24
16. 46
3. 52
N o t e - - S o m e v a l u e s fo r m a l e s a r e in fla ted b e c a u s e i t e m s e l e c t io n and
c o r r e l a t i o n s a r e b a s e d on the s a m e g ro u p . In te r c o r r e l a t i o n s of c o m p u te r t e s t s
a r e in fla te d by s u b s t a n t ia l ite m o v e r la p .
-9on c o m p u t e r s .
E a c h of the t h r e e c o m p u te r w ritten t e s t s c o r r e l a t e s about
a s h ig h ly with the two p a r t s of the Wide R a n g e T e s t a s the two p a r t s c o r ­
r e l a t e with e a c h o th e r; the r e l i a b i li t y of e a c h c o m p u te r t e s t i s so m e w h e r e
betw een the r e l i a b i l i t i e s of the two p a r t s of the Wide R a n g e T e s t ; and the
p r e d i c t iv e v a l i d i t i e s of the c o m p u te r t e s t s a r e , if anything, s lig h t ly h ig h e r
than the v a l i d i t i e s of the Wide R a n g e T e s t .
D iscu ssio n
The r e s u l t s c l e a r l y in d ic a te that it i s p o s s i b l e , in p r i n c i p l e , fo r
c o m p u t e r s to w rite c o lle g e a d m i s s i o n s t e s t s , which s u g g e s t s that, with
a th o ro u gh a p p lic a tio n of e x i s t in g c o m p u te r tech n olo g y , it would be e n ­
t i r e l y f e a s i b l e , in p r i n c i p l e , to a u to m a te a l l a s p e c t s of c o lle g e a d m i s s i o n s
testin g .
It would be p o s s i b l e to i n s t a l l in e a c h high sc h o o l an in p u t-d e v ic e
to a c e n t r a l c o m p u te r at a t e s t i n g a g e n c y .
On th is d e v ic e , the c o m p u te r
w ould d i s p l a y t e s t i t e m s one by one on a s c r e e n ; the stu dent would push
a button to in d ic a te h is r e s p o n s e to e ach ite m ; and h is r e s p o n s e would be
t r a n s m i t t e d b a c k to the c o m p u t e r .
cou ld be t e s t e d at o n ce.
T h ro u g h t im e s h a r in g , m an y stu d e n ts
T he c e n t r a l c o m p u te r would g e n e r a t e new i t e m s ,
in clu d e a few with e a c h t e s t a d m i n i s t e r e d , d e v e lo p and e q u a te a lt e r n a t e
f o r m s of the t e s t , s c o r e e a c h s t u d e n t 's t e s t , and even t r a n s m i t the s c o r e s
d i r e c t l y to a n o th e r c o m p u te r (o r output d e v ic e ) at the c o lle g e of the s t u d e n t 's
ch o ic e .
T h u s it would a l s o be p o s s i b l e to e lim in a t e a n s w e r s h e e t s , s c o r i n g
m a c h i n e s , and s c o r e r e p o r t f o r m s .
A ll of th is could tak e p l a c e e n t ir e ly
"u ntouched by hum an h a n d s. "
L e t m e e m p h a s i z e that while th is i s p o s s i b l e with e x is t in g c o m p u te r
te ch n o lo g y , a p o s s i b i l i t y i s not an i m p e r a t i v e , and t h e r e a r e m an y l e g it i m a t e
-10o b je c tio n s to such a d e h u m a n iz e d s y s t e m .
I hope, h o w e v e r, that h u m a n ­
i s t s and o th e r s a p p a lle d by the 1984 o v e r to n e s w ill m a k e a m o r e s e r i o u s
r e s p o n s e to it than an im p lie d Lu d dite c o m p u te r s m a s h i n g .
F o r if c o m ­
p u ter w ritten c o lle g e a d m i s s i o n s t e s t s a r e o b je c t io n a b le , r e je c t io n of
th em on s o m e b a s i s o th er than p r e d ic t iv e v a lid ity , while le a v in g u n ­
ch an ged the s o c i a l s y s t e m m a k in g high v a lid ity p o s s i b l e , would be a
p a r t i c u l a r l y d is h o n e s t and d e p lo r a b le c a s e of r e m o v in g sy m p to n s r a t h e r
than t r e a t in g the u n d e rly in g d i s e a s e .
Any s e r i o u s co n fro n tatio n of the i s s u e s involved m u s t r e c o g n i z e what
i s , on the e v id e n c e , an in d isp u ta b le f a c t :
n a m e ly , that m u ltip le ch o ice
t e s t s - - e v e n th o se w ritten by c o m p u t e r s - - a r e the b e s t and f a i r e s t c u r r e n t ly
a v a ila b le e s t i m a t e s of p o te n tia l f o r a c a d e m i c s u c c e s s in c o l l e g e s a s they
a r e now c o n stitu te d .
T h is i s t r u e in the s e n s e that an ab le c o l l e g e a p p l i ­
cant, if he i s beh aving r a tio n a lly in h is own s e l f i n t e r e s t , w ill tak e su ch a
t e s t in p r e f e r e n c e to o th e r s w ay s of having h is a c a d e m ic p o te n tia l e v a lu a te d ,
su ch a s tak in g an e s s a y e x a m in a tio n or being in te rv ie w e d .
What convincin g a r g u m e n t , then, can be a d v a n c e d a g a i n s t h avin g su ch
t e s t s w ritten by a c o m p u te r if the p r o d u c ts a r e in d is t in g u is h a b le f r o m , or
s u p e r i o r to, t e s t s w ritten by p e o p le ?
I think any s e r i o u s c r i t i c i s m of such
t e s t s m u s t r a t h e r b eg in with a c r i t i c i s m of co n v en tio n al m e a s u r e s of s u c ­
c e s s in c o lle g e , o r, to put it bluntly, of g r a d e s giv en by c o lle g e p r o f e s s o r s .
If g r a d e s can be p r e d ic t e d quite w ell by a c o m p le te ly d e h u m a n iz e d t e s t ,
j u s t what i s w rong with g r a d e s ?
T h at so m e th in g m a y in deed be s e r i o u s l y
w rong i s in d ic a te d by the m an y s t u d ie s show ing, at b e s t , a n e g lig ib le r e l a ­
tio n sh ip betw een g r a d e s and p e r f o r m a n c e o u tsid e the c l a s s r o o m in im p o r ta n t
a r e a of hum an e n d e a v o r (Hoyt, 1966; R i c h a r d s , H olland, and L u tz ,
1966).
R eferen ces
Hoyt, D, P . C o lle g e g r a d e s and ad u lt a c c o m p lis h m e n t :
research .
O sb u rn , H. G.
E d u c a t io n a l R e c o r d ,
A r e v ie w of
1966, 47, 70 -7 5 .
C o m p u te r aid e d ite m s a m p lin g fo r a c h ie v e m e n t t e stin g .
P a p e r r e a d at A m e r i c a n P s y c h o l o g i c a l A s s o c i a t i o n , New Y o rk ,
1966.
P a g e , E . B.
A u to m a tic g r a d in g of student e s s a y s .
S y m p o s iu m p r e s e n ­
tatio n at A m e r i c a n E d u c a t io n a l R e s e a r c h A s s o c i a t i o n , C h icag o ,
1966 .
R i c h a r d s , J . M. J r . , H olland, J .
L, & L u tz , S a n d r a W.
of student a c c o m p l is h m e n t in c o lle g e .
13.
R o c k , D.
Iow a C ity, Iow a:
The p re d ic tio n
A C T R e s e a r c h R e p o r t No.
A m e r i c a n C o lle g e T e s t i n g P r o g r a m ,
A s s e m b l y of t e s t s by u s e of an a u to m a te d it e m file .
M e m o r a n d u m 6 5 -1 4 .
S e rv ic e ,
1965.
P r in c e t o n , New J e r s e y :
1966.
R esearch
E d u c a t io n a l T e s tin g
ACT R esearch Reports
T h is r e p o r t i s the fifteen th in a s e r i e s p u b lish ed by the R e s e a r c h
and D e v e lo p m e n t D iv isio n of A m e r i c a n C o lle g e T e s tin g P r o g r a m . R e p o r t s
a r e p u b lish e d m onthly and m a ile d f r e e of c h a r g e to e d u c a t o r s and o th er
i n t e r e s t e d p e r s o n s who have a s k e d to be on the s p e c i a l r e s e a r c h r e p o r t
m a i l in g l i s t .
The r e s e a r c h r e p o r t s h av e been d e p o s it e d with the A m e r i c a n D o c u ­
m e n ta tio n In stitu te , ADI A u x ilia r y P u b lic a t io n s P r o j e c t , P h o to d u p lica tio n
S e r v i c e , L i b r a r y of C o n g r e s s , W ashington 25, D. C. (ADI D ocum ent
n u m b e r s and p r i c e s a r e giv en below . ) P r in t e d c o p ie s m a y be obtained
f r e e f r o m the R e s e a r c h and D e v e lo p m e n t D iv isio n , A m e r i c a n C o lle g e
T e s t i n g P r o g r a m . P h o t o c o p ie s and 35 m m , m i c r o f i l m s m a y be obtained
at c o s t f r o m ADI by citin g ADI D o cu m en t n u m b e r. A d v an ce p ay m e n t i s
r e q u i r e d . M ake c h e c k s or m on ey o r d e r s p a y a b le to: C h ief, P h o t o ­
d u p lic a tio n S e r v i c e , L i b r a r y of C o n g r e s s .
R e p o r t s in the s e r i e s a r e :
*N o .
1 A D e s c r ip t io n of A m e r i c a n C o lle g e F r e s h m e n , by C. Abe,
J . L. H olland, S a n d r a W. L u tz , & J . M. R i c h a r d s , J r .
(ADI D oc. 8554; photo, $8. 75; m i c r o f i l m , $3. 00)
* N o . 2 A c a d e m ic and N o n - a c a d e m ic A c c o m p l i s h m e n t : C o r r e l a t e d
or U n c o r r e l a t e d ? by J . L. H olland, & J . M. R i c h a r d s , J r .
(ADI D oc. 8555; photo, $3. 75; m i c r o f i l m , $2. 00)
No. 3 A D e s c r ip t io n of C o lle g e F r e s h m e n : I. Stud en ts with D i f ­
f e r e n t C h o ic e s of M a j o r F i e l d , by C. A b e . , & J . L . Holland
(ADI D oc. 8556; photo, $7. 50; m i c r o f i l m , $2. 75)
No. 4 A D e s c r ip t io n of C o lle g e F r e s h m e n : II. S tu d en ts with D i f ­
f e r e n t V o c a tio n a l C h o ic e s , by C. A be. , & J . L. H olland
(ADI D oc. 8557; photo, $7. 50; m i c r o f i l m , $2. 75)
* N o . 5 A D e s c r ip t io n of J u n io r C o l l e g e s , by J . M. R i c h a r d s , J r . ,
L o r r a i n e M. R an d , & L . P . Rand
(ADI Doc. 8558; photo, $3. 75; m i c r o f i l m , $2. 00)
N o. 6 C o m p a r a t iv e P r e d i c t i v e V a l i d i t i e s of the A m e r i c a n C o lle g e
T e s t s and Two Other S c h o l a s t i c A ptitude T e s t s , by L. Munday
(ADI D oc. 8559; photo, $2. 50; m i c r o f i l m , $ 1 .7 5 )
* T h i s r e p o r t now a v a il a b l e only f r o m ADI.
A C T R e s e a r c h R e p o r t s (cont. )
No. 7 The R e la t io n s h ip B etw een C o lle g e G r a d e s and A dult
A c h ie v e m e n t . A R e v ie w of the L i t e r a t u r e ,
by D. P . Hoyt
(ADI Doc. 8632; photo, $7. 50; m i c r o f i l m ,
$2. 75)
No. 8 A F a c t o r A n a l y s i s of Student " E x p l a n a t i o n s 11 of T h e ir
C h oice of a C o l l e g e , by J . M. R i c h a r d s , J r . , & J . L. H olland
(ADI D oc. 8633; photo, $3. 75; m i c r o f i l m , $2. 00)
N o. 9 R e g io n a l D i f f e r e n c e s in J u n io r C o l l e g e s , by J . M. R i c h a r d s ,
J r . , L . P . R and, & L o r r a i n e M. R an d
(ADI D oc. 8743; photo, $2. 50; m i c r o f i l m , $1. 75)
No. 10 A c a d e m ic D e s c r ip t io n and P r e d i c t io n i n J u n i o r C o l l e g e s , by
D. P . Hoyt, & L. M unday
(ADI D o c. 8856; photo, $3. 75; m i c r o f i l m ,
$2. 00)
No. 11 The A s s e s s m e n t of Student A c c o m p lis h m e n t in C o l l e g e , by
J . M. R i c h a r d s , J r . , J . L. H ollan d,
S a n d r a W. Lutz
(ADI D oc. 8955; photo, $3, 75; m i c r o f i l m , $2. 00)
No. 12 A c a d e m ic and N o n - a c a d e m i c A c c o m p lis h m e n t in a R e p r e ­
se n ta tiv e S a m p le tak e n f r o m a_ P o p u la tio n of 612, 0 0 0 , by
J . L. H olland, & J . M. R i c h a r d s , J r .
(ADI D oc. 8992; photo, $3, 75; m i c r o f i l m , $2. 00)
No. 13 The P r e d i c t io n of Student A c c o m p lis h m e n t in C o l l e g e , by
J . M. R i c h a r d s , J r . , J . L . H ollan d, & S a n d r a W. Lutz
(ADI D oc. 9020; photo, $5. 00; m i c r o f i l m , $2. 25)
No. 14 C h a n g e s in S e l f - R a t i n g s and L if e G o a l s A m on g Stu d e n ts
at C o l l e g e s with D iffe r e n t C h a r a c t e r i s t i c s , by R . W. S k a g e r ,
J . L. H olland, & L. A. B r a s k a m p
(ADI D oc. 9069; photo, $3. 75; m i c r o f i l m , $2. 00)
----------