11/20/2008
MPhil Social and Developmental Psychology
1: Test Construction
Professor John Rust
University of Cambridge
Outline
•
•
•
•
•
•
What is psychometrics?
The history of psychometrics
Break
How to find a test
How to construct a test
How to evaluate a test
2
1
11/20/2008
What is psychometrics?
• “The science of psychological assessment”
• Much assessment is “high stakes”
•
•
•
•
•
•
Recruitment and staff development
Licensing and chartering (eg Accountants, Surgeons)
School and University examinations
Psychiatric and ‘special needs’ diagnosis
Credit ratings
Career guidance
Types of assessment
•
•
•
•
•
•
•
First impressions
Application forms and references
Objective tests (on or off line)
Projective tests
Interviews
Essays and examinations
Biodata
4
2
11/20/2008
The Psychometric Principles
Maximizing the quality of assessment
• Reliability
• Validity
• Standardisation
• Equivalence
5
Reliability
unreliable
reliable
3
11/20/2008
Validity
Quite reliable - but invalid
Standardisation
• Norm referenced
• Criterion referenced
• Self referenced (Ipsative)
4
11/20/2008
Equivalence
History of psychometrics
•
•
•
•
•
The ancient world
Emergence of the medical model
Psychophysics
Intelligence and evolution
Modern psychometrics
5
11/20/2008
The ancient world
•
•
•
•
•
Chan dynasty (C. 1,000 BC)
Chinese Examination system (C. 300 BC)
Paper (Battle of Talas, AD 751)
Jesuits (C. 1600 AD), Padua, Paris
Indian Civil Service 19th Century
The Medical Model
• Paupers and the workhouse
– http://www.mdx.ac.uk/WWW/STUDY/4_13_Ta.htm#London
• Lunatic asylums: Illness vs Handicap
• Institutional care for 'idiots and imbeciles'
• Seguin Form Board and other diagnostics
6
11/20/2008
Psychophysics
•
•
•
•
Wundt and the Leipzig Laboratory
Galton and Anthropometrics
James McKeen Cattell in Cambridge
Alfred Binet
Galton’s Anthropometric Laboratory
14
7
11/20/2008
Venn’s 1886 Cambridge study
• Eyesight, strength (pull and squeeze), head
size and shape etc. N = 1,450 students.
• 'intellectual characteristics‘ assessed by
college tutors: 'first-class man‘,'remaining
honours men‘ & 'poll-men'.
• “There does not seem to be the slightest
difference between one class of our students
and another” (Venn, 1889).
Cattell’s psychometric laboratory
8
11/20/2008
Jim Watson and Jensen’s argument
• In his controversial paper, Jensen argued:
– IQ test scores of races differ
– Twin studies show that IQ is at least 50% inherited
– the 1970s ‘head start’ programme was doomed to failure
as race differences were inherited.
• The counter arguments then:
– Kamin claimed twin studies were flawed
– The Burt controversy
• Jensen argument taken up by Eysenck in his book
“Race and Intelligence”
Race and IQ today
• Charles Murray and Richard Herrnstein
(1994) “The Bell Curve: Intelligence and
class structure in American life”.
• American Enterprise Institute, Pioneer
Fund, Heritage Foundation, Eugenics
Society
9
11/20/2008
The Flynn Effect
“What is intelligence?”(2007)
Wechsler Intelligence
Scale for Children
WISC
1949
Average
at WISC
Rate of
change p.a.
100
WISC-R
1974
107.63
0.311
WISC-III
1991
113.00
0.322
WISC-IV
2003
117.63
0.363
The Flynn effect: Paradox 4
• The identical twins paradox: Twin studies show that genes
dominate individual differences in IQ and that environmental
effects are feeble. IQ gains are so great as to signal the
existence of environmental factors of enormous potency. How
can environment be both so feeble and so potent?
• At a given time, genetic differences between individuals (within
a cohort) are dominant but only because they have hitched
powerful environmental factors to their star. Trends over time
(between cohorts) liberate environmental factors from the
sway of genes and once unleashed, they can have a powerful
cumulative effect.
•
www.thepsychometricscentre.co.uk/publications/beyonetheflynneffect
10
11/20/2008
The Flynn effect: Paradox 2
•
•
The intelligence paradox: If huge IQ gains are intelligence gains, why are
we not stuck by the extraordinary subtlety of our children's
conversation? Why do we not have to make allowances for the limitations
of our parents? A difference of some 18 points in the average IQ over two
generations ought to be highly visible.
Asking whether IQ gains are intelligence gains is the wrong question
because it implies all or nothing cognitive progress. The 20th century has
seen some cognitive skills make great gains, while others have been in the
doldrums. To assess cognitive trends, we must dissect "intelligence" into
solving mathematical problems, interpreting the great works of literature,
finding on-the-spot solutions, assimilating the scientific world view, critical
acumen, and wisdom.
The Flynn effect: Paradox 3
•
•
The MR paradox: In 1900, the average IQ scored against current norms
was somewhere between 50 and 70. If IQ gains are in any sense real, we
are driven to the absurd conclusion that a majority of our ancestors were
mentally retarded.
Our ancestors in 1900 were not mentally retarded. Their intelligence was
anchored in everyday reality. We differ from them in that we can use
abstractions and logic and the hypothetical to attack the formal problems
that arise when science liberates thought from concrete referents. Since
1950, we have become more ingenious in going beyond previously learned
rules to solve problems on the spot.
11
11/20/2008
The stances
• The man of science defending freedom of
expression
• “I’m not saying they (Black, Romany, Hutu,
Scheduled Caste, Irish etc) are inferior.… just
different”
• “Let’s not talk about this” (its too hot!)
• If we don’t talk about it – we can’t teach
people that it isn’t true
What’s in a word?
•Intelligence
•I.Q.
•Ability
•Aptitude
•Achievement
•Competency
•Potential
12
11/20/2008
Spearman’s “g”
psychophysiological energy
Signals in neurones
13
11/20/2008
Brain scan
Some antics from before
•Intelligence as Giftedness
•Gifted by the Gods? (Homer)
•Or Gifted by (Mother) Nature? (Plato)
•Or the Heavenly Beam? (Jenner)
14
11/20/2008
Homer’s Odyssey 600 BC
•“You are an insolent fellow. True it is that the Gods do not grace
all men alike in speech, person and understanding. One man may
be of weak presence, but heaven has adorned this with such a
good conversation that he charms every one who sees him; his
honeyed moderation carries his hearers with him so that he is a
leader in all assemblies of his fellows, and where-ever he goes he
is looked up to. Another may be as handsome as a God, but his
good looks are not crowned with discretion. This is your case. No
God could make a finer looking fellow than you are – but you are a
fool.” (4,233) (Ulysses to Euryalus)
Plato’s Republic Book 5
•Socrates asks Glaucon:
•“When you spoke of a nature gifted or not gifted in any respect,
did you mean to say that one man will acquire a thing hastily,
another with difficulty; that a little learning will lead the one to
discover a great deal whereas the other, after much study and
application, no sooner learns than he forgets; or again, did you
mean that the one has a body which is a good servant of his mind,
while the body of another is a hindrance to him? Would not these
be the sort of differences which distinguish the man gifted by
nature from the one who is ungifted?” (7, 359)
15
11/20/2008
Edward Jenner. The Artist, Vol 19 1807
•“For though all men are, as we trust and believe,
capable of the divine faculty of reason, …, yet it is not
to all that the heavenly beam is disclosed in its
splendour.”
•“In the first and lowest order I place -
•the idiot: the mere vegetative being, totally
destitute of intellect.”
The Dolt
•“In the second rank I shall mention that description of
Being just lifted into intellectuality, but too weak and
imperfect to acquire judgement; who can perform
some of the minor offices of life – can shut a door –
blow a fire – express sensations of pain, etc., and,
although faintly endowed with perceptions of
comparative good, is yet too feeble to discriminate with
accuracy. A being of this degree may, with sufficient
propriety, be denominated the silly poor creature, the
dolt.”
16
11/20/2008
Mediocrity
•The third class is best described by the general term of
Mediocrity, and includes the large mass of mankind.
These crowd our streets, these line our files, these
cover our seas. It is with this class that the world is
peopled. These are they who move constantly in the
beaten path; these support the general order which
they do not direct; these uphold the tumult which they
do not stir; these echo the censure or the praise of
what they are neither capable of criticising or
admiring.”
Mental Perfection
•“… the happy union of all the faculties of the mind,
which conduce to promote present and future good; all
of the energies of genius, valour, judgement. In this
class are found men who, surveying truth in all her
loveliness, defend her from assault, and unveil her
charms to the world; who rule mankind by their
wisdom, and contemplate glory, as the Eagle fixes his
view on the Sun, undazzled by the rays that surround
it.”
17
11/20/2008
Natural selection:
The intellectual hierarchy
•Plants, woodlice, fish, reptiles, birds, mammals,
anthropoid apes, man
•“Savages”, “the lower races”, “the civilized nations”
(sic) (Darwin)
•Spearman’s “g”
•“A sort of psychophysiological energy”
•Eysenck “speed of nerve transmission”
The degeneration hypothesis
• A population is termed dysgenic if it exhibits either
enhanced reproduction levels among organisms with
seemingly undesirable characteristics, or if it exhibits
diminished reproduction among organisms with
seemingly desirable characteristics.
• The eugenics movement argued that we Humans, by
caring for the sick and ‘unfit’ are undergoing
dysgenic degeneration, and that scientists should
become involved in damage limitation.
18
11/20/2008
Racism in Eugenics
•19th century
• Charles Darwin
– Mankind continues to evolve and the races represent different stages
of evolution. Maybe need for eugenic intervention.
• Alfred Wallace
– Human evolution is complete but is now guided by a higher force
(e.g. the emergence of the emancipation of women). No need for
eugenics.
•21st century
– Herrnstein and Murray (The Bell Curve)
– The Flynn Effect
Charles Darwin (1871)
“The Descent of Man”
•“The moral sense perhaps affords the best and
highest distinction between man and the lower
animals” (p 126)
• “There is not the least inherent improbability, as it
seems to me, in virtuous tendencies being more or
less strongly inherited …. Except through the
principle of the transmission of moral tendencies
we cannot understand the differences believed to
exist in this respect between the various races of
mankind.’ (p. 124)
19
11/20/2008
Charles Darwin (1871)
“The Descent of Man”
•The chief causes of the low morality of savages, as judged by our
standards are, firstly, the confinement of sympathy to the same
tribe. Secondly, powers of reasoning insufficient to recognize the
bearing of many virtues, especially the self-regarding virtues, on
the general welfare of the tribe. Savages, for instance, fail to trace
the multiple evils consequent on a want of temperance, chastity
etc. And thirdly, weak power of self command, for this power has
not been strengthened through long-continued, perhaps
inherited, habit, instruction and religion.” (p119 of 2nd edition,
1888)
W. R. Greg,
Fraser’s Magazine (1868)
•“The careless, squalid, unaspiring Irishman, fed on
potatoes, living in a pig-sty, doting on a superstition,
multiplies like rabbits or ephemera: - the frugal,
foreseeing, self-respecting, ambitious Scot, stern in
his morality, spiritual in his faith, sagacious and
disciplined in his intelligence, passes his best years in
struggle and in celibacy, marries late, and leaves few
behind him.”
20
11/20/2008
W. R. Greg
Fraser’s Magazine (1868)
•“Given a land originally peopled by a thousand
Saxons and a thousand Celts – and in a dozen
generations five-sixth of the population would be
Celts, but five-sixth of the property, of the power, of
the intellect, would belong to the one-sixth of the
Saxons that remained. In the eternal “struggle for
existence” it would be the inferior and less favoured
race that had prevailed – and prevailed by virtue not
of its good qualities but of its faults.” (P. 353)
Terman, 1919 (Manual for the
Stanford-Binet Intelligence Test)
•Of children with high IQ on his test
•“really serious faults are not common among
them, they are nearly always socially adaptable, are
sought after as playmates and companions, they
are leaders far oftener than other children, and
not-withstanding their many really superior
qualities they are seldom vain or spoiled.”
21
11/20/2008
Terman, 1919 (Manual for the
Stanford-Binet Intelligence Test)
•“It is safe to predict that in the near future intelligence tests
will bring tens of thousands of . . . high-grade defectives under
the surveillance and protection of society. This will ultimately
result in the curtailing of the reproduction of feeble-mindedness
and in the elimination of enormous amounts of crime,
pauperism and industrial inefficiency. It is hardly necessary to
emphasize that the high-grade cases, of the type now so
frequently overlooked, are precisely the ones whose
guardianship it is most important for the state to assume.”
US Army Examination A >
•1917. The Yerkes Committee set up by US army.
First group test of intelligence (Army Alpha). The
brief:
•Adaptable to group use.
•Correlated with other measures of IQ.
•Measured a wide range of ability
•Could be scored objectively with stencils.
22
11/20/2008
Army Alpha and the S.A.T.
•In 7 working days they constructed 10 sub-tests with
enough items for 10 different forms. Piloted on fewer
than 500 subjects
•By August 1916, norms obtained on 3,129 soldiers and
372 ‘mental defective’ institutional inmates. r with school
achievement= 0.7.
•“Examination A”, subsequently “Army Alpha” with the 8
best sub-tests. Administered to millions.
•Later in 1920s, 30s the Scholastic Aptitude Test (SAT)
developed for University Entry in the US (fairly similar).
Applied Eugenics
•
•
•
•
•
•
In 1907 the USA was the first country to concertedly undertake
compulsory sterilization programs for the purpose of eugenics
The principal targets were the mentally retarded and the mentally ill,
but also targeted under many state laws were the deaf, the blind, the
epileptic and the physically deformed.
Native Americans were sterilized against their will in many states, often
without their knowledge while they were in a hospital for some other
reason (e.g. after giving birth).
Some sterilizations also took place in prisons and other penal
institutions, targeting criminality
Over 65,000 individuals were sterilized in 33 states under state
compulsory sterilization programs in the United States.
Last carried out in 1981
23
11/20/2008
Eugenics in Germany
•
•
•
•
•
•
1927 The Kaiser Wilhelm Institute of Anthropology, Human Heredity and
Eugenics, funded partly by Rockefeller
1934 saw introduction of the “Law for the Prevention of Offspring with
Hereditary Defects”
1935, law amended to allow abortion for the "hereditarily ill“ including
"social feeblemindedness“ and ‘asocial persons’.
1937, “Commission Number 3”. All local authorities in Germany were to
submit a list of all the children of African descent. All such children were
medically sterilized
1939, Euthanasia legalised for mental patients (including homosexuals)
1942, Programme extended to Gypsies and Jews
Support from German science
•Large numbers of academic geneticists and biologists joined
the Nazi Party, the highest membership rate of any professional
group
•“If it should turn out that the mere removal of natural selection
causes an increase in the number of existing mutants and the
imbalance of the race, then race-care must consider an even
more stringent elimination of the ethnically less valuable than is
done today, because it would, in this case, literally have to
replace all selection factors that operate in the natural
environment.” (Konrad Lorenz, 1940)
24
11/20/2008
References
•Flynn, J.R. (2007). What is Intelligence? :
beyond the Flynn Effect. Cambridge University
Press
•Dickins, W. & Flynn, J. Heritability Estimates
Versus Large Environmental Effects: The IQ
Paradox Resolved Psychological Review, 2001,
Vol. 108, No. 2, 346–369
25