Testing the reading and writing of
low educated ESOL learners
Jane Allemano
Oslo, Norway
May 2014
>
Purpose
Interpretation
To amuse
Appreciation/
misunderstanding
To inform
Understanding
To promote action
Support/
inconvenience
Pragmatics?
‘The study of how meaning is created in
context’
McCarthy M (1991) Discourse Analysis for language teachers Cambridge: CUP
‘Context
dependent rather than context
independent’
Levinson S C (1983) Pragmatics Cambridge: CUP
‘A branch of linguistics concerned with the use
of language in social contexts and the ways in
which people produce and comprehend
meanings through language’.
Mey J L (2001) Pragmatics: An Introduction, 2nd ed.
Wiley-Blackwell
Text 2. Can you clean the coffee machine with
soap and water?
Text 3. When can you have the Flu jab at Tesco?
Text 4. What would you eat in Istanbul?
Text 7. How much is the discount at Burger King
if I join the AA?
Reading as social practice?
Whilst some reading and writing activities
are ‘carried out as an end in themselves,
typically literacy is a means to some other
end’
Barton & Hamilton 2009:19
Exams!
What are we doing when we test L2
reading?
Testing:
Linguistic understanding
Ability to decode
Ability to synthesize information
Ability to infer
Rarely, evaluation of content
What are we doing when we test L2
writing?
Testing:
Format
Tone
Register
Coherence
Accuracy
Range
Who are the learners?
Learners who have
A1 L2 language skills but high L1 literacy in the
Roman script
A1 L2 language skills but high literacy in another
script
A1/A2/B1 L2 language skills and low L1 literacy
Fluent readers and writers in their L1
• Are aware of the universal aspects of reading
• Have developed meta linguistic awareness
• Understand how language differs in different
socio linguistic contexts
• Have developed ‘habits of mind, instilling
specific processing mechanisms’ (Koda,
2004:9)
What happens in the classroom?
•
•
•
•
•
Activation of schema
Connection with social practice
Language experience
Whole – part – whole
scaffolding
Text A
Hi,
I work in the mornings. I want to find a nursery for my daughter – a place where I can
leave her. She is three years old. I can pay £25 a week. I need a nursery near the
station. Can you help me? Fozya
Text B
• Woodlands Nursery
• 23 Woodlands Road
• 5 minutes from the
station
• For children 3 – 5 years
old
• 9.00 a.m. – 12.30 p.m.
• £22 per week
• Tel: 020 9872 1245
Text C
• Park Lane Nursery
• 14 Park Lane
• Very near the centre of
town
• Open 12.30 – 3.30 p.m.
(lunch provided)
• Children 4 – 5 years old
• £25 per week
• Phone: 020 9873 1289
In an exam, where are
• context?
• other people?
• real lives?
However, there are:
• rubrics and questions or tasks
• Reading questions that require written answers
• Distracters
• Writing tasks that have to be read
Aspects of the research on reading
The barriers presented to new readers by
• the format of examination papers
• questioning techniques on examination
papers
thus affecting the construct validity of the
assessment
• a significant part of ‘the target knowledge and
skills of the examinations (i.e literacy) are also
embedded in the very structure upon which
the examination is built and is being taken by
candidates with no previous experience of the
conventions of testing, and with rudimentary
command of the language’. (Allemano
2013:67)
The study
Participant observation and semi structured
interviews with those who
• Had minimal reading skills in any other
language
• Speaking ability of at least E2/A2
Reasons for wrong answers Test A
27%
concept of the question
language of the question
59%
14%
the reading of the text
Reasons for wrong answers Test B
27%
44%
concept of the question
language of the question
the reading of the text
29%
Reasons for wrong answers Test C
(B2 level control group)
2.5% each
concept of the question
language of the question
the reading of the text
95%
Reading issue
Where can you see this text:
Today’s sport on back page
•
•
•
in a sports centre
in a newspaper
in a book
What work does Zahia want to do? Tick YES or
NO for each.
YES
A Shop work
B A driving job
C Housework
NO
What is this text?
Ali,
Can you get the newspaper please?
I want to look at the jobs page
Thanks
Zahia
A an advert
B an email
C a note
Text: ‘My job isn’t very difficult and the pay is
good’
Question: Patience thinks her work is hard T/F
Luton Job centre
Text:
Monday – Saturday: 9.00am – 4.30pm
Sunday:
Closed
Question:
What day is the job centre not open?
Koretz’s 3 factors
• ‘Failing to measure adequately what ought to
be measured’
• ‘Measuring something that shouldn’t be
measured’.
• ‘Using a test in a manner that undermines
validity’
Koretz D(2008:220) Measuring Up. Cambridge, Mass:.: Harvard University
Press
Construct irrelevant
variance
You want to invite your friend Samira to dinner.
Write a note to Samira. In your note
• Invite Samira to dinner
• Say when the dinner is
• Say where your house is
Write about 30 words
•
•
•
•
•
•
•
•
•
•
•
My fried’s name Abdi.
…………………………………………………………………………………………………………………
He is from Somali.
…………………………………………………………………………………………………………………
He is funny man.
…………………………………………………………………………………………………………………
He is a student.
…………………………………………………………………………………………………………………
He speak English.
……………………………….…………………………………………………………………………………
He speak Somali.
What is the construct?
•Text
•Sentence
•Word
% of correct answers
Text
sentence word
Paper A
(Entry 1)
72%
86%
67%
Overall
score
74%
Paper B
(Entry 1)
58%
68%
70%
61%
Paper C
(level 1)
65%
48%
62%
(79%)
56%
% of correct answers:
Fact Deductive Inference External
knowledge
reasoning
Paper A
83% 67%
51%
75%
Paper B
66% 60%
(83%)
54% 41%
68%
55%
83%
N/A*
Paper C
Distraction at A1
Which month can Zahia start working? (There
are 3 dates on the form)
Does Zahia want a driving job? (She says she
has a driving licence)
% of correct answers
distraction
Overall score
Paper A
(Entry 1)
58%
74%
Paper B
(Entry 1)
56%
61%
Paper C (level
1)
61%
56%
‘Preliterate students are
beginning readers, but they
are not beginning problem
solvers’
Fish, Knell &Buchanan (2007)
Easy questions
Out of 14 items:
• 10 involved identification of fact
• 10 were multiple choice
• 8 were sentence level
• None contained distraction
5 combined sentence + multiple choice + fact
Language or literacy?
• For beginner readers, it is important that all
the language they encounter when reading is
familiar to them
• They need to build up a bank of sight words to
increase fluency
Two writers
1 The writer of the text being studied
2 The writer of the rubric and the questions
Two target readers
1 A hypothetical audience
2 The examiner
The test writer is an unknown phenomenon
with a poorly understood agenda :

to keep a text, sentence, word balance,

to maintain the required level of difficulty
of the examination
Is this construct irrelevant
variance?
‘It is clearly important that what tests
measure is as little contaminated as possible
by the test method and that the results of a
reading test can be generalised to a nontesting situation, as far as is possible’
Alderson 2000: 123
Ways forward
• Portfolio based testing
• Face to face questioning for reading
?
References
• Alderson, J.C. 2000 Assessing Reading. Cambridge:
CUP
• Barton, D. Hamilton, M., & Ivanic, R. (eds.) 2000.
Situated literacies: reading and writing in context.
London: Routledge.
• Fish, B., Knell, E., & Buchanan, H. 2007 Teaching
literacy to preliterate adults: The top and the bottom.
TESOL Adult Interest Section Electronic Newsletter,
August 2007. Website: http://www.tesol.org.
• Koretz, D. 2008. Measuring up. Cambridge,
Massachusetts: Harvard University Press.