When might good teaching practice become a testing problem?

When might good teaching practice
become a testing problem?
Diane Schmitt
Nottingham Trent University
Task authenticity
“Authenticity is the degree of correspondence of the
characteristics of a given language test task to the features of
a TLU task” (Bachman and Palmer, 1996:23)
Authenticity
Characteristics
of the TLU tasks
Characteristics
of the test task
Describe the target language use domain
• What are the characteristics
of the type(s) of writing
students do at university?
•
Demonstrate knowledge, understanding or
applicability of disciplinary content
•
Timed exams – 1-3 hours
•
Coursework essays – work done at
home/library
•
Essay task set by lecturer
•
Guidance on reading may be given, source
selection is up to student
•
Normally requires references to external
sources
•
2000+ words in length
•
Plagiarism heavily penalised
“Authenticity…
• Provides a means for investigating the extent to which score
interpretations generalize beyond performance on the test to
language use in the TLU domain” (Bachman and Palmer,
1996: 24).
Generalisability
“If the test tasks reflect real-life tasks in terms of important
identified conditions and operations it is easier to state what a
student can do through the medium of English…unless steps
are taken to identify and incorporate such features it would
seem imprudent to make statements about a candidate’s
ability to function in normal conditions in his/her future target
situation” (Weir, 1993:28 in Shaw and Weir, 2007: 63).
Generalisability
“If the test tasks reflect real-life tasks in terms of important
identified conditions and operations it is easier to state
what a student can do through the medium of
English…unless steps are taken to identify and incorporate
such features it would seem imprudent to make statements
about a candidate’s ability to function in normal conditions in
his/her future target situation” (Weir, 1993:28 in Shaw and
Weir, 2007: 63).
Context validity (Weir, 2005)
• Context validity takes account not just of the linguistic
content of an assessment task, but also the social and
cultural contexts in which the task is performed.
• purpose
• content knowledge
• setting
• linguistic demands
• time available
• writer/reader relationship
• text length (input/output)
• marking criteria
Pre-sessional students’ writing experience
What’s the longest piece of writing you have ever done in English?
IELTS essays (250-300 words) x 11
TOEFL essays (250-300 words) x 2
1000 word final year dissertation x 1
1000 word essay for a friend x 1
Have you ever had to use sources in your writing before?
Yes x 2
No x 13
The teaching task
At the end of this course, students should be able to…
Write increasingly longer essays – 1500-2000 words
Select, read, understand and synthesize source material
Integrate relevant information and ideas into essays
Develop a coherent organisational structure and stick to it
Correctly use in-text citation
Correctly format a bibliography
Ensure that language use effectively conveys ideas and does not obscure meaning
Effectively proofread and self-edit
A recent thread on the BALEAP listserve
•
At university, language skills are used in an integrated way, e.g.
students listen to lectures and take notes, discuss and refine their
ideas with peers, increase their understanding through reading, notetaking and summarizing and provide evidence of their learning through
written and oral work. However, on many EAP courses, these
language skills are timetabled and taught separately, possibly to
facilitate different teachers sharing classes.
•
To what extent do we risk under-preparing students for their
university experience if we separate out the skills?
•
Olwyn
A common approach to writing
• An set of integrated classes which culminate in students having to
write a long-source based essay (and possibly a lead a seminar and/or
give a presentation). These aim to simulate the university
environment and “scaffold” the task of writing an academic essay.
•
Cazden (1983) defined a scaffold as “a temporary framework for construction in
progress” (p. 6).
•
In Vygotsky’s words, “what the [learner] is able to do in collaboration today he
will be able to do independently tomorrow” (Vygotsky, 1987, p. 211).
•
(I confess, I got these last two quotes off Wikipedia!)
The pre-sessional at NTU
• 21 contact hours per week
• 9 hours of integrated skills – leading towards writing one
source-based essay on a shared topic
• Assessment
• 50% coursework essay
• 25% coursework presentation
• 25% listening/reading exam
Blurring the boundary between teaching and
assessment
Teaching to the test?
Source texts are the basis for reading instruction so substantial
comprehension work is done
Virtually all writing work is related to this one assignment
Writing process is staged with students getting both written and tutorials
feedback along the way
Is the picture any clearer elsewhere?
• 44 BALEAP members responded to a survey
•
32 use a coursework essay in their final assessment for progression
•
The essays’ weightings range from 17.5 – 80% of the final mark
•
12/17 of these use the same essay to teach essay writing as they do to assess
essay writing
•
Students commonly write only 1 source-based essay during their course
(Course lengths range from 4 – 10 weeks)
•
The amount of support provided is considerable although it varies from
programme to programme
“The essay is given a lot of weight because the vast majority of our students will have
•
to write long, take home essays on their future programmes.”
Painting by numbers
Questions?