TESOL Centre
Assignment Cover Sheet
Name: Mat Plews
Course: Specialist Research Areas in TESOL (A-2009/0)
Date: 12th March, 2010
Title of Report/Assignment:
A review of current best-practice
in EAP and how its effectiveness
can be measured.
ABSTRACT
Few would dispute the fact that the field of English for Academic Purposes (EAP),
one of the largest branches of TESOL, is at the forefront of research-based teaching.
Consequently, there is a vast amount of literature available, including a great deal of
research publications, covering a multitude of issues. This review seeks to inform
practitioners about these issues and to promote further research and debate both in
and out of the classroom. To achieve this aim, I commence by defining EAP itself,
before presenting the latest developments regarded as best-practice in the areas of
teaching and learning. In order to seek to justify current best-practice, the following
section examines possible methods for measuring its effectiveness, and presents
information on a number of research projects with this goal. This process leads me to
believe that quantitative data is required if effectiveness is to be accurately
measured. In order to obtain such data, I propose the use of learning-based microevaluations in which detailed analyses of classroom interactions are conducted, with
the aim of further improving EAP best-practice.
1
ABSTRACT
1
TABLE OF CONTENTS
2
1.
INTRODUCTION
3
2.
CURRENT BEST-PRACTICE IN EAP
2.1. Introduction
5
2.2. Review of literature on EAP best-practice
3.
2.2.1. What is EAP?
6
2.2.2. Current best-practice in EAP
8
2.2.3. Language course specifics
11
2.3. How to apply EAP best-practice
15
2.4. Concluding remarks on best-practice
16
MEASURING THE EFFECTIVENESS OF EAP PROGRAMMES
3.1. Introduction
17
3.2. Review of literature on measuring the effectiveness
of EAP programmes
3.2.1. What is effectiveness ?
18
3.2.2. What can be measured ?
19
3.2.3. Attempts to measure the effectiveness
of EAP programmes
20
3.3. A way forward.
22
3.4. Concluding remarks on measuring effectiveness.
23
4.
CONCLUSION
24
5.
FUTURE RESEARCH POSSIBILITIES
25
GLOSSARY OF KEY TERMS
26
REFERENCES AND BIBLIOGRAPHY
27
2
1.
INTRODUCTION
The field of English for Academic Purposes (EAP) 1 has existed since at least
1974, when Tim Johns first coined the term at a meeting, although work had officially
been going on in this area from the beginning of the 1960s at Birmingham University
in the UK (Jordan, 2002 :70).
What started as an advice service for a small number of overseas students
has since expanded into the global industry that we have today, with millions of
students all over the globe learning English for use in academic contexts.
A vast amount of research on EAP has been conducted since those early
days, in areas ranging from methodologies and materials, through genre analysis, to
feedback and assessment (see Hyland, 2006; Hyland and Hamp-Lyons, 2001;
Flowerdew and Peacock, eds, 2001 for research overviews). However, although
there have been a number of studies which have attempted to measure the
effectiveness of parts of this industry (eg. Clifton, 2004; Gillet and Wray, eds, 2006;
Elder and O’ Loughlin, 2003; Storch and Tapper, 2009) there is still much work to be
done in this area. Despite the fact that there is a general indication from studies such
as these that current practice in EAP is working, there is not much information
available as to exactly what it is that makes students successful and why this is the
case.
In this literature report, it is hoped that the reader will gain a detailed overview
of what it is that appears to be working as well as an insight into methods which can
be used to pinpoint the reasons for this success.
As the vast majority of EAP research has been conducted in countries such as
the UK, the USA, Australia and Hong Kong, these geographical areas have provided
most of the sources for this report. Although I have endeavoured to include the
voices of researchers from other areas, there is inevitably a bias in terms of the
amount of information coming from the aforementioned countries, where ‘Classic
EAP’ (Goh, 1998: 21) has been in existence for a substantial period of time and
where the high number of students studying EAP to cope with degree courses has
led to much more research being done than in other parts of the world.
This is certainly the case in terms of studies concerned with the effectiveness
of EAP programmes, the bulk of both quantitative and qualitative research in this
area having been carried out in Anglophone countries.
However, despite this restriction, it is clear that EAP practitioners in other
geographical areas, whether they are in Chile, India, Germany (my current teaching
situation) or elsewhere can learn many lessons from the studies conducted by those
in countries at the “forefront” of EAP research.
The knowledge gained from familiarisation with current best-practice in EAP
can only be beneficial to the industry and TESOL as a whole and lead to better
teaching, studying and researching in English worldwide. The status quo of bestpractice is, nevertheless only a step on the path to improving the global ability of EAP
1
Although the term EAP, in its broadest sense, can be applied to any educational situation from primary school
onwards, for the purposes of this report the scope of the term is restricted to the area of higher education.
3
practitioners, and it is for this very reason that the second part of this report is
concerned with seeking to identify the best ways of measuring its effectiveness, in
order for us to identify current strengths and weaknesses so that we can use this
information to create improved best-practice for the future. This report seeks to
inform and promote further research both in and out of the classroom.
As is undoubtedly true for other stakeholders of the EAP community, whether
they be fellow teachers or researchers, students, subject tutors, university deans or
others, I firmly believe that we need to constantly strive to improve standards and,
given that English is currently by far the dominant language of global academia,
thereby enhance the standards of communication within the global academic
discourse community, as this is likely to be where future research developments are
made in all areas.
4
2.
CURRENT BEST-PRACTICE IN EAP
2.1. Introduction
Before beginning an examination of current best-practice in the area of English
for Academic Purposes, it is necessary to define what is meant by ‘best-practice’.
According to the Macmillan Dictionary (Macmillan Education, 2007), the term is
defined simply as “the best, most effective way to do something”. What we are
concerned with in this section, therefore, is what, according to research, are generally
considered to be the techniques and strategies that can best be employed in the area
of English for Academic purposes in order to achieve the aims of improving the use
of English language amongst the academic community.
Given the vast amount of research which has already been conducted in the
field of EAP and the fact that this is only to be a relatively short report I can, however,
only attempt to provide an overview of the main trends, which I nevertheless hope will
prompt readers to carry out further investigations themselves.
Therefore, this section is restricted to providing an outline of the current
situation with regard to the teaching and learning of EAP in general. Nevertheless,
the information presented should certainly be useful to those who are developing
courses, those teaching on them, those who aim to develop student autonomy and
those who are involved in student assessment.
The importance of providing this overview of best-practice in EAP in order to
further enhance activities in this area can only be re-iterated.
5
2.2. Review of literature on EAP best-practice
2.2.1. What is EAP?
As mentioned in the Introduction to this report (p.3), the term ‘English for
Academic Purposes’ has been in usage since the mid-1970s. During that time it has
undergone a change from meaning merely an advice service for foreign students to
encompassing a worldwide field of expertise. The term itself has also been the
subject of criticism, with a call (Pennycook, 1997: 257) for the word ‘for’ to be
removed as it infers that English is a neutral language, which it clearly is not (see
Swales, 1997; Benesch, 2001; Flowerdew and Peacock, 2001; Pennycook, 1994;
Kennedy, 2001). Perhaps, at some point, the term will become ‘English with
Academic Purposes’.
Alexander et al (2008: 2-4) define the “highly specific” term ‘EAP’ as
possessing various distinct characteristics. These include:
i. It is goal-driven, time-bound, and concerned with the academic community.
ii. Its teachers are often graduates in specific evidence-based disciplines (e.g.
science, social science, business studies).
iii. Teachers and students are generally perceived to be relatively equal.
iv. Grammar and vocabulary are restricted to academic discourse.
v. The main focus is on reading and writing.
vi. Materials are often taken from degree courses, meaning that they are often
long and lexically dense (Hyland, 2006) and that clarity and objectivity are
valued.
vii. Texts are usually from academic genres, linked by a theme, and are fullyexploited by teachers.
viii. Study skills, learner-independence and critical-thinking are stressed.
To this list, we can add the following (Gillet and Wray, 2006: 3-7):
ix. Cultural awareness and knowledge of the target culture are imparted.
x. EAP courses are increasingly embedded within degree courses.
xi. The majority of EAP teaching is task-based.
Hyland (2006) mentions these characteristics:
xii. Teachers are committed to “research-based language education”
xiii. EAP is becoming more discipline-specific.
xiv. It will become more important at the pre-tertiary level.
The QAA language benchmarking statement (Quality Assurance Agency for Higher
Education, 2007) also mentions:
xv. Explicit knowledge of language is a typical need.
Hyland and Hamp-Lyons (2002) note that:
xvi. Most EAP teachers are non-native speakers.
6
As early as 1998, Goh described three different “environments” of EAP:
i. ‘Classic EAP’, where non-Anglophones go to Anglophone countries such as
the UK, the USA and Australia. This is by far the most common situation,
from which much of the research mentioned in this report is derived.
ii. ‘Domestic EAP’, in which courses in non-Anglophone countries are taught in
English e.g. Master’s courses in Germany are sometimes conducted in
English.
iii. ‘New EAP’, where non-Anglophones go to other non-Anglophone countries
e.g. Singapore, Hong Kong.
It is clear from the above just how multi-faceted EAP is, and how difficult it
therefore is to create a clear definition of the term.
What is, however, clear is that EAP students have very specific needs,
meaning that needs analyses and feedback are essential parts of any course of
study. This brings us to the next section, in which techniques for teaching and
learning are discussed, based on the above characteristics.
7
2.2.2. Current best-practice in EAP
This section is concerned with the current situation in EAP, seeking to present
and justify the latest developments regarded as best-practice in the areas of
teaching and learning, with the aim of informing practitioners and hopefully promoting
reflection on their own practice.
Before we look at individual areas of teaching and learning, there are a
number of points which are worth keeping in mind with regard to EAP in general.
First, EAP courses are primarily language courses. Although it is clearly of
great importance (see the points listed in 2.2.1) to gear any course to the needs of
the students and the academic discourse community in which they will act, we must
not forget that the students’ main aim in taking EAP courses is language
improvement.
Second, we must not be too prescriptive in our teaching. Although the core
elements of language required are the same for all students (Hyland, 2006: 13;
Swales and Feak, 2000; Swales and Feak, 2004) it is clearly not a case of “one size
fits all”. Teaching needs to be geared as far as possible to the courses of study that
students will pursue, and we must also make use of the skills and knowledge that
students bring with them from their own cultures. One example (Biggs, 2003) is the
rote-learning ability of Chinese students, which many western teachers fail to exploit.
Another (Hitchcock, 2007) would be the great willingness of some students to
contribute to online VLE discussions compared with their reticence to speak in class.
There is also a place for students’ opinions on what and how they should learn and
be helped to learn, even to the extent of challenging the “powers that be” of
academia (see e.g. Benesch, 2001). There appears to be the problem that many
institutions adopt an ‘accommodationist’ stance with regard to their systems, forcing
students to accept their hierarchies and methods.
Some of the challenges faced by EAP teachers are summed up by Alexander
et al (2008: 19-21), and we will address these in turn. Firstly, there is a need for
teachers to become familiar with their students’ subjects. Given training and
educational opportunities for teachers, this situation could be largely resolved. The
problem seems to be more one of constraints on time and funding.
A further challenge mentioned is the need for teachers to use authentic,
subject-specific materials which are interesting and motivating for students. There are
various ways forward here and there has been a move towards Content Based
Instruction (CBI) in recent years, with courses combining input on both language and
subject content. This either takes the form of using materials similar to those on
degree courses in class (known as ‘sheltered CBI’) or language classes following up
directly on degree classes (‘adjunct CBI’). Both of these strategies, and especially the
latter, clearly require a high degree of cooperation between language teachers and
subject tutors. However, it may well be the case that there are not enough students
from each subject area to form these distinct classes and, given financial limitations
on the number of teachers that can be employed and the number of classrooms
available, these students may well be placed together with those of other disciplines
in an EGAP (‘English for General Academic Purposes) class. Still, let us assume that
there is the possibility of forming at least a few subject-specific classes.
8
There is a further qualm which teachers may have with the above suggestion,
however, which is that authentic, subject-specific materials may well not be the best
ones for promoting interest and motivation. We have probably all experienced
situations where students complain about having to do the “same boring old stuff”
again and request something “interesting” e.g. something current and from the news.
There are even cases of e.g. computer students who are fed up with using computers
all day and who do not wish to use them in a language-learning classroom.
Personally, I sympathise with these viewpoints and have nothing against the use of
the odd journalistic text for motivational purposes, as long as the students are
involved in authentic tasks.
If we manage to organise such courses and have the correct kind of set-up
and materials, there is still the problem of giving students the techniques necessary
to cope with them. Some suggestions as to how this might be done are offered in
Section 2.2.3 – language course specifics (p.11).
Alongside the needs for teachers to focus on the right kind of language, avoid
being too prescriptive, become familiar with our students’ subjects, and employ the
right kind of materials in the classroom, there are a couple of other “great debates”
currently taking place in the EAP world which affect what is done on language
courses. The first one concerns the role of English as an international language and
the second one is about the place of study skills. Both of these issues are
summarised below.
With regard to the role of English, now that there are more non-native
speakers of English than native-speakers (Firth, in Hyland, 2006: 29), and the
majority of English conversations are conducted between non-natives, the question
has arisen as to what kind of English we should be teaching. If standard varieties
such as British and American English are not being used by most speakers of the
language, then we should perhaps be teaching something else, perhaps a simplified
version of the language, containing the core elements as used by the largest
proportion of English language users. This question is critical with regard to the
international academic discourse community, where the proportion of non-native
speakers of English is growing daily. And it is not only the use of the variety of
language which is being questioned, but the discourse practices of academics from
different countries and cultures (see Goh, 1998: 28). It seems that we need to
address the “west-centric” view of many institutions. This has obvious consequences
for materials development.
Summing up the next issue, the teaching of English for Academic Purposes is
often divided into the “skills” of reading, writing, listening, speaking, and other ‘study
skills’. However, this tends to ignore the need for language for academic purposes
(Alexander et al, 2008), meaning that linguistic expression is often sacrificed for the
sake of mere communicability (Turner, 2004). In her article ‘Language as Academic
Purpose’, Turner traces this back to the fact that EAP is an industry that has always
been concerned with making as much money in as short a time as possible, thereby
only requiring students to achieve minimally (: 97). This probably led to the
introduction of non-discipline-specific, ‘high-stakes’ tests such as TOEFL and IELTS
and the increased proliferation of the ‘communicative approach’, allowing students
with the minimum levels necessary to gain entrance to courses of study quickly and
9
relatively cheaply, whilst making the tasks of university registrars easier.
Consequently, many EAP courses seem to be focused on preparing students to pass
tests such as these, rather than on the real language abilities necessary to do well on
their chosen subject courses. This approach reinforces the idea that skills can be
taught in isolation from the real work required on a degree course.
Thankfully, there seems to now be a move away from the teaching of
individual skills and an increased focus on ‘academic literacies’ (see Hyland, 2006:
21-23), where language is placed in the context that students experience it. Much of
what is defined as ‘best-practice’ in this report draws on this idea.
10
2.2.3. Language course specifics
This section looks at some particular techniques that appear to be bestpractice, which can be applied by EAP teachers.
i. Finding motivating, authentic subject-specific materials
According to Alexander et al (2008), there are five key criteria which need to be met
in order to find suitable ‘texts’ (defined in Johnstone, 2008: 20, as “instances of
discourse” i.e. written, spoken or otherwise) for use on an EAP course. They are:
i. The register must be academic. Examples of ‘academic register’ would be
formal style and linking, specific vocabulary, use of citations and definitions,
and hedging of ideas.
ii. Any ‘text’ should be of the correct ‘genre’ (Johnstone: 182 defines genres as
“… the categories of text which a person has to learn to recognize, reproduce,
and manipulate in order to become a competent member of a particular
community”). In other words, ‘texts’ used need to be suitable for the correct
audience and purpose, and to be organised for this. Examples of academic
genres are lectures and research reports.
iii. Suitable examples of functions should be present in the ‘text’. These might
include cause and effect, comparing, defining, and classifying.
iv. There has to be suitable cohesion in any ‘text’, linking the various parts and
ideas together, and it is important that information flows throughout the ‘text’.
Examples could be ‘signposting’ in presentations or conjunctions in an essay,
amongst others (see Alexander et al, 2008: 57-74).
v. Appropriate organisation has to be in evidence. Academic ‘texts’ are often
organised deductively i.e. going from the more general to the more specific. A
few examples of this would be debates and essays, which both usually begin
with some kind of introduction.
If these criteria are met, then the teacher can be confident that their materials are
suitable for use.
ii. Designing an EAP course
As Hamp-Lyons (2001) stated, “[EAP] begins with the learner and the
situation”. Nothing is more fundamental to an EAP course than conducting a
thorough needs analysis at (or before) the beginning of a course. If we do not know
who the stakeholders in the course are, what they want and what they need, then we
cannot hope to provide them with a suitable course. It is also necessary to procure
constant feedback during and after the course, so as to be able to make any changes
that are required.
Once the participants’ needs have been determined, it is important to define
the course objectives and also exactly what the students will need to be capable of
doing by the end of the course. A series of ‘learning outcomes’ need to be recorded,
11
so that everyone knows what they are working towards. These can, for example, be
written as ‘can do statements’ e.g. “The student can take appropriate notes in an
academic lecture”. These are helpful for keeping the course on track, making
students more aware of their learning, and for assessment purposes (see Hughes,
2003:20-22 for further information on ‘criterion-referenced testing’).
The next step, bearing in mind possible constraints, is designing a course
syllabus. It is suggested (Alexander et al, 2008: 86-87) that key abilities (e.g. learner
autonomy, critical learning) should be integrated throughout the course and that the
main topic areas should form separate units. However, each unit should build, to
some extent, on knowledge gained from the previous ones, and there should be a
certain amount of explicit recycling.
The methodologies to be used then need to be determined. Two common
possibilities are ‘text-based’ and ‘task-based’ methodologies, the former using
models from appropriate genres, which students analyse before creating their own
‘texts’, and the latter involving students in attempting to accomplish some kind of
realistic task, analysing how it went as compared to a model, and then possibly
attempting another similar task which is, in turn, analysed. There are many advocates
of using task-based learning on EAP courses e.g. Richards and Rogers (2001); Feez
(2002); Robinson, Strong, Whittle and Nobe (2001); Willis (1996); Van den Branden
(2006); Edwards and Willis (2005); Skehan (1998) (in Alexander et al, 2008) and I
would have to say, based upon my own experience of using task-based learning, that
I am with them.
In his analysis of reports from EAP teaching practice, Todd (2003) noted that
there appear to be six key approaches which are commonly taken. These are:
i. inductive learning (in which students learn through problem-solving. This sits
well with a task-based methodology).
ii. process syllabuses (“learning by doing”, essentially task-based and projectbased syllabuses).
iii. learner autonomy (discussed on page 13).
iv. authenticity (see the reference to CBI on page 8).
v. integration of technology (e.g. VLEs, CD-ROMS, concordancing software).
vi. team-teaching (co-operation with subject tutors).
With reference to the fifth point, I would add that, although there are clearly
many advantages to using ICT (Information and Communications Technology),
teachers must always make sure that there is a clear rationale for its use, and not
use it merely because it is new and everybody else is using it!
As regards team-teaching, although this is desirable, it is unfortunately not
always possible to set it up. However, collaborative learning and teaching in general
is to be encouraged as it is a natural way to promote learning (see Vygotsky: 78).
This explains why group-work is so common on modern courses.
12
iii. Facilitating ‘Critical thinking’
The term “critical thinking” seems to be a buzzword these days. The clearest
definition I have come across is that of Alexander et al (2008: 269), which is “taking a
stance; evaluating by means of criteria; seeing new relationships between ideas.”
Anything which makes students think or question is a valuable source for promoting
critical thinking. This can be done by simply asking students to re-tell something in
another way e.g. giving a presentation on a research paper they have written, or by
giving students lists of questions which make them reflect on their and others’ actions
and ideas. In my opinion, the use of interesting and motivating journalistic ‘texts’ has
a place in developing critical thinking skills, as does task-based learning.
Critical thinking skills are demanded in western academia and, for study in this
sphere, need to be taught. However, we must be careful to value other skills that
foreign students bring with them from their own cultures (such as the ability to learn
by rote) just as highly, and not see our view of ‘critical thinking’ as the only one.
iv. Learner autonomy
As noted on page 12, learner autonomy is very much the norm in western
universities. It is clearly a valuable skill to possess and one which empowers students
to find and meet their own needs (Penaflorida, 2002). This skill encompasses many
sub-skills, ranging from the ability to draw up a timetable and keep to it, to
proofreading one’s own work. To help students gain independence from the teacher,
tasks such as peer review and correction should be encouraged. Eventually, once in
possession of the array of sub-skills required, students should be able to work
completely independently, if necessary. Thankfully, however, I believe that there is
still a role for the teacher as facilitator and consultant, and we will not go as far as
Field (2007) believes when he states, “… the most effective teacher is one who
provides for her/his own redundancy.”
v. Assessment
According to Hughes (2003), any useful assessment method (whether a test
or continuous assessment) needs to be valid (it should measure what it is supposed
to), reliable (consistent in its measurements), and have a beneficial effect on teaching
(known as ‘positive backwash’). To this list we can add two further criteria,
particularly relevant to EAP: assessment items should be authentic, and practical to
develop and deliver (Alexander et al, 2008: 315).
As mentioned in section ii (above), it is now relatively common practice to
define course objectives and learning outcomes as a series of statements, so that we
know exactly what is covered on a course. Any good assessment method must test
according to these objectives and outcomes – it would clearly be unfair to test
students on areas that they have not been taught! Students can also be involved in
shaping objectives, outcomes and assessment methods at the needs analysis stage.
It has already been remarked upon (p.9) that tests for university-entry
purposes such as TOEFL or IELTS fail to address the discipline-specificity and real
academic needs of students. Instead, we must design tests wholly related to our
13
students’ needs if we are to develop the skills that they will need for academic study
in English.
The way forward seems to be the use of continuous assessment methods,
which can be designed to meet all our criteria in that they can provide true
measurements of authentic tasks, are based on learning outcomes to ensure
reliability, and thus provide positive motivation for students. They are also relatively
easy to set up, and this approach fits in well with the use of task-based learning.
14
2.3. How to apply EAP best-practice
In this section on current best-practice in EAP, we have tried to define EAP,
identifying its key features and examining the shape and content of current EAP
courses. The key findings are summarised below:
Firstly, although due to limited numbers of students, teachers or classrooms,
there are many EGAP classes, the ideal situation would be to have discipline-specific
classes so that students can work more easily towards their goals, if possible, in
liaison with subject tutors. Ideally, the language teacher should also be a subjectspecialist. Materials should be subject-specific, authentic, interesting and motivating.
Next, we must make sure that we are teaching the right kind of language i.e.
language ‘for (or with) academic purposes’, and not simply discrete skills. As well as
helping students to learn to understand and use discipline-specific language, EAP
courses should aid students in integrating into the cultures that they will be studying
in, at the same time as developing their critical thinking and learner autonomy.
Especially as non-native speakers of English are now in the majority, we must
be careful not to adopt a ‘west-centric’ view, and should instead recognise the
benefits that teachers and students from non-western cultures bring to the classroom
in terms of their ways of thinking, their approaches, use of English and discourse
practices. We need to listen to the voices of our students and accept and appreciate
their involvement in course design.
All courses should begin with a needs analysis, and feedback should be
sought throughout. Following the needs analysis, course objectives, learning
outcomes and the types of assessment to be used need to be specified (in liaison
with students). These should be adhered to throughout the course, only being
modified if feedback or evidence demands it. Some form of continuous assessment
seems to present the best possibilities for measuring student performance.
A suggested course structure would be to have separate topic units with key
abilities integrated throughout, progressing in a developmental manner. A
collaborative, task-based approach seems to be the most suitable for EAP courses,
which should also contain a certain amount of recycling.
Clearly, the suggestions above depict an ideal situation, and it is unlikely that
anyone has managed to implement all of the measures proposed. I myself am no
subject specialist and do not have the opportunity to teach fully discipline-specific
classes, and my materials are certainly not always authentic, interesting and
motivating! Although needs analyses and feedback are an integral part of my
courses, I have yet to develop specific learning outcomes. This is an area in which
researching for this report has prompted me into action.
The ideas I have presented in this section are certainly approaches that we
can aim to put into practice as far as is possible. Hopefully, this section has inspired
readers to at least try a few of them out, or to possibly conduct their own research
and come up with better ideas of ‘best-practice’.
15
2.4. Concluding remarks on best-practice
Reviewing the literature on EAP best-practice is not an easy task, given the
sheer volume of research that has been conducted and work that has been
published. As stated, my aim in this report has been to provide an overview of the
main trends as regards best-practice in EAP.
Being aware of what appears to be currently working in the EAP world is only
a small step in the right direction, however. What is ideally required is concrete proof
that the suggestions given really do work and an indication of exactly why they work.
If we possess this knowledge, then the task of designing more appropriate forms of
teaching and learning for the future will be just that little bit easier. In the next part of
this report, accounts of attempts to generate this knowledge are described; in other
words, attempts to answer the question as to how the effectiveness of EAP can be
measured and explained.
16
3.
MEASURING THE EFFECTIVENESS OF EAP PROGRAMMES
3.1. Introduction
The Macmillan Dictionary (Macmillan Education, 2007) defines ‘to measure’ as
“to find the exact size, amount, speed etc of something using a special tool or special
equipment“, and ‘effective’ as an adjective meaning “something that works well and
produces the result that was intended”. ‘Program’ (US English, British English
‘Programme’), in the educational sense, is defined by Lynch (1996) as “any
instructional sequence” e.g. a curriculum, a workshop, a teaching unit or even selfaccess instructional software.
Therefore, in attempting to measure the effectiveness of EAP programmes, we
are seeking to quantify to what extent an instructional sequence works well and
produces the intended result, using special tools.
This definition of our aim is rather revealing. First of all, we are seeking to
quantify i.e. to generate some kind of concrete results, perhaps in the form of figures.
This implies some kind of quantitative analysis, as opposed to a qualitative one. This
differs from the first part of this report, in which judgements were made on the quality
of EAP practice, in terms of what is said to be ‘best-practice’. If we want to find out
what definitely works well, then we have to prove it, not merely assemble
judgements. This presents a number of difficulties, as we will see later, but is not
impossible.
The second point to note is the varied nature of a ‘programme’ in this context;
in other words, we can carry out our analysis on activities ranging from a classroom
sequence to a student’s entire learning experience. The studies we will look at in
section 3.2 are almost exclusively conducted on short (one to three-month) language
courses.
Finally, it should be noted that ‘effective’ is a highly subjective term. This
means that we have to be extremely careful to define what we mean by “works well”
and “the intended result”. Therefore, we have to be able to justify our measurement
tools as producing a clear indication of effectiveness, grounding them in previous,
ideally quantitative research. Further, the separate parts of the definition focus on
different aspects: one is concerned with the process itself (or, rather, parts of the
process) and the other with the end result. It is necessary to focus on both of these
elements if we are to give any kind of indication as to the success or otherwise of an
EAP programme.
If we succeed in measuring the effectiveness of EAP programmes, and our
choice of measures is justified, then we will have gone a long way towards answering
the questions of what works and why. This is clearly of great importance to EAP
professionals, enabling them to employ methods and approaches that are fully
proven and beyond doubt useful in the teaching and learning of English for Academic
Purposes.
17
3.2. Review of literature on measuring the effectiveness of EAP programmes
In this section, we will look at what has been written and recorded concerning
the quest to measure effectiveness in EAP programmes, examining alternative views
on ‘effectiveness’ and what can and should be measured, as well as accounts of a
number of studies conducted in this area. The aim, as stated, is to inform practice
and to promote reflection and further research in this area.
3.2.1. What is effectiveness ?
In section 3.1, I proposed that, for our purposes, anything which is ‘effective’
has to be quantifiable and cover the whole instructional sequence from the individual
parts to the final result.
Lynch (1996) does not directly discuss a definition of the term ‘effectiveness’,
leaving it up to each individual to decide what it means to them. His influential book,
‘Language Program Evaluation’ instead provides a heuristic to help researchers carry
out their own analyses. However, he does not dismiss the usefulness of qualitative
(‘naturalistic’) methods and actively encourages a move away from purely
quantitative studies which rely on some kind of test prior to and post the instructional
sequence (eg. Storch and Tapper, 2009). I am in full agreement that we must look at
the whole process, as is clear from the above paragraph and I do believe that there is
great usefulness in conducting qualitative studies alongside quantitative ones but, if
we are really to prove that our teaching methods are working, we must have robust,
solid evidence, which naturalistic methods can never fully provide. Lynch rightly
criticises the fact that too many studies have been conducted merely based on
comparing results at the beginning and end of a course, but I would argue that too
many studies (see section 3.2.3) have been conducted based primarily on qualitative,
subjective data. Lynch (2001) himself seems to recognise this when he calls for more
research on how we can improve qualitative data.
Hence, for the purposes of my own research in this area, I propose to rely
mostly on quantitative data, using naturalistic data only as background information.
18
3.2.2. What can be measured ?
Clearly, a reason for the common use of qualitative data in EAP is the fact that
it often appears impossible to gather quantitative information in an area full of
opinions about what is best-practice and what is not (the whole of section 2 is an
example of this). How can someone’s views be measured? A second reason is that,
until we know exactly how the human brain functions and how language really is
acquired and learned, it will be almost impossible to connect any item of new
knowledge to its source.
However, there are still a number of entities that are quantifiable, including
attendance, assessment and test results in language and subject courses, the
frequency of certain actions in the classroom (Ellis, 1997) and even oral and written
discourse (see Johnstone, 2008; Morley, 2006).
In my opinion, we need to collect as much quantifiable data as we can and
detailed information on e.g. classroom activities and exchanges, and I would propose
for this purpose some form of action research, as advocated by Ellis (1997: 169),
probably in the form of micro-evaluations (ibid.: 201-203). If enough small-scale
research based on quantitative data is conducted, then we will have a much clearer
idea of the bigger picture.
19
3.2.3. Attempts to measure the effectiveness of EAP programmes
In this section, we will look at various studies conducted with the aim of finding
out if language programmes worked or not, analysing their successes and failings.
In 2006, BALEAP (British Association of Lecturers in English for Academic
Purposes) published a series of papers initially presented at a Professional Issues
Meeting, entitled ‘Assessing 2 the Effectiveness of EAP Programmes’ (Gillet and
Wray, eds, 2006). This edited collection contains six detailed studies, which almost
exclusively focus on pre-sessional courses at British universities, and are briefly
summarised below.
In the first paper, Barbara Atherton (2006) claims that to assess a course the
criteria for success have to be based on the needs and expectations of the
stakeholders (in this case the students, the language and subject tutors, course
directors and university authorities) and for this reason she compiled data from a
variety of sources, in the form of entry and exit test scores, questionnaires, in-course
comments and non-attendees’ comments. Although this study seems to have been
very inclusive and relatively successful, it fails to provide any evidence about which
aspects of the task-based course contributed to the students’ success.
The next piece of research, conducted at Sheffield Hallam University between
January 2000 and December 2002 by Diana Ridley (2006), posed the question “Is
there a relationship between Pre-sessional exit test results and subsequent academic
course completion?”, in other words, what level of proficiency is required for students
to study successfully? This question was addressed by tracking a cohort of students
for two years from the start of their Pre-sessionals and comparing their exit scores
with subsequent performance in their disciplines. Again, despite the fact that the
study confirmed the value of the pre-sessional course and university support in
general, there was no information about exactly what happened on the pre-sessional
courses that helped students to become successful.
Mary Martala’s (2006) contribution concerns the tracking of students’ writing
abilities on a pre-sessional course. The participants were asked to complete five
questionnaires – one at the beginning of the course, three at the end, and one when
they had completed their Master’s degrees. Samples of students’ writing from the
Pre-sessional and from their degree course were also compared. Once more, the
pre-sessional courses are described as successful, but we will never know exactly
why.
Nick Pilcher’s (2006) study of students’ perceptions of the effectiveness of
their EAP course is also rather subjective, as it is based purely on qualitative data.
Although it makes interesting reading and showed a favourable view of the EAP
courses from the students, it provides us with no concrete information about what
worked.
2
Note the use of the word ‘assessing’ in the title, which differs from the word I have chosen,
‘measuring’, in that ‘assessments’ are based on both qualitative and quantitative data in these studies.
This differs from Lynch’s (1996) definition of ‘assessment’ as essentially quantitative, a part of
‘evaluation’ which does not include e.g. questionnaires, unstructured interviews and observation.
20
The last two papers are theoretical in nature and contain some further
interesting ideas.
John Morley (2006) discusses possibilities for measuring students’ oral
performance in order to compare their performance at the beginning of a course with
that at the end. He proposes methods to obtain quantitative measures of spoken
fluency, complexity and accuracy. Although he recognises the limitations of his
proposals and the difficulties of carrying out such a study, he concludes that “it
should be possible, using some of the measures described here, to identify mean
gains in certain core aspects of oral proficiency over a 10 or 12 week period of
intensive EAP study” (ibid.: 83). This seems to be a very good starting point for
further, classroom-based research of the type I described earlier, although we have
to be certain that our measurements are valid. There would also be the need to link
any gains in oral proficiency to the teaching methods used on the course.
The final paper in this collection is by Simon Kinzley (2006), who proposes the
application of ‘innovation theory’ (see ibid. for details) to the study of the
effectiveness of EAP writing programmes. A combination of student interviews, tutor
and lecturer feedback, and analysis of all the essays written during the period of the
study is designed to provide enough data to make an accurate evaluation of a
programme’s effectiveness. There are, as yet, no published results of the research,
but I am sure that they will make interesting reading. I still have reservations,
however, about the use of interviews and feedback as accurate measures of
effectiveness.
A further study on effectiveness is that by Rebecca Clifton (2004), in which
she conducted an evaluation of a one-year EAP programme in China designed to
prepare participants for study in the UK. Although information from student
questionnaires and interviews with tutors seems to have been useful in gauging the
effectiveness of the programme, it is still highly subjective. An example of the
subjectivity of obtaining data through student reporting occurs in this study when
students and third-parties mis-report their own test scores. If accurate data cannot be
obtained (eg. due to data-protection regulations or unwillingness on the part of
university authorities) then I would argue that data has to be collected in a different
way, to ensure that it is quantitative and accurate. Of course, in long-term studies,
this would necessitate data-collection from the very beginning of the initial courses.
A more recent study is that by Storch and Tapper (2009), entitled ‘The impact
of an EAP course on postgraduate writing’. Students were asked to write an essay at
the beginning and end of a pre-sessional course and these were compared. They
were also asked to complete a short questionnaire. Although attempts were made to
link the knowledge gained through the course to the teaching methods and materials
used, this was speculative in most cases. Also, the authors acknowledged that
basing the study on only two pieces of writing, at the start and end of the course, was
not enough to gauge its true impact and called for more research to be conducted in
this field.
21
3.3. A way forward
From the examples of the studies conducted given in the previous section, it is
clear that our task of measuring the effectiveness of EAP programmes is far from
easy.
There is often evidence that something works on an EAP course, but being
able to say exactly what that is and why it works remains rather elusive. Although the
studies mentioned in the previous section contribute a number of useful ideas to the
search to measure EAP effectiveness such as the need to include all stakeholders
(Atherton, 2006), the necessity of tracking the students from the beginning to the end
of the process (Ridley, 2006), the use of quantitative data gathering methods to
obtain valid measurements (Morley, 2006), the basing of research on well-founded
theory (Kinzley, 2006), and the importance of attempting to link gains to classroom
activity (Storch and Tapper, 2009), none of them managed to produce anything more
than tenuous links between the EAP course and the success of the students.
In order to do this, as I suggested in section 3.2.2, I believe we need to carry
out micro-evaluations of classroom and course activities in order to obtain as much
quantitative data as possible, so that our indications of effectiveness or otherwise are
valid and based on solid evidence. Ellis (1997: 221) suggests carrying out an
‘external evaluation’ to discover:
“… whether a task actually results in any new language being learnt or in the
development of some skill. In other words, it requires teachers to determine
empirically whether the assumptions about learning that task designers make
when they design tasks are valid. This calls for a learning-based evaluation.”
(ibid.)
This could be done by recording and analysing classroom interactions to see if
any learning has taken place. If enough research of this kind is conducted, then we
will be able to make accurate judgements about the effectiveness of teaching and
learning methods.
22
3.4. Concluding remarks on measuring effectiveness
In section 3, I have attempted to answer the question as to how the
effectiveness of EAP can be measured and explained. We have looked at various
opinions regarding the best way to approach this task and the advantages and
disadvantages of the methods employed.
Taking all of this information into account, I firmly believe that any data we
gather has to be quantifiable, so that our evidence as to the effectiveness of course
activities is based on fact. The way to collect such data is via learning-based microevaluations, in which detailed analyses of classroom interactions are conducted. The
results gained from such research will provide a solid basis for best-practice in EAP.
23
4.
CONCLUSION
The aim of this report has been to provide an overview of current EAP bestpractice, and methods that can be used to inform this. It is hoped that, as well as
taking away some practical tips for course design, teaching and evaluation, the
reader has been inspired to conduct some research of their own, ideally in the form of
learning-based micro-evaluations.
If enough of us carry out such research, then we will be able to improve bestpractice, leading to better global communication within English-speaking academia,
thereby raising standards of cooperation and research and improving the work of
academics worldwide.
24
5.
FUTURE RESEARCH POSSIBILITIES
The nature of the research project that I aim to carry out has become
increasingly clear during the writing of this report. From the conclusion, it is clear that
a learning-based micro-evaluation of some aspect of one of my EAP courses is in
order.
As an advocate and practitioner of task-based learning, and given the fact that
its use is so widespread in EAP, it would seem logical for me to conduct research on
tasks that students are involved in during an EAP course. In my current context, I
could imagine recording and analysing a series of lessons in which students prepare
for and debate a controversial issue, leading to the production of argumentative
essays. This could well take place within the bounds of the ‘English for students of
Politics and Economics’ course that I currently teach, utilising the possibilities
available in the university’s state-of-the-art multimedia language laboratory for
digitally recording each individual utterance.
At the same time as reading and conducting research on measuring the
effectiveness of a task-based EAP activity, I intend to pay even closer attention than
before to new developments in the EAP world. Writing this literature report has
proved a true inspiration!
25
GLOSSARY OF KEY TERMS
academic discourse community
teachers, researchers and students who share
discourse
authentic
produced for use by native speakers of a
language
cohesion
the way text is tied together
concordancing software
computer program that produces lists of
sentences showing real examples of the usage of
a chosen word or phrase
Content-based instruction (CBI)
course combining language and content input,
either sheltered (using materials similar to those
of degree course) or adjunct (language classes
follow up on degree classes)
criterion-referenced testing
assessment based on pre-given criteria
Critical EAP
a branch of EAP which challenges academic
practices that disadvantage
discourse
instances of communication
EGAP
English for General Academic Purposes i.e. nondiscipline-specific Academic English
function
pattern used to achieve a rhetorical purpose
genre
category of texts with the same audience and
purpose
hedging
intentionally non-committal or ambiguous
language
heuristic
a systematic approach to problem-solving or
discovery (Johnstone, 2008)
‘high-stakes’ test
a test for gaining entry to university
learning outcome
statement of what a student should be able to do
as a result of what they have learned
Pre-sessional
a course which takes place before degree
subject courses begin
register
style of language used in a certain context
study skills
techniques for effective study
TESOL
Teaching English to speakers of other
languages, also known as TEFL (Teaching
English as a foreign language)
VLE
Virtual Learning Environment, an online learningsupport system
26
REFERENCES AND BIBLIOGRAPHY
Alexander, O., Argent, S., and Spencer J. (2008) EAP Essentials: A teacher’s guide to principles and
practice. Reading: Garnet.
Atherton, B. (2006) ‘How successful can a Pre-sessional course be?’, in Gillet, A. and Wray, L. (eds)
Assessing the Effectiveness of EAP Programmes. London: BALEAP.
Banarjee, J. and Wall, D. (2006) ‘Assessing and reporting performances on pre-sessional EAP
courses: Developing a final assessment checklist and investigating its validity’, Journal of English for
Academic Purposes, 5: 50-69.
Benesch, S. (2001) Critical English for academic purposes: theory, politics, and practice. London:
Lawrence Erlbaum Associates.
Biggs, J. (2003) Teaching for Quality Learning at University. Maidenhead,UK: The Society for
Research into Higher Education and Open University Press.
Clifton, R. (2004) ‘An evaluation of the Sheffield Hallam English for Academic Purposes (SHEAP)
Program’. MA (TESOL) dissertation, Sheffield Hallam University.
Dudley-Evans, T. (2004) ‘Thoughts on the past and the future of EAP’, in Sheldon, L. (ed) Directions
for the future: issues in English for academic purposes. Oxford: Peter Lang.
Ellis, R. (1997) SLA Research and Language Teaching. Oxford: Oxford University Press.
Field, J. (2007) ‘Looking outwards, not inwards’, ELT Journal, 61/1: 30-38.
Flowerdew, J. and Peacock, M. (eds) (2001) Research Perspectives on English for Academic
Purposes. Cambridge: Cambridge University Press.
Flowerdew, J. and Peacock, M. (2001) ‘Issues in EAP: A preliminary perspective’, in Flowerdew, J.
and Peacock, M. (eds), Research Perspectives on English for Academic Purposes. Cambridge:
Cambridge University Press.
Gillet, A. and Wray, L. (eds) (2006) Assessing the Effectiveness of EAP Programmes. London:
BALEAP.
Gillet, A. and Wray, L. (2006) ‘EAP and success’, in Gillet, A. and Wray, L. (eds) Assessing the
Effectiveness of EAP Programmes. London: BALEAP.
Goh, C. (1998) ‘Emerging environments of English for Academic Purposes and the Implications for
Learning Materials’, RELC Journal, 29/1 :20-33.
Hamp-Lyons, L. (2001) ‘English for Academic Purposes’, in Carter, R. and Nunan, D. (eds), The
Cambridge guide to teaching English to speakers of other languages. Cambridge: Cambridge
University Press.
Hitchcock, R. (2007), ‘Evolving academic identities: expectation and understandings affecting
transition from the home (Chinese) to a UK EAP setting’, EAP in a Globalising World: an Academic
Lingua Franca. BALEAP Conference 2007.
Hughes, A. (2003) Testing for Language Teachers. Cambridge: Cambridge University Press.
Hyland, K. (2006) English for Academic Purposes: an advanced resource book. Abingdon, UK:
Routledge.
Hyland, K. and Hamp-Lyons, L. (2002) ‘EAP: issues and directions’, Journal of English for Academic
Purposes, 1: 1-12.
Johnstone, B. (2008) Discourse Analysis. Oxford: Blackwell.
27
Jordan, R. (2002) ‘The growth of EAP in Britain’, Journal of English for Academic Purposes, 1: 69-78.
Kennedy, C. (2001) ‘Language use, language planning and EAP, in Flowerdew, J. and Peacock, M.
(eds), Research Perspectives on English for Academic Purposes. Cambridge: Cambridge University
Press.
Kinzley, S. (2006) ‘Establishing the effectiveness of EAP writing programmes for Chinese
undergraduates’, in Gillet, A. and Wray, L. (eds) Assessing the Effectiveness of EAP Programmes.
London, BALEAP.
Lynch, B. (1996) Language Program Evaluation. Cambridge: Cambridge University Press.
Lynch, B. (2001) ‘Language Assessment and Program Evaluation’, TESOL Quarterly, 35/4: 603-605.
Macmillan Education (2007) Macmillan English Dictionary. http://www.macmillandictionary.com
[accessed: March 6 2010].
Martala, M. (2006) ‘Tracking Pre-Sessional students’ writing abilities at the University of Hertfordshire’,
in Gillet, A. and Wray, L. (eds) Assessing the Effectiveness of EAP Programmes. London: BALEAP.
Morley, J. (2006) ‘Measuring gains in oral performance on Pre-Sessional courses’, in Gillet, A. and
Wray, L. (eds) Assessing the Effectiveness of EAP Programmes. London: BALEAP.
Elder, C. and O’ Loughlin, K. (2003) ‘Score gains on IELTS after 10-12 weeks of intensive English
study’, IELTS Research Report, 4: 62-87.
Penaflorida, A. (2002) ‘Nontraditional forms of assessment and response to student writing: a step
toward learner autonomy’, in Richards, J. and Renandya, W. (eds.) Methodology in Language
Teaching: an Anthology of Current Practice. Cambridge: Cambridge University Press.
Pennycook, A. (1994) The cultural politics of English as an International Language. London: Longman.
Pennycook, A. (1997) ‘Vulgar pragmatism, critical pragmatism, and EAP’, English for Specific
Purposes, 16/4: 253-69.
Pilcher, N. (2006) ‘Mainland Chinese postgraduate students during their Masters dissertations: Some
reflections and thoughts on the effectiveness of EAP from the student’s perspective’, in Gillet, A. and
Wray, L. (eds) Assessing the Effectiveness of EAP Programmes. London: BALEAP.
Quality Assurance Agency for Higher Education (2007) Subject benchmark statements: Languages
and related studies.
http://www.qaa.ac.uk/academicinfrastructure/benchmark/statements/languages07.asp#p9 [accessed:
February 22 2010].
Ridley, D. (2006) ‘Tracking a cohort of Pre-Sessional students at Sheffield Hallam University’, in Gillet,
A. and Wray, L. (eds) Assessing the Effectiveness of EAP Programmes. London: BALEAP.
Rosenfeld, M., Leung, S. and Oltman, P. (2001) ‘The Reading, Writing, Speaking, and Listening Tasks
Important for Academic Success at the Undergraduate and Graduate Levels’, TOEFL Monograph
Series RM-01-03. Princeton, NJ: Educational Testing Service.
Saunders, S. (2006) ‘Does EAP work? A personal view’, in Gillet, A. and Wray, L. (eds) Assessing the
Effectiveness of EAP Programmes. London: BALEAP.
Smith, A. (2007) ‘Developing critical awareness through case-based teaching’, in Alexander, O. (ed),
New Approaches to Materials Development for Language Learning. Oxford: Peter Lang.
Storch, N. and Tapper, J. (2009) ‘The impact of an EAP course on postgraduate writing’’, Journal of
English for Academic Purposes, 8: 207-223.
28
Swales, J. (1997) ‘English as Tyrannosaurus Rex’, World Englishes, 16: 373-82.
Swales, J. (2001) ‘EAP-related linguistic research: An intellectual history’, in Flowerdew, J. and
Peacock, M. (eds), Research Perspectives on English for Academic Purposes. Cambridge: Cambridge
University Press.
Swales, J. and Feak, C. (2000) English in Today’s Research World: A Writing Guide. Ann Arbor:
University of Michigan Press.
Swales, J. and Feak, C. (2004) Academic Writing for Graduate Students. Ann Arbor: University of
Michigan Press.
Todd, R. (2003) ‘EAP or TEAP?’, Journal of English for Academic Purposes, 2: 147-156.
Turner, J. (2004) ‘Language as academic purpose’, Journal of English for Academic Purposes, 3: 95109.
Vygotsky, L. (1978) Mind in Society. Cambridge, MA: Harvard University Press.
29