2008 年8 月
第 31 卷　第4 期
中国英语教学
（双月刊）
CELEA Journal（Bim onthly）
Aug．2008
Vol．
31 No．
4
AN INTROSPECTIVE STUDY ON
TEST－
TAKING PROCESS FOR BANKED CLOZE
Gao Xiaoying
PLA Logistics Engineering University
Gu Xiangdong
Chongqing University
Abstract
taking process for banked cloze ，focusing on the follo wing
The present study investigates the testees test－
tw o aspects ：the information sources and strategies． Em ploying sim ultaneous introspection and im mediate
retrospection ，
the study is conducted am ong 18 non－
English major students．After examining their test－
taking
processes ，
a framew ork for analyzing testees test－
taking processes for banked cloze test is put forward． And
then the researcher applies this framew ork to analyze the testees protocols． Results sho w that regarding
information sources ，clause－
level information accounts for the largest proportion ，follo wed by text－
level
information ，
sentence－
level information and extra－
textual information ，and in terms of strategies ，
the testees
sho w a clear preference for botto m －
up processing strategies ，follo wed by top－
do w n processing strategies and
test－wise processing strategies．The verbal protocols have further revealed that the proficient readers see m to
have a context－
based reading m odel and they prefer to grasp the gist of the text ，w hereas the less proficient
readers often e m ploy w ord－
based approaches and they are apt to identify parts of the text．
Key w ords
banked cloze ；
test－
taking process ；
introspection
1．Introduction
Cloze test is a testing procedure in w hich the testees are required to restore the letters or w ords that
have been syste m atically deleted fro m a continuous text（Shoha m y 1983）
．Banked cloze testis one of the
variants of cloze test．In a banked cloze test ，all the deleted w ords are put together in a bank and the
testees are required to choose the appropriate ones fro m the bank．
Cloze test has been widely applied in the fields of language testing and teaching at ho m e and abroad．
However ，
for testing researchers and language teachers ，there has been a long running argu m ent about
the construct validity of cloze test ，
that is ，w hat cloze test actually m easures and w hether it can m easure
co m prehension that ranges beyond the sentence level．Severalstudies（e．g． Bach m an 1982 ；Brow n 1983）
have found support for the claim that cloze test is capable of m easuring testees co m prehension ability at
text level ，w hereas other studies（e．g． Alderson 1983 ；Mark m an 1985 ）conclude that cloze test can not
m easure co m prehension beyond the sentence level．
However ，m ost of these previous studies are product－
oriented studies． They tend to focus on
participants test perform ance （．
i e． scores）with little inform ation available on participants cloze test－
This paper is a sm all part of the research project ，
“ A Chronological Study on the CET Washback ”（07BYY030 ），
supported by National Philosophy and Social Science Foundation of China．The tea m leader is Gu Xiangdong．
3
An Introspective Study on Test－
taking Process for Banked Cloze Gao Xiaoying ＆Gu Xiangdong
taking process itself．It m ay not always be the case that the test ite ms m easure the ability that the
researchers are interested in because testees m ay use quite different processes to answer the ite ms fro m
those anticipated（Ya m ashita 2003）
．Thus the validation of tests involving cognitive processes will not be
co m plete unless itincludes so m e exa mination of the test taking processes（Storey 1997）
．In order to reach
this goal ，
recently so m e researchers（e．g． Feld m ann ＆Ste m m er 1987 ；Storey 1997 ）began to resort to
m ore direct research m ethods ，
introspective m ethods．Nunan（1992）definesintrospection as“the process
of observing and reflecting on one s thoughts ，
feelings ，
reasoning processes ，
and m entalstate with a view
to determining the ways in w hich these processes and states determine our behavior”
（p．115）
． He uses
the term “introspection ”to “cover techniques in w hich data collection is coterminous with the m ental
events being investigated”，as well as“research contexts in w hich the data are collected retrospectively ，
that is so m e tim e after the m ental events the mselves have taken place”
（Nunan 1992 ：115）．
Feld m ann and Ste m m er （1987 ） use the m ethods of sim ultaneous introspection and im m ediate
retrospection to identify the strategies used in the learners protocols w hen w orking on C－
test． The
learners general proble m －
solving behavior is investigated and specific proble m －
solving strategies are
further identified．Seven recall strategies and three evaluation strategies are discovered in the learners
protocols．And then the researchers find that these strategies see m m ore likely to reflect botto m －
up
processing and top－
dow n processing． And these kinds of processing are represented as a continuu m on
w hich the different strategies approaches are projected．
Storey（1997）e m ploys sim ultaneous introspection followed by im m ediate retrospection to investigate
EFLlearners processes of taking MC discourse cloze test ，w hich is designed to generate the discourse
processing strategies m ainly in terms of intersentential co m prehension．Four categories of deletions are
applied ：discourse m arkers ，anaphoric pronouns ，lexical substitutes and lexical ite ms． Exa mining the
learners verbal protocols ，the researcher finds that different ite ms entail varying degrees of construct
validity．Overall ，however ，
the ite ms are proved to stim ulate the students construct－
relevant processing ，
and this test is considered to have a good degree of validity．
Ya m ashita（2003 ）investigates how skilled and less skilled readers answer a gap－
filling test （or
rational deletion cloze test）in order to exa mine w hether such tests can m easure text－
level processing
ability． Twelve Japanese EFL students （six skilled and six less skilled readers）are asked to provide
sim ultaneous introspection （or think－
aloud verbal protocols） w hen they co m plete a gap－
filling test． A
m odified Bach m an s（1985）classification syste m of cloze ite m types is e m ployed as a basic fra m ew ork to
analyze students protocols．In her fra m ew ork ，Bach m an s （1985 ）“ Across Sentences ，Within Text”
category is further subdivided into tw o types ，．
i e．“ Across Sentences ，Within Paragraph ”and “ Across
Paragraphs ，Within Text”
seven
．And the categories of“Guessing”and“ Missing”are added．As a result ，
categories are used to classify the length of textinform ation required for answering each ite m in the cloze
test． The results of this study show that both skilled and less skilled readers use text－
level inform ation
m ore frequently than other types of inform ation． The skilled readers ， however ，use text－
level
inform ation m ore frequently than the less skilled readers．Overall ，
the gap－
filling test generates text－
level
processing and differentiates well between skilled and less skilled readers．This study ，
therefore ，
supports
the claim that a gap－
filling test can be used as a test to m easure higher－
order processing ability．
Cheng Xiaokou and Li Shaoshan（2006）conduct a process－
oriented study of the construct validity of
the MC cloze test in TEM －
4（2003）
．Em ploying im m ediate retrospection ，the study is conducted a m ong
20 English m ajors． Bach m an s（1985 ）classification syste m of cloze ite m types is e m ployed as a basic
fra m ew ork to analyze students protocols． The results show that construct of using redundancy
inform ation in the cloze test is well m easured and TEM －
4 cloze test not only m easures test－
takers lower－
order processing ability（i
．e． understating inform ation across clause ，within sentence）and also higher－
order processing ability（．
i e． understanding inform ation across sentence ，within text）
．
But there still exist so m e limitations in these e m pirical researches e m ploying introspective m ethods．
These studies either investigate the types of inform ation utilized by testees to co m plete cloze ite ms（e．g．
Ya m ashita 2003），or explore the strategies e m ployed by the testees（e．g
． Feld m ann ＆Ste m m er 1987 ；
Storey 1997 ）
． So m e of the m （e．g． Ya m ashita 2003 ） even confuse these tw o ele m ents and involve
Guessing in their categorization fra m ew ork of inform ation sources．The researcher holds that it is m ore
4
CELEA Journal 80
suitable to recognize Guessing as one of the strategies． And using inform ation sources and e m ploying
strategies to answer ite ms are tw o indispensable ele m ents in cloze test－
taking process．Therefore ，w hen
we do such process－
oriented study ，
both ele m ents are supposed to be involved in one study．Furtherm ore ，
the previous e m pirical studies focus on the cloze types such as the C－
test（Feld m ann ＆Ste m m er 1987 ），
m ultiple－
choice cloze test（Cheng ＆Li 2006）and gap－
filling test（Ya m ashita 2003），however ，
studies on
the test－
taking processes in banked cloze test is not yet available in the literature．
However ，
in China ，
the innovated CET－
4 ，w hich has been carried out fro m Dece m ber 2006 ，
applies
Banked Cloze as a new part of Reading Co m prehension． As a brand－
new test form at in CET－
4 ，the
banked cloze deserves urgently to be investigated．And it is never know n that how the testees solve the
banked cloze ite ms in their test－
taking processes． Under these circu mstances ，the present study ，with
introspective m ethods e m ployed ，is conducted to explore the testees test－
taking processes for banked
cloze test．It focuses on the following research questions ：
Q1． What inform ation sources do testees use in their processes of taking banked cloze ？How do they
use these inform ation sources ？
Q2． What strategies do testees e m ploy in their processes of taking banked cloze ？How do they
e m ploy these strategies ？
2． Method
2．
1 Subjects
Eighteen non－
English m ajor students in PLA Logistic Engineering University participated in this
study．These subjects were grouped according to their reading proficiency as follows ：High Group （n ＝
6 ），Mediu m group（n ＝6），Low group（n ＝6）
．They were selected and grouped fro m 212 sopho m ores of
this university according to their scores of the Reading Co m prehension Partin CET－
4 w hich they recently
took．
2．
2 Instru ments
The banked cloze test used in the present study was chosen fro m the prototype paper of CET－
4（June
of 2006）for its appropriate difficulty level and content．This banked cloze test consists of a text（246
w ords）in w hich 10 w ords have been deleted ，
and a bank w hich contains 15 options and is placed after the
text．The subjects were required to restore the deleted w ords through selecting the appropriate options in
the bank．The researcher had already m ade sure that the selected subjectsin the present study didn t take
this banked cloze test before．
2．
3 Procedure
The m ain m ethodological proble m in this study is to decide how to discover the testees test－
taking
processes．Followed by the previous introspective researches（e．g． Feld m ann ＆Ste m m er 1987 ；Storey
1997），
the m ethods of sim ultaneousintrospection and im m ediate retrospection were chosen in the present
study．In terms of sim ultaneous introspection w hich was also know n as concurrent talking or thinking
aloud or verbalization of specific cognitions ，verbal protocols were generated at the sa m e tim e as the
individual was w orking on the task（Faerch ＆Kasper 1987）
．Regarding im m ediate retrospection ，verbal
protocols were produced im m ediately after co m pletion of the task w hile traces of the original cognition
were still present in short term m e m ory （Faerch ＆ Kasper 1987 ）
． The experim ental procedures were
carried out in the following phases．
The subjects were firstly orally inform ed about the objectives of the study and were then trained in
how to provide sim ultaneous introspection protocols by using verbal arith m etic（Ericsson ＆Sim on 1984 ；
Green 1998 ）
． And then a trial banked cloze test was given to the m for the purpose of m aking sure
w hether they have understood everything correctly and fa miliarizing the m with the task．
Subsequent to the training phase ，the subjects were asked to proceed to the sim ultaneous
introspection phase one by one． Each subject was left alone in a quiet roo m and asked to provide
sim ultaneousintrospection either in Chinese or English during their w orking on the banked cloze．Subjects
were pro m pted to continue talking by the researcherifthey fellsilentfor a period oftime．No timelimit wasset
5
An Introspective Study on Test－
taking Process for Banked Cloze Gao Xiaoying ＆Gu Xiangdong
for the subjects to co m plete the test with introspection．All the subjects utterances were audio－
taped．
And then in the im m ediate retrospection phase ，
the subject and researcher listened to the audio－
tape
together im m ediately after the subject had co m pleted the test．The subject could spontaneously co m m ent
on his or her utterances during the previous test phase and answer the researcher s questions concerning
the utterances．Itis intended to provide additionalinform ation about test－
taking process for banked cloze
test．
2．
4 Data Analysis
The verbal protocol data obtained fro m the testees covered all the processes of taking the banked
cloze test ，
including general reading processes ，
ite m answering processes ，and retrospective interview ．
For the purposes of analysis ，
the recorded verbal protocols were divided into idea units consisting of one
clause（．
i e． subject ，verb ，and m odifiers）
．Each idea unit was exa mined in terms of inform ation sources
and strategies in order to answer the tw o research questions．
First ，each idea unit was labeled in terms of inform ation sources and strategies． Second ，the
inform ation source or strategy ，w hich occurred at least 5 tim es in the testees protocols ，was given a
descriptive na m e．Third ，
three testees protocols were rando mly selected and another rater was asked to
independently categorize the protocols in terms of inform ation sources and strategies． After that ，his
classifications were co m pared with those of the researcher and the percent of inter－
rater agree m ent was
calculated as 86 percent ，
indicating that the coding of the verbal protocols is reliable．
3．Findings and Discussions
The research questions of the present study focus on tw o m ain aspects ，
i
．e． inform ation sources and
strategies．Therefore the data will be analyzed and discussed m ainly fro m these tw o aspects． After
exa mining the protocols obtained fro m the testees ，a fra m ew ork for analyzing test－
taking for banked
cloze is discovered（see Figure 1）
．Four categories of inform ation sources have been revealed ，including
clause－
level ，sentence－
level ，text－
level and extra－
textual inform ation． Furtherm ore ，twelve types of
strategies are also found in the protocols and they can be further classified into three categories ，
top－
dow n
processing strategies ，botto m －
up processing strategies and test－
wise processing strategies．
Figure 1．A fra m ew ork for analyzing testees test－
taking processes for banked cloze test
6
CELEA Journal 80
The further discussion and explanation on each inform ation source and strategy in the fra m ew ork will
be carried out in the following parts．Before that ，so m e w ords of caution are necessary． Although we
have listed all the inform ation and strategies so far identified in these protocols ，
the listis ，of course ，
an
open one and will be extended and m odified as necessary．
3．
1 Inform ation Sources
All the testees protocols of solving the cloze ite ms are classified and the frequency and proportion of
each inform ation category is calculated and presented according to the level of the testees reading
proficiency and the correct or incorrect responses to the ite ms．In doing so ，the researcher intends to
reveal how these categories of inform ation w ork in the testees test－
taking process．The results are show n
in Table 1．
Table 1．Frequencies and proportions of inform ation categories used by three groups of testees
Categories
Correct or
incorrect
High
Mediu m
Low
Total
F（n） P（％） F（n） P（％） F（n） P（％） F（n） P（％）
Clause
Correct
Incorrect
37
19
12．
6
6．
5
26
24
8．
9
8．
2
13
19
4．
4
6．
5
76
62
25．
9
21．
2
Sentence
Correct
Incorrect
19
3
6．
5
1．
0
13
9
4．
4
3．
1
10
11
3．
4
3．
8
42
23
14．
3
7．
8
Text
Correct
Incorrect
12
8
4．
1
2．
7
20
14
6．
8
4．
8
10
3
3．
4
1．
0
42
25
14．
3
8．
5
Extra－
textual
Correct
Incorrect
7
3
2．
4
1．
0
2
1
0．
7
0．
3
5
5
1．
7
1．
7
14
9
4．
8
3．
1
Total
Correct
Incorrect
75
33
25．
6
11．
3
61
48
20．
8
16．
4
38
38
13．
0
13．
0
174
119
59．
4
40．
6
As is show n in Table 1 ，the totalinform ation sources are supposed to be 180 ，for there are 18 testees
and 10 ite ms in the test．However ，
the nu m ber of the inform ation sources identified in the present study
is counted to be 293（174 ＋119）
it see ms reasonable to suggest that the testees
．In the light of this result ，
tend to use tw o or three inform ation sources to answer one ite m ． This result see ms to reflect the
interactive nature of reading processes in w hich various inform ation sources interact with each other to
help readers construct a m eaning representation（Carr ＆Levy 1990 ）
．And Table 1 also displays that the
a m ounts of inform ation sources used by the High Group and Low Group are counted to be 108（75 ＋33）
and 79（38 ＋38）respectively．In order to exa mine w hether there is a significant difference between the
a m ount of inform ation sources used by the tw o groups ，Chi－
Square test is conducted here ，w hich is a
nonpara m etric procedure to test frequencies w hen the total nu m ber of observations is m ore than 40 and
the expected frequencies of allthe cells are m ore than 5．The significance levelis set at p ＜0．
05 ，w hich
will be applied in the following analyses．The result obtained by Chi－
Square test is that the significance is
0．
003 ，and less than 0．05 ，suggesting that the total nu m ber of inform ation sources used by the High
Group is significantly larger than that of Low Group．This result indicates that the proficient readers are
m ore active in their atte m pt to co m prehend the text and m ost of their answers are induced by m ore than
one inform ation source．Not surprisingly ，
the High Group perform ed m ore successfully than Low group in
alm ost all the categories （ with 25．6 ％ and 13 ％ respectively ）
． This is also presu m ably due to the
proficient readers better understanding of the text ，w hich m akes it easier for the m to find various kinds
of and co m binations of inform ation sources in order to solve the cloze ite ms．
1）Clause－
level Inform ation
Clause－
levelinform ation refers to the inform ation provided by the clause in w hich an ite m appears．
Of the four categories of inform ation ，clause－
level inform ation is utilized by the testees m ost frequently
filling
（see Table 1）．This resultis not expected by the researcher for Ya m ashita（2003）s research on gap－
test had claim ed that cognitive processes requiring text－
level inform ation are generally provoked m ore
7
An Introspective Study on Test－
taking Process for Banked Cloze Gao Xiaoying ＆Gu Xiangdong
than any other type of inform ation． Accordingly ，we m ake the further exploration on the cases of
applying the clause－
level inform ation in order to find w hy the testees often utilize the clause－
level
inform ation as the clues to answer ite ms and how they use this category of inform ation．
As a result ，a co m m on pattern is observed in the testees protocols．They often determine the part of
speech of the deleted w ord and then choose the appropriate answer fro m the options with such part of
speech．It is hypothesized that this behavioral pattern found in the testees test－
taking processes in the
banked cloze test m ay be attributed to the effects of such test m ethod ，as such behavioral pattern is not
identified in the Ya m ashita s study on the gap－
filling test and Chen Xiaokou and Li Shaoshan s（2006 ）
study on the MC cloze test． When asked w hy they often determine the part of speech of the deleted
w ord ，m any testees reportthatthe answer can be reached easier and faster by doing so．In a banked cloze
test ，
the options are put together in one bank ，w hich offers an opportunity for the testees to classify the
options according to their parts of speech and then choose the answer fro m the options with the sa m e part
of speech as that of the deleted w ord． This co m m on pattern observed in the testees protocols m ay
expound the testees frequent use of the clause－
level inform ation sources． According to Bach m an s
（1990），test perform ance is affected by the characteristics of the m ethods used to elicit test perform ance
and it is im portant not only to understand the nature and extent of these effects ，but also to control or
minimize the m ．Therefore ，as for designing the options in the banked cloze test ，
itis suggested to create
enough distractors with the sa m e part of speech as that of the correct answer．
As is show n in Table 1 ，with the help of the clause－
levelinform ation the testees of High Group can
get m ore correct answers（12．
6 ％）than Low group（4．
4 ％）
．It see ms that the proficient readers answer
the banked cloze test m ore successfully with the help of clause－
level inform ation than the less proficient
readers．Different processing behaviors of the tw o groups can be observed in so m e protocols of the
testees ，w hich m ay provide one possible explanation for this finding．The testees of High Group tend to
use clause－
levelinform ation as a supple m entary source to get the answers．For instance ，after reaching
the answer with the text－
levelinform ation ，
the testee of High Group in Exa m ple 1 turned to the clause－
level inform ation to confirm his choice（see Exa m ple 1）
．But w hen w orking on the sa m e ite m ，the testee
of Low Group e m ployed clause－
levelinform ation as the sole source to generate the answer（see Exa m ple
2）
．
Exa m ple 1（a testee of the High Group ，w orking on ite m 6）
Brought the m ost ，m ost．That should be the w orst weather because the w hole text is talking about
so m ething bad and the nu m bers in the following sentence show thatit has caused great da m age．And
the answer should be an adjective．So Ichose J ，
“destructive”．
Exa m ple 2（a testee of the Low Group ，w orking on ite m 6）
Brought the m ost． What weather． Describe the weather． An adjective．“ Deliberately” weather ，
w hat s m eaning ，it is an adverb．“Tropical”and “notify ”are adjective． Worth is a noun．It s
“notify”．（His retrospective protocols ：Ipresu m ed that the answer is either“tropical”or“notify”
and then Ichose“notify”at rando m ．
）
83 EI Nino brought the m ost 6 weather in m odern history．
［The original text ：The 1982－
］［The
Key ：destructive］
2）Sentence－
level inform ation
Sentence－
levelinform ation refers to the inform ation provided by a larger context than the clause in
w hich an ite m appears ，
but within the sentence．There are 42 cases（with proportion of 14．
3 ％）in w hich
the testees get the correct answers by e m ploying the sentence－
levelinform ation（see Table 1）
．This result
is consistent with that of the previous researches（e．g． Bach m an 1982 ；Chen ＆Li 2006 ）
． All of the m
m aintain that processing ability in the sentence level inform ation can be tapped in the cloze test－
taking
process．
It can also be inferred fro m the statistics of Table 1 that 86．4 ％（19 out of 22 ）of sentence－
level
inform ation used by the High Group can help to reach the correct answers w hile only 47．
6 ％（10 out of
8
CELEA Journal 80
21）of such inform ation can be used correctly by the Low group．Further exa mination on the protocols of
these groups provides so m e possible explanations for this pheno m enon． A behavioral pattern frequently
spotted a m ong the testees of High Group is that they tend to take the sentence as one unit and do not
m ake the closure until they finish the w hole sentence． However ，the testees of the Low Group often
m ake haste to seek the answer fro m the options w henever they m eet the blank．This behavioral pattern
can be best illustrated by the following tw o exa m ples ：
Exa m ple 3（a testee of the High Group ，w orking on ite m 9）
Nowadays weather experts can forecast w hen an EI Nino will ，will co m e ，will． A verb should be
filled in this blank．But they are still not ，not co m pletely sure w hat leads to it or w hat affects how
strong it will be． Will ，a verb ，co m e ，
it should be“strike”
．
Exa m ple 4（a testee of the Low Group ，w orking on ite m 9）
Nowadays weather experts are able to forecast．Forecast is supposed to have the m eaning of find．
When an EI Nino will ，will co m e or happen．Yes ，
“stable”
．Ichoose“stable”．
［The original text ：Nowadays ，weather experts are able to forecast w hen an EINino will 9 ，but
they are still not 10 sure w hat leads to it or w hat affects how strong it will be．
］［The Key ：
strike］
The above protocols reveal that although the testee of High Group in exa m ple 3 had inferred the
deleted w ord might have the m eaning of“co m e”，he still tried to co m plete the w hole sentence． On the
contrary ，
the testee of Low Group in exa m ple 4 m ade choice of“stable”im m ediately after he got so m e
idea of the deleted w ord．
3）Text－
level inform ation
Text－
levelinform ation refers to the inform ation provided by a larger context than the sentence in
w hich an ite m appears ，but a context fro m within the text． As has been m entioned above ，the m ost
frequently discussed topic in the cloze researches is w hether the cloze test can m easure co m prehension
that ranges beyond the sentence level．And the previous researchers never reach a consensus on it．One of
the m ain purposes of the present study is to try to brush off this controversy by applying introspective
m ethods to look into the m ental processes of the testees w hen they take the cloze．And it is expected to
discover w hether the testees applied the text－
levelinform ation during their process of taking the banked
cloze test and how they use such category of inform ation to answer the test．
As is show n in Table 1 ，
the text－
levelinform ation is the second m ost frequent category to be invoked
w hen the testees take the banked cloze ，with 42 cases（14．
3 ％）to reach the correct answers．It im plies
that testees have to utilize text－
level inform ation frequently in order to co m plete the banked cloze test．
This resultlends support to the conclusions of the previous researches（e．g． Bach m an 1985 ；Chavez－
Oller
et al．1994 ；Ya m ashita 2003）that cloze tests are sensitive to intersentential or text－
level constraints and
can be used as a procedure to m easure global－
level co m prehension．
A closer inspection of the protocols shows that so m e individual differences exist between the High
Group and Low Group in their application of text－
level inform ation．So m e testees of High Group reach
their answers quickly by using text－
level inform ation （see Exa m ple 5 ），w hereas the Low Group testees
see m to have used various inform ation sources gradually in their atte m pts to get their answers （see
Exa m ple 6）
．
Exa m ple 5（a testee of the High Group ，w orking on ite m 1）
Change in the clim ate of the w orld． This strange ，happens every five to eight years． It s
“pheno m enon”．（Her retrospective protocol ：Ididn t get the answer w hen Iread the passage for the
first tim e．But w hen Iread it again Irealized that “pheno m enon ”should be filled in this blank
because the w hole passage talks about the natural disaster．
）
［The original text ：This strange 1 happens every five to eight years．
］［The Key ：pheno m enon］
9
An Introspective Study on Test－
taking Process for Banked Cloze Gao Xiaoying ＆Gu Xiangdong
Exa m ple 6（a testee of the Low Group ，w orking on ite m 4）
The rainfallisincreased across South Am erica．The blank is within this sentence．Flood．The deleted
w ord should be related to“flood”
． And it is so m ething that can cause the da m age since the w hole
passage talks about da m age．And also it is said that it has far－
reaching effect in the first sentence of
this paragraph ，w hich indicates thatit has greatinfluence．The deleted w ord should be an adjective．
“Destructive”has been selected．There are only tw o adjectivesleft ，
“tropical”and“stable”．It see ms
that“stable”has the similar m eaning with“destructive”
．Idon t know the m eaning of“tropical”and
usually I w on t select the unknow n w ord in the test．So Ichoose“stable”
．
［The original text ：The rainfall is increased across South Am erica ，　4 floods to Peru．
］［The
Key ：bringing］
The proficient reader in Exa m ple 5 reached the answer decidedly just with the help of text－
level
inform ation． However ，w hen the less proficient reader in Exa m ple 6 answered the ite m ，he firstly
utilized the clause－
level inform ation to identify the sentence structure ，and then used the text－
level
inform ation to infer the m eaning of the deleted w ord． Moreover he judged the part of speech of the
deleted w ord with the help of the clause－
level inform ation． Finally he e m ployed such extra－
textual
inform ation as his test experience to m ake a decision．
4）Extra－
textual inform ation
Extra－
textualinform ation refers to inform ation not provided by the text．There are 14 cases（with
the proportion of 4．
8 ％）in w hich the testees apply the extra－
textual inform ation to reach the correct
answers（see Table 1）
textual inform ation utilized by the testees in
．The protocols reveal that the extra－
the present study derives fro m tw o sources ：
their background knowledge and their test－
taking experience．
Background knowledge ，
itis also called prior knowledge of contentin tests of reading co m prehension
（Bach m an 1990 ）． Many previous researches ，such as Erickson and Molloy （1983 ），Alderson and
Urquhart（1983），have provided quite convincing evidence of an interaction between testees fa miliarity
with content area and perform ance on tests of listening and reading co m prehension（as cited in Bach m an
1990 ：273）
． The findings in the present study concur with those of the previous studies．It can be
illustrated by the following exa m ple．
Exa m ple 7（a testee of the High Group ，w orking on Ite m 9）
Weather experts are able to forecast w hen an EI Nino will ，will ，co m e ，but they are not sure w hat
leads to it or w hat affects how strong it will be．But there is not“co m e”in the bank．So it should be
“strike”．（His retrospective protocol ：The w hole text is m ainly talking about a wind ，storm wind．
And according to m y general knowledge ，
it is often said that the storm wind strikes so m ew here．
）
［The original text ：Nowadays ，weather experts are able to forecast w hen an EINino will 9 ，but
they are still not 10 sure w hat leads to it or w hat affects how strong it will be．
］［The Key ：
strike］
In addition to the background knowledge ，the testees test－
taking experience is also applied as the
extra－
textualinform ation w hen they are w orking on the ite ms．For instance ，in Exa m ple 8 the testee
inferred the m eaning of the deleted w ord with sentence－
levelinform ation and then used his previous test－
taking experience ，
i e．“attraction ”is the vocabulary frequently tested ，as a supple m entary source to
．
reach the answer．
Exa m ple 8（a testee of the High Group ，w orking on Ite m 2）
The te m peratures of ocean rise．“ As”here m eans “w hen ”
．It see ms that w hen the wind co m es
together ，
the ocean te m perature will rise as a result．
“Attraction” m eans“exciting the interests”as
well as“causing to co m e together”
． Furtherm ore ，
“attraction ”is often tested in the exa mination
w hich Ihave taken．Ialways encounter this vocabulary in the test．So Ichose“attraction”
．
the ocean te m peratures rise ，
causing the Peru
［The originaltext ：As the trade windslessen in 2 ，
10
CELEA Journal 80
current flowing in fro m the east to warm up by as m uch as 5 ℃．
］［The Key ：strength］
3．
2 Strategies
Strategies are defined deliberate ，
cognitive steps thatlearners can take to assistin acquiring ，
storing ，
and retrieving view inform ation（Paris 1983 ，as cited in Anderson 1991）
．After exa mining the protocols
of the testees ，twelve strategies are discovered． We m ake the further categorization of these twelve
strategies into three groups．Following Feld m ann and Ste m m er analysis（1987），we co m bine eight of the
twelve strategies into tw o categories ：top－
dow n processing strategies （．
i e． previewing the text before
answering the ite ms ，evaluating critically w hat is read ，predicting or guessing the m eaning of the text ，
asking oneself questions）and botto m －
up processing strategies（i
．e． guessing the m eanings of unknow n
w ords ，
stating failure to understand unknow n w ords ，
su m m arizing the inform ation of the text ，analyzing
the syntax）
．The four re m aining strategies（i．e． reading the options before reading the text ，eliminating
the options ，m atching the blank of an ite m with the options ，guessing blindly）are labeled as test－
wise
processing strategies．The frequencies and proportions of strategies in each category used by three groups
of testees are calculated as follows ：
Table 2．Frequencies and proportions of strategy categories used by three groups of testees
High
Mediu m
Low
Total
Categories of strategies
F（n） P（％） F（n） P（％） F（n） P（％） F（n） P（％）
Top－
dow n Processing Strategies
39
12．
3
28
8．
9
20
6．
3
87
27．
5
Botto m －
up Processing Strategies
62
19．
6
60
19．
0
38
12．
0
160
50．
6
Test－
wise Processing Strategies
21
6．
6
22
7．
0
26
8．
2
69
21．
8
Total
122
38．
6
110
34．
8
84
26．
6
316
100
Table 2 displays thatthere are 316 protocols w hich are reported to use strategies．The testees of High
Group e m ploy strategies 122 tim es w hile those of Low Group e m ploy 84 tim es．The Chi－
Square analysis
shows that the frequency of the strategies e m ployed by High Group is significantly higher than that of
2
Low Group （ χ
＝14．019 ，p ＝0．000 ），indicating that the proficient readers tend to e m ploy m ore
strategies in their test－
taking processes than the less proficient readers．And the testees of High Group can
also use m ore types of strategies than those of Low Group since the High Group e m ploys twelve types of
strategies and the Low Group only uses nine．Evaluating critically w hat is read ，asking oneself questions
and su m m arizing the inform ation of the text are three strategies w hich do not occur in the protocols of
Low Group． These findings reveal that the positive relationship exists between the testees reading
proficiency and the frequency of their used strategies．The proficient readers use strategies m ore often
than less proficient readers． Reading is an active process in w hich the reader m akes efficient use of
strategies to understand the text（ Olshavsky 1976 ）
．In the light of this result ，it see ms reasonable to
suggest that the proficient readers are m ore active in their atte m pt to co m prehend than less proficient
ones．
Of three categories of strategies ，
testees show a clear preference for botto m －
up processing strategies ，
followed by top－
dow n processing strategies and test－
wise processing strategies． Botto m －
up processing
strategies are applied m ost frequently by the testees ，with 160 cases found in the protocols（see Table 2）
．
And difference can be found in the frequencies of the top－
dow n processing strategies used by the testees of
High Group and Low group ，with High Group 39 cases （12．3 ％）and Low group 20 cases （6．3 ％）
respectively（see Table 2）
dow n processing strategies m ore
．The proficient readers tended to e m ploy top－
frequently than less proficient readers．Finally ，
test－
wise processing strategies occurred 69 tim es（21．
8 ％）
（see Table 2 ）in the testees protocols．They were m ainly e m ployed w hen the testees failed to answer
ite ms through the inform ation of text． More detailed analyses of these three categories of strategies will
be carried out in the following section．
11
An Introspective Study on Test－
taking Process for Banked Cloze Gao Xiaoying ＆Gu Xiangdong
1）Top－
down processing strategies
Top－
dow n reading m odels suggest that processing of a text begins in the mind of the readers with
m eaning－
driven processes ，or an assu m ption about the m eaning of a text（ Dechant 1991 ）
． Fro m this
perspective ，
readers identify letters and w ords only to confirm their assu m ptions aboutthe m eaning of the
text（Dechant 1991）
dow n strategies have been discovered in the testees top－
dow n processing．
．Four top－
Table 3 provides the results of the further exa mination on these four top－
dow n strategies．
Table 3．Frequencies of top－
dow n processing strategies used by three groups of testees
Top－
dow n Strategies
High
Mediu m
Low
Total
Preview the text before answering the ite ms
5
4
1
10
Evaluate critically w hat is read
5
5
0
10
25
18
19
62
4
1
0
5
Predict or guess the m eaning of the text
Ask oneself questions
Of these four strategies ，predicting or guessing the m eaning of the text occurs m ost frequently w hen
testees process the text（see Table 3）
．The testees perform ance in the present study supports Good m an
state m ent（1982）that reading is a psycholinguistic guessing ga m e ，
in w hich readers guess or predict the
text s m eaning on the basis of textualinform ation ，and their existing ，activated ，knowledge（as cited in
Alderson ，2003 ：17）
．The form at of the banked cloze test has actually activated such psycholinguistic
guessing ga m e since so m e w ords are deleted fro m the text and replaced with blanks． Many testees try to
guess the deleted w ord w henever they encounter a blank（see Exa m ple 9）
．
Exa m ple 9（a testee of the High Group ，w orking on Ite m 6）
EI Nino brought the m ost ，m ost awful ，or w orse weather．
（ Her retrospective protocols ：It see m ed
that the w hole text was taking about a bad thing ，
and the statisticsin the sentence following thisite m
were very large ，w hich im plied that it had caused great da m age．
）
83 EI Nino brought the m ost 6 weather in m odern history．
［The original text ：The 1982－
］［The
Key ：destructive］
Previewing the text before answering the ite msis the second frequently used top－
dow n strategy by the
testees．And the testees of High Group have higher frequent use of this strategy．As is show n in Table 3 ，
five out of six High Group readers chose to preview the w hole text before they w orked on the ite ms．On
the contrary ，only one out of six Low group readers tended to preview the text．One possible explanation
for this resultis that proficient readersincline to answer the ite m only after they have possessed an overall
understanding of the text．They want to grasp the gist of w hat the author is trying to say regardless of the
details．However ，w hen less proficient readers process the text ，they might perceive that it rese m ble a
jigsaw puzzle and just identify parts of the text． Therefore they often co m plete the ite ms without
previewing the text．
Within these four strategies ，
it is especially w orth highlighting that none of testees in Low group
report to use such strategies as evaluating critically w hat is read and asking oneself questions． However ，
the testees of High Group apply the strategy of critically evaluating 5 tim es and asking oneself questions 4
tim es（see Table 3）
．Proficient readers higher frequent use of these tw o strategies m ay im ply that they
understand the text m ore actively than less proficient readers．
2）Botto m －
up processing strategies
Botto m －
up m odels operate on the principle that the written text is hierarchically organized（i
．e． on
the grapho－
phonic ，phone mic ，
syllabic ，m orphe mic ，w ord ，and sentence levels）and that the reader first
processes the sm allest linguistic unit ，gradually co m piling the sm aller units to decipher and co m prehend
the higher units（e．g． sentence syntax）
up reading m odel e m phasizes a single－
（Dechant 1991）．A botto m －
12
CELEA Journal 80
direction ，part－
to－
w hole processing of a text．Itis also know n as part－
to－
w hole m odel．Four strategies are
discovered in the testees botto m－
up reading m odels（see Table 4）
．
Table 4．Frequencies of botto m－
up processing strategies used by three groups of testees
Botto m －
up Strategies
High
Mediu m
Low
Total
Guess the m eanings of unknow n w ords
8
6
2
16
State failure to understand unknow n w ords
3
5
13
21
Su m m arize the inform ation of the text
4
2
0
6
47
47
23
117
Analyze the syntax
up processing strategies ，analyzing the syntax is m ost frequently used by the
Of the four botto m －
testees（see Table 4）
．Many testees tend to analyze the sentence structure to identify part of speech of the
deleted w ord and then seek the option with such part of speech w hen they answer the ite ms． And the
findings in Table 4 reveal that there is greater frequency of such strategy in High Group than in Low
group．It m ay be explained by the reason thattestees of High Group have betterlinguistic knowledge than
those of Low group．
So m e differences can be found w hen testees with different levels of reading proficiency encounter
unknow n w ords．The results in Table 4 show that the testees of Low group state failure to understand
unknow n w ords 13 tim es ，w hile testees of High Group state only 3 tim es．The testees of High Group see m
to allocate their attention selectively ，probably realizing that they do not need to understand allthe w ords
in a sentence ，or perhaps avoiding pronouncing an unfa miliar w ord．So m e of High Group testees incline
to guess the m eaning of unknow n w ords w hich is e m ployed 8 tim es ，w hereas testees of Low group use this
strategy only 2 tim es（see Table 4 ）
．Less proficient readers tend to read sentence by sentence and thus
experience m ore lexical proble ms． When they don t understand w ords ，they feel they can t understand
the sentence and see m to feel totally defeated． However ，w hen proficient readers have difficulty in
understanding a w ord ，
they use various and co m bined approaches to solve their vocabulary proble m ，
such
as resorting to textual inform ation ，fleshing it out ，m aking it concrete． As a result ，their vocabulary
proble ms do not interfere their on－
going reading． The e m phasis of the less proficient readers on
identifying unknow n w ords suggests their w ord－
level m odel of reading． The proficient readers see m to
have a m ore m eaning－
based approach and don t w orry about the m eanings of w ords if they could extract
the gist of the sentence．
None of Low group testees e m ploy the strategy of su m m arizing the inform ation of the text ，w hile the
testees of High Group do use this strategy in their test－
taking processes （see Table 4 ）
．It supplies
additional evidence that less proficient readers might perceive the text as a jigsaw puzzle and they often
identify parts of the text w hen they process the text． Proficient readers are m ore interested in
constructing overall co m prehension of the text ，as is show n in the following exa m ple．
Exa m ple 10（a testee of the High Group）
Nowadays ，weather experts are able to forecast w hen an EI Nino will ，but they are still not ，sure
w hat leads to it．This text is talking about winds．Now begin to answer the ite ms．
3）Test－wise processing strategies
Test－
wise processing strategies is defined as testee s capacity to utilize the characteristics and form ats
of the test and／
or test taking situation to receive a high score （ Crehan ，Koehler ＆ Slakter 1974 ）
．
Exa mination of the protocols reveals that four test－
wise processing strategies are exploited w hen the
testees take the banked cloze test．
13
An Introspective Study on Test－
taking Process for Banked Cloze Gao Xiaoying ＆Gu Xiangdong
Table 5．Frequencies of test－
wise processing strategies used by three groups of testees
Test－
wise processing Strategies
High
Mediu m
Low
Total
Read the options before reading the text
4
4
3
11
Eliminate the options
8
7
4
19
Matching the blank of an ite m with the options
5
10
8
23
Guess blindly
4
1
11
16
Matching the blank of ite m with the options is the test－wise processing strategy m ost frequently
e m ployed during answering the banked cloze ite ms（see Table 5）
． When using the strategy of m atching
the blank of an ite m with the options ，
the testees often put the given options one by one in the blanks in
order to see w hich is suitable in the context．So m etim es the testees don t m atch the blanks with all of the
options．They firstly identify the part of speech of the deleted w ord in the blank and then m atch this
blank with the options with such part of speech（see Exa m ple 11）
．
Exa m ple 11（a testee of the High Group ，w orking on Ite m 3）
The warming of the ocean has far－
reaching effects．The hot ，hu mid air over the ocean causes severe ，
severe ，thunderstorms． The blank should be filled with an adjective．“ Stable ”，
“tropical”，
“destructive”．It is caused by the hot and hu mid air．So“tropical”is m ore suitable．
［The original text ： The hot ，hu mid （ 潮 湿 的） air over the ocean caused severe 3 thunderstorms．
］　［The Key ：tropical］
The second frequently used test－
wise processing strategy is eliminating the options．So m etim es the
testees select an answer not because itis thought to be correct ，but because the other options do not see m
reasonable or are not understandable． Such strategy is na m ed eliminating the options． There is an
exa m ple for this ：When the testee of Exa m ple 12 read the text for the first tim e ，he could not figure out
Ite m 5 so he skipped it． When he ca m e back to Ite m 5 again ，there were only three options left w hich
were“starvation”，
“pheno m enon”and“exhaustion”．And then he reported as follows ：
Exa m ple 12（a testee of the Low Group ，w orking on ite m 5）
Iseldo m encounter starvation and exhaustion according to m y test experiences ，and I will not
choose the w ords w hich Ia m not fa miliar．pheno m enon is m ore fa miliar to m e than the other tw o
options．So Ichoose pheno m enon although I m not very sure w hat it m eans．
［The original text ：So w hile so m e parts of the w orld prepare for heavy rains and floods ，other parts
face drought ，poor crops and 5 ］　［The Key ：starvation］
Guessing blindly is labeled to describe the perform ance that testees rando mly choose one fro m the
given options without any particular consideration．Exa m ple 13 serves as a good illustration for the use of
this strategy．Itis observed thatthis testee just m ade the rando m choice of“strike”and not surprisingly he
didn t choose the correct answer．
Exa m ple 13（a testee of the Low Group ，w orking on Ite m 4）
The rainfallis ，across South Am erica．Idon t know how to answer this ite m ．Ichoose O ，
“strike”．
（His retrospective protocol ：Ijust chose it at rando m ．）
［The original text ：The rainfall is increased across South Am erica ，　4 floods to Peru．
］［The
Key ：bringing］
There are also 11 cases of reading the options before reading the text．The testees might e m ploy this
strategy to m ake the m fa miliar with the options and keep the options in mind w hen they answer the
ite ms．A co m m on pattern can be found w hen the testees read the options． They tend to identify the
m eaning and part of speech of each option．So m e of the m even take so m e notes on the test paper．
14
CELEA Journal 80
4．Conclusions
The present study has investigated the testees processes of taking a banked cloze test． After
exa mining their verbal protocols reported sim ultaneously w hen they are taking the test and im m ediately
after that ，a fra m ew ork for analyzing testees processes of taking the banked cloze test is proposed ，
w hich contains tw o aspects ：
inform ation sources and strategies．
In terms of inform ation sources ，
the inform ation they use derives fro m clause ，
text ，sentence ，and
extra－
text．Of these four categories ，
clause－
levelinform ation occupies the largest proportion ，
followed by
text－
level ，
sentence－
level ，and extra－
textual inform ation． The reason w hy the testees frequently apply
clause－
levelinform ation lies in the fact that they often use the syntax clues to determine the parts of
speech of the deleted w ords．Text－
level inform ation is second m ost frequently used in the testees test－
taking processes ，w hich im plies that banked cloze test can m easure co m prehension that ranges beyond the
sentence level．
Regarding strategies ，strategies identified through exa mining testees verbal protocols include
predicting or guessing the m eaning of the text ，asking oneself questions ，previewing the text before
answering the ite ms ，
evaluating critically w hatis read ，
analyzing the syntax ，
stating failure to understand
unknow n w ords ，guessing the m eanings of unknow n w ords ，su m m arizing the inform ation of the text ，
reading the options before reading the text ，eliminating the options ，m atching the blank of an ite m with
the options ，and guessing blindly． And these twelve strategies can be further classified into three
categories ：top－
dow n processing strategies ，botto m －
up processing strategies and test－
wise processing
strategies．The testees show a clear preference to botto m －
up processing strategies ，
followed by top－
dow n
processing strategies and test－
wise processing strategies．In addition ，proficient readers see m to have a
context－
level m odel of reading and they are looking for the gist of w hat the author is trying to say
regardless of the details ，w hereasless proficient readers often e m ploy w ord－
based approaches ，
they might
perceive the text rese m ble a jigsaw puzzle ，and only parts of text can be identified w hen they process the
text．
Finally ，
there are severallimitations in the study w hich call for cautions in interpreting the results．
Firstly ，because of the tim e－
consu ming nature of the introspective m ethodology ，
this study has only been
conducted a m ong eighteen subjects and with one banked cloze test．Therefore ，it should be careful to
generalize the results to a wider range．The study should be replicated and the results should be confirm ed
by other studies with a greater variety of subjects and cloze ite ms．In addition ，
the present study does not
set a tim e limit for the testees for fear of influencing their introspective procedure． However ，the
condition of no tim e limit is different fro m norm al test－
taking situations．
References
Alderson ，J． C ． 1979． The cloze procedure and proficiency in English as a foreign language． TESOL
Quarterly 13／
2 ：219－
227．
Alderson ，
J．C ．1983．The cloze procedure and proficiency in English as a foreign language．In J． W ．Oller
217．Ro wley ：New bury House．
（ed．）．Issuesin Language Testing Research ，pp．205－
Alderson ，
J．C ．2003． Assessing Reading．Cam bridge ：CUP．
Anderson ，N ．J．1991．Individual differences in strategy use in second language reading and testing． Modern
Language Journal 75／
4 ：460－
472．
Bach man ，L．F．1982．The trait structure of cloze test scores． TESOL Quarterly 16／
1：
61－
70．
Bach man ，L．F．1985．Performance on cloze tests with fixed－
ratio and rational deletions． TESOL Quarterly
19／
3 ：535－
556．
Bach man ，L．F．1990． Fundamental Considerationsin Language Testing．Oxford ：OUP．
Bro w n ，
J．D ．1983．Acloser look at cloze ：Validity and reliability．In J． W ．Oller（ed．
）．Issuesin Language
Testing Research ，pp．237－
250．Ro wley ：New bury House．
Carr ，T ． H ．＆ B． Levy．1990． Reading and Its Development ： Component Skills Approaches． San Diego ：
Acade mic Press．
Chihara ，T ．，
J． W ．Oller ，K ． Weaver．＆M ．A ．Ch vez－Oller．1994．Are cloze ite mssensitive to constraints
across sentences ？Language Learning 27／
1 ：63－
73．
15
An Introspective Study on Test－
taking Process for Banked Cloze Gao Xiaoying ＆Gu Xiangdong
Ch vez－Oller ，M ．A ．，T ．Chihara ，K ．A ． Weaver ，＆J．W ．Oller．1994． When are cloze ite ms sensitive to
constraints across sentences ？In J． W ． Oller ＆J．Jonz （eds．
245．
）． Cloze and Coherence ，pp．229－
London ：Associated University Press．
Crehan ，D ． K ．，A ． R ． Koehler ＆J．S．Slakter．1974． Longitudinal studies of test－wiseness． Journal of
Educational Measurement 11／
2 ：209－
212．
Dechant ，E．1991． Understanding and Teaching Reading ：An Interactive Model．Hillsdale ：Lawrence Erlbau m ．
Ericsson ，K ．A ．＆H ．A ．Sim on．1984． Protocol Analysis ：Verbal Reports as Data．Cam bridge ：MIT Press．
Faerch ，C ．＆G ．Kasper．
1987．Fro m productto process —Introspective methodsin second language research．
In C ． Faerch ＆ G ． Kasper （eds．
23． Clevedon ：
）． Introspection in Second Language Research ，pp． 5－
Multilingual Matters Ltd．
Feld mann ，U ．＆B ．Ste m mer．1987． Thin
aloud a
retrospective da
in C－
te
taking ：
differ
languages — diff
learners — sa
approaches ？
In C ．Faerch ＆G ． Kasper（eds．
）．
Introspection in Second Language Research ，pp．252－
253．Clevedon ：Multilingual Matters Ltd．
Green ，A ．1998． Verbal Protocol Analysisin Language Testing Research ：A Handbook．Cam bridge ：CUP．
Mark man ，P．L．1985．Rational deletion cloze and global co m prehension in German． Language Learning 35／
3 ：423－
430．
Nunan ，D ．1992． Research Methodsin Language Learning．Cam bridge ：CUP．
Shanahan ，T ．，M ．L．Kamil ＆A ． W ．Tobin．1982．Cloze as a measure of intersentential co m prehension．
Reading Research Quarterly17／
2 ：229－
255．
Shoham y ，E ．1983．Interrater and intrarater reliability of the oralinterview and concurrent validity with cloze
procedure in Hebrew ．In J． W ．Oller（ed．
236．Ro wley ：
）．Issuesin Language Testing Research ，pp．299－
New bury House．
Storey ，P．1997． Examining the test－
taking process ：A cognitive perspective on the discourse cloze test．
Language Testing 14／
2：
214－
231．
Yamashita ，
J．2003．Processes of taking a gap－
filling test ：Co m parison of skilled and less skilled EFLreaders．
Language Testing 20／
3：
267－
293．
2006 ，
TEM －
4 完型填空测试结构效度研究—
陈晓扣、李绍山，
——答题过程分析法，
《 现代外语》第1 期。
（．．．continued fro m p．28）
Milanovic ，M ．，N ． Saville ＆ S．Shen．1996． A study of the decision－making behavior of co m position
markers．In M ． Milanovic ＆N ．Saville（eds．
）． Performance Testing ，Cognition and Assessment ：Selected
Papers from the 15th Language Testing Research Colloquium （ LTRC ），Cambridge and Arnhem ，pp．92－
114．
Cam bridge ：Cam bridge University Press．
Nakam ura ，Y ．1996．A study of raters scoring tendency of speaking ability through verbal report methods
and questionnaire analysis．Journal of Com munication Studies 5 ：3－
17．
Orr ，M ．2002．The FCEspeaking test ：Using rater reports to help interprettest scores． System 30 ：
143－
154．
Rubin ，H ．J．＆I．S． Rubin．1995． Qualitative Interviewing ：The Art of Hearing Data． Thousand Oaks ，
London ，New Delhi ：Sage．
Sakyi ，A ． A ． 2000． Validation of holistic scoring for ESL writing assessment ： Ho w raters evaluate
co m positions．In A ．J． Kunnan （ed．
152．
）
． Fairness and Validation in Language Assessment ，pp．129－
Cam bridge ：University of Cam bridge Local Examinations Syndicate and Cam bridge University Press．
Shoham y ，E．1983． The stability of oral proficiency assessment on the oral interview testing procedure．
Language Learning 33／
4 ：527－
540．
Shoham y ，E ．1988．A proposed framew ork for testing the orallanguage of second ／
foreign language learners．
Studiesin Second Language Acquisition 10 ：165－
179．
Vaughan ，C ．1991． Holistic assessment ：What goes on in the rater s mind ？In L． Ham p－
Lyons （ed．
）．
Assessing Second Language Writing in Academic Contexts ，pp．111－
126． Norw ood ，New Jersey ：Ablex
Publishing Corp．
Weigle ，S．C ．1994．Effects of training on raters of ESLco m positions． Language Testing 11 ：197－
223．
16