CONCURRENT
ANDLR
ETROSPECTIVE
ERBAL
EPORTS
INTERNATIONAL JOURNAL
OF APPLIED
INGUISTICS
, Vol.V13,
No.R2,
2003
201
Concurrent and retrospective
verbal reports as tools to better
understand the role of attention in
second language tasks
JOAQUIM CAMPS
Universtiy of Florida
This study investigates how the use of think-aloud protocols, both
concurrent and retrospective, can contribute to the study of the role
of attention in second language acquisition. It is based on the analysis
of think-aloud protocols produced by 74 first-year learners of Spanish
during and immediately after a reading and multiple-choice activity.
The activity consisted of a text with direct object pronouns and 16
blanks for which a choice of three possible antecedents was given.
The key to making the right choice was attending to both form and
meaning in the input. The data in the think-aloud protocols was
classified into mentions of the pronouns in the text as well as references
to gender and number agreement. The results showed that mention of
the targeted structure in the think-aloud protocols was related to
better performance on the task for second-semester students, but not
for first-semester students. There was some difference in the results
for the concurrent and retrospective protocols. The possible complementary nature of these two sources of data is discussed.
Introduction
Recent studies have shown the usefulness of think-aloud protocols in understanding learners’ cognitive processes as they perform tasks designed to help
them make form–meaning connections when processing input (Alanen 1995;
Jourdenais et al. 1995; Jourdenais 1998; Leow 1997, 1998a, 1998b, 2000, 2001;
Rosa and O’Neill 1999). Think-aloud protocols, described by Ericsson and
Simon (1993), have been used in psychology for over a century. In the past few
© Blackwell Publishing Ltd. 2003,
2003 9600 Garsington Road, Oxford OX4 2DQ, UK and
350 Main Street, Malden, MA 02148, USA
202
JOAQUIM CAMPS
decades they have also been applied to a variety of areas in second language
acquisition, such as writing (Cumming 1989, 1994; Raimes 1994), reading (Davis
and Bistodeau 1993; Hosenfeld 1977), strategies for language use and interpretation (A.D. Cohen 1984; Mangubhai 1991), and discourse (A.D. Cohen and
Olshtain 1993). Jourdenais (2001), Kasper (1998), and Kormos (1998) offer
recent discussions of the application of verbal reports to L2 research.
In concurrent think-aloud protocols, participants are asked to verbalize
their thought processes at the same time as they are carrying out a certain
activity (reading, writing, a problem-solving task, etc.). As Ericsson and Simon
(1993: xiii) indicate: “It is important to note that subjects verbalizing their
thoughts while performing a task do not describe or explain what they are doing
– they simply verbalize the information they attend to while generating the
answer” (emphasis in the original).
Despite the fact that think-aloud protocols have been commonly used in
other areas of second language acquisition (SLA), their use in analyzing the role
of attention in SLA is relatively recent. Empirical studies on the effect of
instruction and the role of attention in SLA have usually relied on a design
consisting of pretest, instructional treatment and posttest(s). The assumption
was that if the results in the posttest were significantly better than those in the
pretest, we could safely conclude that learners had paid attention to the
structures targeted during the instructional treatment. The introduction of
think-aloud protocols in studies on the role of attention in SLA has enriched
the experimental designs in this line of research by offering the possibility of
collecting online evidence of the actual processes learners go through when they
are exposed to linguistic input. By collecting data through verbal reports, we
can obtain more accurate and detailed information that was impossible to obtain
by means of the pretest–treatment–posttest experimental design mentioned
above.1 Studies that have used think-aloud protocols to investigate the role of
attention in SLA have analyzed the data provided in the verbal reports that the
participants produced while they performed an experimental task. This was
done in order to assess the level of awareness of the participants in a task
designed to help them attend to certain aspects of the input. This information
has allowed researchers to compare the posttest results of different groups of
learners based on the levels of awareness they showed in their verbal reports.
This methodological improvement has strengthened the empirical support for
the “noticing hypothesis” (Schmidt 1990, 1994, 1995).2
Think-aloud protocols in studies on the role of attention
to input
Think-aloud protocols have been used in studies on the role of attention to the
input in second language acquisition in order to analyze the effect of a variety
of experimental tasks or treatments. Most of these studies suggest that the
learner’s awareness of the target forms is positively related to recognition and
© Blackwell Publishing Ltd. 2003
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
203
learning of those forms. Alanen (1995) studied the acquisition of a modified
version of Finnish and reported that awareness of the target forms (as reflected
in think-aloud protocols) was positively correlated to learning in some, but not
all, experimental groups. Another example is provided by Leow (1997, 1998a,
1998b), where learners performed think-aloud protocols while doing a crossword puzzle involving irregular forms of the Spanish preterit. Leow found that
those participants who showed a higher awareness of the target forms in their
protocols performed better in the later recognition and production of those
forms. Similar results were obtained by Rosa and O’Neill (1999) in a study that
used multiple-choice jigsaw puzzles with five different treatment groups using
conditional sentences in Spanish. In that study, learners whose protocols
indicated levels of awareness classified as understanding obtained better results
than those who showed awareness at the level of noticing or no awareness at all.
Rosa and O’Neill applied the distinction between noticing and understanding
proposed by Schmidt (1990, 1995). Their study analyzed intake but not production data. Leow (2001) also found a significant positive relationship between
reported noticing of the target form (Spanish formal commands) during a reading
activity and its subsequent recognition. His study found no difference between
the enhanced and the unenhanced condition. Overall, the studies mentioned
above support Schmidt’s position that noticing, that is, attention to a particular
form in the input, together with some level of awareness of that form, allows for
further processing of that input.
Concurrent and retrospective verbal reports
Most of the research on the role of attention in SLA that used verbal reports
has relied on concurrent protocols. However, concurrent protocols are not the
only type of verbal report available to researchers: the collection of retrospective reports is another interesting option. Retrospective protocols are typically
performed immediately after the completion of a task, and participants are
asked to report on the thoughts they had while they were completing the task.
While concurrent think-aloud protocols reflect thought processes as they occur,
retrospective protocols may draw from short- and long-term memory depending
on the length of the task performed and the amount of time between the end of
the task and the beginning of the verbal report. In their discussion of thinkaloud protocols, Ericsson and Simon (1993) favor the use of concurrent protocols,
collected as the processing task is being performed, but they also recognize the
benefits of retrospective protocols as complementary data and recommend the
use of both. They indicate that even though in long cognitive tasks (beyond two
minutes in length) the retrospective reports may be incomplete, having data
from two sources may be very helpful. An important methodological point they
emphasize is that retrospective reports need to be collected as soon as possible
after the cognitive task. This would minimize the possibility that participants
may start relying on inferences rather than reporting what happened.
© Blackwell Publishing Ltd. 2003
204
JOAQUIM CAMPS
Next, we will discuss a few studies that have used retrospective measures to
assess the learners’ level of awareness of the particular target forms analyzed.
P. Robinson (1996, 1997a, 1997b) asked participants a few questions, in writing,
regarding their level of awareness of the rules and target forms they had been
exposed to. This allowed him to classify his participants according to whether
they noticed any rules, were looking for rules, or were able to verbalize the
rules. In his (1997a) study of dative alternation in English, only instructed
learners were able to refer to the key features of the rule, but many incidental
and enhanced learners mentioned the position of direct and indirect objects in
the sentence. On the other hand, few implicit learners verbalized any aspect of
the rules. The methodology used by P. Robinson departs from that typically
used in verbal reports, as described by Ericsson and Simon (1993), in that the
reports were written, rather than oral, and the questions he asked were very
direct (some were yes/no questions). This may have prompted specific responses
from the participants. Ericsson and Simon favor more open-ended methods of
letting participants retrospect on their experience.
In a study on the role of output in promoting noticing, Izumi and Bigelow
(2000) conducted retrospective interviews with four of the nine participants in
their experimental group (they do not provide details on how the interviews
were carried out). They attested great individual variation in what learners
found problematic in their production and in what they noticed in the input
they were subsequently exposed to. To assess noticing, they asked the learners
to underline those elements that they thought would be necessary for their
subsequent production.
Kormos (2000) followed more closely the methodology proposed by Ericsson
and Simon for retrospective reports. The goal of Kormos’ study was to analyze
the self-correction behavior in the oral production of Hungarian learners of
English as a foreign language. In her study, participants listened to a recording
of their production, which included examples of self-correction, and were
encouraged to stop the tape and comment on their production whenever they
had something to say. The researcher also paused the tape when she felt there
was something worth discussing.
A few studies on the role of attention in SLA have complemented the use of
concurrent think-aloud protocols with retrospective measures. In addition to
using concurrent think-aloud protocols, Alanen (1995) measured her participants’
awareness by means of a questionnaire in which they were asked to state the
rules that governed the use of the linguistic forms in the input they received.
The participants who received textual enhancement were also asked whether
they had noticed the highlighting used. Leow (2000) is a recent study that
combined a variety of measurements of learners’ awareness. He conducted
concurrent think-aloud protocols not only during the exposure task but also
during the post-exposure assessment task. Before and after the assessment task,
participants answered, in writing, two probe questions to further assess their
awareness of the target forms (the irregular preterit in Spanish). Finally, three
weeks after the completion of the experiment, Leow interviewed participants
© Blackwell Publishing Ltd. 2003
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
205
who had been classified as being unaware of the target forms and asked them to
elaborate on their answers to the probe questions and to explain their choices
in the assessment task. Although the measures Leow used in addition to the
concurrent think-aloud protocols would not match the ideal for retrospective
reports described by Ericsson and Simon (written responses, a long delay between
the end of the experiment and the interviews), they are an interesting example
of how to gather data from a variety of sources in order to increase the reliability
of the results obtained.
Several studies in SLA have successfully implemented the use of both
concurrent and retrospective verbal reports. Mangubhai (1991) conducted a
study that investigated and classified the different behaviors learners engage
in when they extract meaning from the input they receive. Learners received
individualized Total Physical Response instruction in Hindi while wearing lapel
microphones that recorded everything they said. They were encouraged to
constantly verbalize whatever was going through their minds as they performed
the TPR activities. Mangubhai collected concurrent think-aloud protocols during
20 teaching sessions. At some points during a lesson, when a comprehension
problem was observed, participants were probed to report on those problems.
These reports produced immediate retrospective data. Further retrospective
verbal reports were conducted at the end of some of the teaching sessions.
M. Robinson’s (1991) goal was to evaluate verbal reports as a means of
investigating interlanguage pragmatics knowledge. She gathered concurrent
think-aloud protocols from 12 female native speakers of Japanese as they planned
how to fill out written discourse completion items using refusals. Participants
were instructed not to think aloud while they were writing but only before they
started and whenever they paused to plan what to write. Immediately after the
task the researcher interviewed each participant, basing the discussion on the
tape recording of the think-aloud protocol the participant had just produced.
M. Robinson concluded that the combination of concurrent and retrospective
reports enhances the informative value of verbal reports, and that data from
verbal reports provide information regarding language-processing strategies that
would not be available from the analysis of discourse completion tasks alone.
Rationale and research questions
In the review presented above, we saw that research on the role of attention in
second language acquisition supports a positive relationship between awareness
of the target forms and their subsequent recognition and learning, thus favoring
Schmidt’s (1990, 1994, 1995) “noticing hypothesis”. This research points to the
usefulness of think-aloud protocols as a tool to gather data regarding the actual
online processing of input by language learners, and it is mostly based on
concurrent verbal reports to establish awareness. However, we also saw how
retrospective verbal reports have been successfully used in combination with
concurrent verbal reports in studies in other areas of SLA research. The
© Blackwell Publishing Ltd. 2003
206
JOAQUIM CAMPS
discussion of these different procedures suggests that using multiple datacollection procedures (in this case, concurrent and retrospective verbal reports)
may greatly enrich our knowledge of how learners attend to input.
By designing a study that requires participants to complete both concurrent
and retrospective protocols based on one given task, we will not only benefit
from two different sources of information but will also be able to compare the
information obtained through each type of verbal report.
The present study was designed to explore whether those participants who
noticed the targeted forms performed differently in the experimental task than
those who did not. In this study, the learners’ noticing was operationalized as
mentioning in their verbal protocols the targeted object pronouns, the object
pronouns and their agreement features, or just the agreement features as the
basis for their choice of answer in a reading and multiple-choice task.
The scope of this project was limited to analyzing how learners processed
the input present in one particular task. It did not use any post-exposure
measures to test the recognition or production of the target forms included in
the experimental task, since the aim was to carefully investigate the characteristics of this specific type of task and to analyze how learners processed the
input it provided. This study also sought to research a methodological question:
whether the concurrent and retrospective protocols, collected during the
experimental task and right after it was completed, produced similar results.
The following research questions were investigated:
a) Will the participants who notice the target forms obtain better scores in the
experimental task than those who do not?
b) Will the results vary depending on the type of think-aloud protocol considered
(concurrent or retrospective)?
c) Will the results vary depending on whether students are in first- or secondsemester courses?
The experimental study
PARTICIPANTS
A total of 74 first-year Spanish college students, all native speakers of English,
participated in the study. The pool of participants was drawn from five sections
of first-semester Spanish classes and seven sections of second-semester Spanish.
The final distribution by groups was the following: 29 in the first-semester group
and 45 in the second-semester group.3 All participants took part in the study
during the same week in July of 1998. The first-semester students performed
the task the week after they had been introduced to the target structure as part
of their regular Spanish class, while the second-semester students had been
introduced to the target structure several weeks earlier, during their first
semester of study. Although no independent measures of proficiency were used
to compare the two groups, the second group had had many more opportunities
© Blackwell Publishing Ltd. 2003
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
207
to be exposed to those forms in the input they received and to produce them
during their regular class activities.
TARGETED LINGUISTIC FORM
The study focused on the processing of third-person object clitic pronouns in
Spanish (lo, la, los, las). Sentence (1) is an example of the structure considered,
where la refers to una carta:
1) Juan escribe una carta y después la envía por correo.
‘John writes a letter, and then he mails it.’
The structural characteristics of the constructions with clitic pronouns, and
especially the fact that clitic pronouns can never appear in isolation, make
them an appropriate target for studies that deal with form–meaning connections.
The clitic pronouns considered in this study must always have a referent (an
antecedent), either overtly present in the linguistic context or understood in a
more general, situational context. Thus in example (1), the agreement features
(gender and number) of the clitic pronoun la have to match with those of its
antecedent una carta.
MATERIALS
All participants received the same text, a narration of events which contained
four examples each of the four third-person direct object clitic pronouns (lo, la,
los, las) and 16 blanks. The learners’ task was to read the text and choose one
of the three options provided for each blank (see Appendix). Sentence (2) is an
example taken from the text, with the possible answers available to the learners:
2) Recoge ____ del mes pasado que tiene por el suelo y las lleva a reciclar.
‘(She) picks up ____ from last month, which are lying on the floor, and
takes them out for recycling.’
a. las revistas (‘magazines’)
b. los periódicos (‘newspapers’)
c. las blusas (‘blouses’)
The participants had to check both how well the meaning of a possible answer
fit in the context and whether the gender and number of the noun phrase
agreed with the clitic pronoun related to it. In the example given, if the learner
just focused on meaning, both answers (a) las revistas and (b) los periódicos
would appear to be correct, since they can be from last month and can be
recycled. However, only las revistas agrees in gender and number with the direct
object pronoun las. If the learner just focused on the form, the agreement
features of the pronoun las, then both (a) las revistas and (c) las blusas would
appear to be correct choices. However, blouses is not a good choice, because
the text talks about something from last month that María is going to recycle.
© Blackwell Publishing Ltd. 2003
208
JOAQUIM CAMPS
The order of the three possible types of answers was varied randomly from item
to item.
PROCEDURE
The entire data collection process took place in a language laboratory where
students sat at individual recording booths. After the researcher introduced
himself and the general purpose of the study, students read and signed the consent
form, which contained detailed information about the study. Next, participants
completed a questionnaire with personal and language background information.
The researcher read aloud instructions on how to perform think-aloud
protocols for the activities that were going to follow. The participants were
asked to perform both concurrent and retrospective think-aloud protocols. They
were told about the retrospective protocol only after the experimental task was
completed. The script used for the concurrent protocol, adapted from Ericsson
and Simon (1993), was the following:
In this experiment we are interested in what you think about when you perform a
reading and multiple-choice activity in Spanish. In order to find out, I am going to
ask you to THINK ALOUD while you perform the task. What I mean by “think
aloud” is that I want you to tell me EVERYTHING you are thinking while you are
doing this activity. I would like you to talk CONSTANTLY while you perform the
activity. I don’t want you to try to plan out what you say or to try to explain what
you are saying. Just act as if you were alone in the room speaking to yourself.
What’s most important is that you keep talking, and talk clearly and loudly enough
into the microphone. If you are silent for any period of time I will remind you to talk
by saying “What are you thinking?” Similarly, if you begin to speak softly, I may ask
you to speak a little more loudly. I will not be able to help you in any way, but you
can use the vocabulary list provided. You will be audiotaped while you work on the
task. Do you understand what I want you to do?
After that, the researcher provided an example of a think-aloud protocol
by performing a multiplication task (24 × 26) on the board for the entire class
to see. Next, the participants were asked to perform another multiplication
task (14 × 34) while thinking aloud and recording their voices on tape. These
examples were included in order to provide the participants with some training
on the procedure involved in think-aloud protocols, as suggested by Ericsson
and Simon (1993).
In the next step, participants were given a bilingual vocabulary list with
items relevant to the main activity that would follow. They received another
page with five different categories, e.g. Cosas que se pueden comer (‘Things you
can eat’). Their task was to classify the Spanish words into their corresponding
categories. They were asked to perform a think-aloud protocol while doing this
task and were given eight and a half minutes to carry it out. This vocabulary
activity served a double purpose. On the one hand, it provided additional
practice in performing a think-aloud protocol before the main task. On the
other hand, it helped to control for vocabulary familiarity by providing a review
of the vocabulary participants would need in order to carry out the main task.
© Blackwell Publishing Ltd. 2003
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
209
After the vocabulary activity, the participants moved on to the main activity,
for which they were asked to provide a think-aloud protocol while completing
the reading and multiple-choice activity. They were allowed to check with the
vocabulary list as many times as needed and to complete the task at their own
pace. Most of the participants finished in 7 to 12 minutes.
As each participant finished the main task, they were given the following set
of written instructions for a retrospective think-aloud protocol: “Please, talk
about what you remember with regard to how you went about performing the
multiple-choice activity you just completed.” Most participants completed this
final task in approximately 2 minutes.
SCORING PROCEDURE AND ANALYSIS
In scoring the multiple-choice task, 1 point was given for every correct answer.
The maximum score possible was 16. A coding system was developed to classify
the different types of comments found in the think-aloud protocols. Two slightly
different sets of categories were developed: one for the concurrent think-aloud
protocols and one for the retrospective think-aloud protocols.
The analysis of the concurrent protocols identified specific mentions of the
object pronoun forms which indicated that the choice of answer was based on
that pronoun, as well as general mentions of gender and number. The specific
mentions of pronouns were further subclassified as making reference to gender
or to number features. Below are the categories used in the concurrent thinkalouds, with some examples.
Mention of a pronoun:
3) I’ll say lettuce ’cause of la . . . yeah, la.
Mention of a pronoun and reference to gender:
4) Let’s see, lo is masculine so it would have to be plato [dish].4
Mention of a pronoun and reference to number:
5) I thought bananas ’cause it’s plural, so it’s las pone [puts them], so it can’t
be un tenedor [a fork] . . . and spoon . . . so, that makes sense.
General mentions of gender or number:
6) Let’s go back to these. One, feminine, and this is still feminine, oh
man . . . that’s not plural though.5
The data in the retrospective protocols were classified in terms of general
mentions of object pronouns and agreement (gender and number). These types
were considered to be equivalent to the categories used in the concurrent
protocols, because these mentions made reference to either the object pronouns or
the agreement features that make possible the matching of pronouns and their
antecedents. Below are the categories used in the retrospective think-alouds,
followed by some examples.
© Blackwell Publishing Ltd. 2003
210
JOAQUIM CAMPS
Reference to object pronouns:
7) I would . . . look at the direct object pronouns and determine whether it was
male, I mean masculine or feminine.
Reference to gender:
8) I looked for what went with the masculine and feminine.
Reference to number:
9) Usually I had to finish the sentence to see to, like, see what it is they are
referring to later on . . . if it’s, like, plural or singular.
While in the concurrent protocols, most references were directly related to
specific examples of pronouns in the input, although there were also general
mentions of gender and number, the comments in the retrospective protocols
were more general. They included references to paying attention to pronouns
and to gender and number features in order to solve the multiple-choice task,
but they did not focus on specific items in the task.6 In view of the limited
number of mentions of pronouns and their agreement features, and given the
methodological difficulty of assigning specific examples to categories reflecting
different levels of awareness (e.g. noticing vs. understanding), a conservative
approach to the analysis of the data was taken. Thus the participants were
sorted into two groups: those who mentioned pronouns and their agreement
features and those who did not.
Results
The number of participants who mentioned pronouns, pronouns and their
agreement features, or just the agreement features in their protocols as the
basis for their choice of answer was fairly limited. Only 6 out of 29 students in
first semester and 10 out of 45 in second semester did so in the concurrent
think-aloud protocols. This represents only about 20% of the participants, as
seen in Table 1. The percentage was slightly higher in the retrospective protocols
(around 27%).
Table 1. Participants who mentioned pronouns and agreement in protocols
1st semester
(N = 29)
Concurrent
Retrospective
2nd semester
(N = 45)
Total
(N = 74)
no.
%
no.
%
no.
%
6
8
20.7
27.6
10
12
22.2
26.7
16
20
21.6
27.0
© Blackwell Publishing Ltd. 2003
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
211
Table 2. 2 × 2 ANOVA for Group × Level based on concurrent and retrospective
protocols
df
SS
MS
F-value
p
Concurrent
Group
Level
Group × Level
Error
1
1
1
70
9.35
12.36
20.27
248.76
9.35
12.36
20.27
3.55
2.63
3.48
5.70*
.109
.066
.020
Retrospective
Group
Level
Group × Level
Error
1
1
1
70
5.47
6.09
8.91
269.22
5.47
6.09
8.91
3.85
1.42
1.58
2.32
.237
.213
.133
* statistically significant
The scores in the multiple-choice task for each of the 74 participants were
computed. Recall that the maximum score possible was 16, corresponding to the
16 blanks. Table 2 shows the results of the 2 × 2 ANOVA performed on the data
based on the concurrent and the retrospective protocols. For the concurrent
protocols, neither Group (mention vs. no mention) nor Level (1st vs. 2nd
semester) were significant, although Level showed a trend towards significance.
However, the interaction Group × Level was significant. Similar results were
obtained based on the retrospective protocols: neither of the two independent
variables showed significance, but in this case the interaction between Group
and Level was not significant, either.
Table 3 shows a comparison of mean scores and the results of t-tests based
on the type of protocol, semester and mention of pronouns. When comparing
those who mentioned pronouns and those who did not, there were significant
differences in accuracy on the multiple-choice task only for second-semester
participants; this was true for both types of protocol, but the effect size was
larger for the concurrent protocols.7 Looking at all those who mentioned
pronouns, within the concurrent protocols the mean for the second-semester
participants was significantly higher than for those in the first-semester, with a
large effect size. There were no other significant effects.
Table 4 shows the individuals in each group who mentioned pronouns and
agreement during at least one of the two types of protocols, how many times
they did so, and the types of mentions they made. For instance, participant X27
from the first-semester group mentioned pronouns and their agreement three
times in her concurrent think-aloud protocol and two times in her retrospective
protocol. We can see that in the first-semester group only three participants
mentioned pronouns or their agreement features in both their concurrent and
retrospective protocols, while seven of the second-semester students did so.
With regard to what features those participants made reference to in their
© Blackwell Publishing Ltd. 2003
212
JOAQUIM CAMPS
Table 3. Comparison of results on multiple-choice task by type of protocol, mention of pronouns and course level
Concurrent
No. of
students
© Blackwell Publishing Ltd. 2003
1st semester
Mention
No mention
6
23
2nd semester
Mention
No mention
10
35
Mention
1st semester
2nd semester
6
10
No mention
1st semester
2nd semester
23
35
* = statistically significant; d = effect size
Mean
score
10.67
11.09
p = .666, d = .20
13.00
10.80
p = .001*, d = 1.27
10.67
13.00
p = .028*, d = 1.27
11.09
10.80
p = .575, d = .15
Retrospective
s.d.
No. of
students
1.97
2.13
8
21
1.76
1.73
12
33
1.97
1.76
8
12
2.13
1.73
21
33
Mean
score
10.88
11.05
p = .850, d = .08
12.33
10.91
p = .045*, d = .76
10.88
12.33
p = .151, d = .70
11.05
10.91
p = .797, d = .07
s.d.
2.17
2.09
2.02
1.81
2.17
2.02
2.09
1.89
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
213
Table 4. Frequency of mention of pronouns, gender and number by participant
1st semester
Stud.
X27
X31
X67
3
X11
X53
X63
3
X7
X9
X21
X35
X49
5
11
Concurrent
3 (O, G)
1 (G)
2 (G, N)
6
Retrospective
2 (O, G, N) +
1 (O, G) +
2 (G, N) =
5
2nd semester
Stud.
Y11
Y53
Y55
Y57
Y61
Y93
Y105
7
2 (N)
2 (G, N)
1 (O)
5
0
Y13
Y33
Y97
3
0
11
1 (G, N)
1 (G, N)
1 (G)
1 (N)
1 (O, G)
5
10
Y15
Y23
Y79
Y83
Y101
5
15
Concurrent
3 (O, G)
1 (O)
1 (N)
2 (G, N)
3 (O, G)
1 (G)
1 (N)
12
Retrospective
1 (G) –
1 (G) ≠
1 (G) ≠
1 (G, N) =
1 (O, G) =
1 (G) =
1 (N) =
7
1 (G)
1 (O)
1 (O)
3
0
0
15
1 (N)
1 (G, N)
1 (G)
2 (G, N)
1 (N)
6
13
O = Object; G = Gender; N = Number; totals in italics
+ increase in features; – decrease in features; = same features; ≠ different features
comments, half of the participants mentioned the same types of elements in
both protocols (e.g. X67 referred to both gender and number), two others
provided additional information in the retrospective protocol (e.g. X27 had
not mentioned number in the concurrent protocol), and only one provided
less information in the retrospective report (Y11). All but one of those 10
participants made reference to gender.
There were the same number of participants at both levels who made
mention of pronouns or agreement in only one of their protocols. Overall, more
second-semester students mentioned pronouns or agreement. The total number
of mentions of relevant features was slightly higher in the concurrent protocols
for both groups.8
When considering the cases where the learners mentioned the pronouns
or some of their characteristics in the concurrent think-aloud protocols, it is
important to look into the relationship between mentioning a pronoun or its
agreement features and getting the corresponding item right. The learners
mentioned pronouns on 26 occasions in the concurrent protocols, and on 24
of those occasions they chose the correct answer. That is an accuracy level of
© Blackwell Publishing Ltd. 2003
214
JOAQUIM CAMPS
92%, compared to only 69% accuracy for those items that were not mentioned
(802 out of 1158). This gives a clear indication that when the learners
mentioned a pronoun, they made the right choice of antecedent in the vast
majority of the cases.
The research questions in this study explored the possible effect of three
different factors on the results obtained: (a) the difference between noticing and
not noticing the target forms, (b) the effect of the type of protocol considered,
and (c) the course level at which the learners were enrolled. The results
obtained show an interaction of factors that lead us to address the answers to
the three questions in an interrelated manner. In summary, when considering
all participants, those who mentioned the object pronouns and their agreement
features in their think-aloud protocols did not obtain higher scores than the
participants who did not mention the pronouns and their features. However,
among the second-semester students, those who mentioned the pronouns and
their features did better than those who did not, and that was the case with
both types of protocols. Additionally, among all those who mentioned the target
forms, the scores for the second-semester group were significantly higher than
for the first-semester group when we considered the concurrent protocols.
Discussion
The results of this study are in line with those of Leow (1997, 1998a, 1998b,
2000, 2001), and Rosa and O’Neill (1999), in that the second-semester students
who noticed the targeted forms and mentioned them in their protocols also
obtained the best results in the experimental task. However, that was not the
case with the first-semester students. This finding, together with the lack of
difference between first- and second-semester students within the group of those
who did not mention pronouns or their features, may lead us to postulate a
difference in the ability of the two groups to make use of the information they
attended to. This could be due to their ability to rehearse in short-term memory
the information they had just attended to (P. Robinson 1995). We could argue
that the first-semester students who mentioned the pronouns did not benefit
much from paying attention to that particular aspect of the input because they
were not able to process the information provided deeply enough to activate
its rehearsal in short-term memory. On the other hand, the second-semester
students who mentioned the pronouns may have been able to process the
information at a deeper level and activate its rehearsal in short-term memory.
This may have helped them obtain significantly different results from those of
the second-semester students who did not mention the pronouns in the input.
The first-semester students’ more limited processing capacity (VanPatten 1996;
Wickens 1989) may have prevented them from using the input they noticed with
the same success as the second-semester students.9
Regarding the comparison of the results based on the concurrent and the
retrospective reports, in the statistical comparisons the results for concurrent
© Blackwell Publishing Ltd. 2003
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
215
and retrospective protocols agreed in all but two of the measures considered.
One of the differences was in the analysis of variance of the whole database,
where no interaction was found between Group and Level based on the
retrospective protocols. However, in the descriptive analysis we saw that many
participants provided as much information, if not more, in the retrospective
protocols as in the concurrent protocols.
Despite the absence of complete agreement between the two different measures, it seems that using both types of verbal reports in the same study may
help us obtain richer data. This is particularly important when we try to assess
the attentional processes of the participants in a study. As Leow (2000) points out,
exploring the learners’ internal processes by means of multiple data-elicitation
measures may offer the necessary evidence to better understand how participants
go about completing a specific task. If only concurrent protocols had been used
in this particular study, we would have counted as many as ten of the participants
among those who did not notice the targeted forms, because they only commented
on those forms in their immediate retrospective protocols. In this study, that
would have meant undercounting the number of participants who noticed the
targeted forms by 37%.
A final point regarding retrospective protocols is their level of costeffectiveness. There are various ways to use them. In some research designs
researchers may need to meet individually with participants, replay a recording
of their performance, and ask them to comment.10 The present study sought to
obtain general comments about the task that the participants had performed.
Since the data were collected from all the participants at the same time
(recording their comments in their individual laboratory booths), the collection
of retrospective reports only added three or four minutes to the total datacollection time.
An issue that needs to be discussed here is the relatively low frequency
of mentions of the target forms that appear in the verbal reports (none of
the participants made more than three mentions in the concurrent protocols).
A possible explanation is the nature of the task. Although the task consisted of
16 blanks with three choices for each, which could technically be considered
similar to the problem-solving tasks in other previous research, such as the
crossword puzzle used in Leow (1997, 1998a, 1998b, 2000) and the jigsaw
puzzle used in Rosa and O’Neill (1999), there was one crucial difference. In
the current study the 16 blanks were embedded in a long text (237 words).
While tasks in previous studies generated much more evidence of awareness
from the participants, possibly because they may have focused on each element
of the puzzles one at a time as discrete units, the learners in the present study
may not have felt the need to focus and report on each individual item, since
they may have approached the task more globally and not as a series of tasks to
be completed one item at a time. A similar explanation is offered in Leow
(2001), where a lower frequency of reported noticing was explained by the
fact that the target forms were part of a text and not part of a discrete-point
problem-solving task.
© Blackwell Publishing Ltd. 2003
216
JOAQUIM CAMPS
With regard to verbalization of thought processes, it is important to keep
in mind a potential limitation of any study involving think-aloud protocols:
although such data can provide evidence of awareness of the elements targeted
on the part of the learners, lack of verbalization of their thought processes does
not mean that learners are not aware of the elements they are processing (A.D.
Cohen 1987; Seliger 1983). In connection to this point, the effectiveness of
think-aloud protocols may be limited if participants do not provide enough
useful data due to a lack of practice with the think-aloud procedure. This study
provided learners with training on how to perform think-aloud protocols in
order to address this limitation and produce more reliable results. However, it
is difficult to assess the degree of familiarity with the procedure the participants
achieved with the practice examples carried out before the main task.
Another factor to keep in mind when carrying out concurrent think-aloud
protocols is the issue of reactivity, that is, the possibility that performing a
think-aloud protocol may affect the learners’ thought processes. Jourdenais
(2001) offers a discussion of such concerns, while Ericsson and Simon (1993),
based on a review of studies in psychology, state that, when carried out properly,
think-aloud protocols do not influence the sequence of thought, although they
may increase the amount of time needed to complete a task.11
There are other limitations which are more specific to this study. First, it
was purposely designed to explore only how learners processed information in
one particular experimental task, without measuring the possible effects of
exposure to this task in terms of subsequent recognition or production of the
targeted forms. The decision to limit the scope of the study was based on the
belief that it is important to carefully investigate the characteristics of a specific
task before attempting to explore its effects as a possible instructional task in a
pretest–posttest experimental design. Another limitation is that the discussion
in this article is mostly confined to a quantitative analysis of the data. Future
work will benefit from an analysis of the comments in the participants’ verbal
reports from a qualitative point of view by comparing the remarks in their
concurrent protocols to those in their retrospective reports. This type of analysis
should further develop our understanding of the differences between the two
types of reports and how they can complement each other in investigating how
learners attend to input.
Conclusion
This study adds further evidence to research that has shown the connection
between noticing targeted forms and success in linguistic tasks involving these
forms. The results of the study suggest a possible difference between two levels
of beginning learners with regard to the benefits they obtain from attending to
forms targeted in the input. From a methodological point of view, this study has
discussed the differences and similarities between the results obtained from
concurrent and retrospective protocols, both at the level of statistical analysis
© Blackwell Publishing Ltd. 2003
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
217
and at the descriptive level. The evidence shows that retrospective protocols
are an effective complement to the use of concurrent protocols. It would be
to the benefit of the field if more studies on the role of attention in SLA
implemented the use of both concurrent and retrospective verbal reports in
order to more fully assess the potential advantages of using both types together
when studying attentional processes.
Acknowledgement
This research was supported by a grant from the College of Liberal Arts and Sciences
at the University of Florida. An early version of this paper was presented at the
Conference on Form–Meaning Connections in Second Language Acquisition held in
Chicago in February, 2002. I am grateful to Renée Jourdenais and Gillian Lord for
their comments on earlier versions of this article. I would also like to thank the two
anonymous reviewers for their suggestions.
Notes
1. Leow (1998b) pointed out this internal validity issue as something affecting many
studies in SLA. Further discussion can be found in Leow (1999).
2. Other on-line measures used to assess learners’ attention to input include eyetracking and some applications of computer-based instruction. See Tanenhaus and
Spivey-Knowlton (1996) and Blake (1998) for respective descriptions.
3. In all, data were collected from 155 participants, but 81 of them were exposed to a
type of treatment that will not be discussed in this article.
4. Translation of Spanish words is provided in brackets.
5. In some cases, participants made reference to several traits in a given mention, for
example, mentioning gender and number at the same time.
6. The two types of protocols also differed in that the retrospective protocols elicited
comments on how the participants performed the task, while the concurrent protocols
did not.
7. An effect size of .80 or greater is considered a large effect. Values between .50 and
.80 indicate a medium effect, while anything below .50 indicates a small effect
(Norris and Ortega 2000, following J. Cohen’s 1988 recommendation).
8. See note 5.
9. As one of the reviewers pointed out, the differences in processing may also be due to
individual differences between learners. It is plausible that a specific first-semester
learner may have benefited more from attending to the input than a second-semester
student, despite the fact that the group scores are higher for the second semester.
For example, in the analysis based on the participants who mentioned pronouns in
their concurrent protocols, the top four learners in the first-semester group had higher
scores than the two weakest learners in second-semester group. However, 8 of the
10 learners in the second-semester group had higher scores than all the learners in
the first-semester group.
10. An example of this would be the application of stimulated recall methodology (Gass
and Mackey 2000).
© Blackwell Publishing Ltd. 2003
218
JOAQUIM CAMPS
11. Leow and Morgan-Short (2002) empirically addressed the issue in the context of a
reading task in Spanish as a foreign language and found no reactivity effects – that
is, no difference between learners who performed think-alouds and those who did not.
References
Alanen, R. (1995) Input enhancement and rule presentation in second language
acquisition. In R. Schmidt, Attention and awareness in foreign language
learning (Technical Report #9). Honolulu: University of Hawai’i, Second
Language Teaching and Curriculum Center. 259–302.
Blake, R.J. (1998) The role of technology in second language learning. In
H. Byrnes, Learning foreign and second languages. New York: The Modern
Language Association of America. 75–104.
Cohen, A.D. (1984) On taking language tests: what the students report. Language
Testing 1: 70–81.
—— (1987) Using verbal reports in research on language learning. In C. Faerch
and G. Kasper, Introspection in second language research. Clevedon:
Multilingual Matters. 82–95.
—— and E. Olshtain (1993) The production of speech acts by ESL learners.
TESOL Quarterly 27: 33–56.
Cohen, J. (1988) Statistical power analysis for the behavioral sciences (2nd
ed.). Hillsdale, NJ: Erlbaum.
Cumming, A. (1989) Writing expertise and second language proficiency.
Language Learning 39: 81–141.
—— (1994) Writing expertise and second-language proficiency. In A. Cumming,
Bilingual performance in reading and writing. Ann Arbor: John Benjamins.
173–221.
Davis, J. and L. Bistodeau (1993) How do L1 and L2 reading differ? Evidence
from think aloud protocols. The Modern Language Journal 77: 459–72.
Ericsson, K.A. and H.A. Simon (1993) Protocol analysis. Cambridge, MA: MIT
Press.
Gass, S. and A. Mackey (2000) Stimulated recall methodology in second
language Research. Mahwah, NJ: Lawrence Erlbaum.
Hosenfeld, C. (1977) A preliminary investigation of the reading strategies of
successful and nonsuccessful second language learners. System 5: 110–123.
Izumi, S. and M. Bigelow (2000) Does output promote noticing and second
language acquisition? TESOL Quarterly 34: 239–78.
Jourdenais, R. (1998) The effects of textual enhancement on the acquisition of
the Spanish preterit and imperfect. Unpublished doctoral dissertation,
Georgetown University, Washington, DC.
—— (2001) Cognition, instruction and protocol analysis. In P. Robinson,
Cognition and second language instruction. Cambridge University Press.
354–75.
—— , M. Ota, S. Stauffer, B. Boyson, and C. Doughty (1995) Does textual
enhancement promote noticing? A think-aloud protocol analysis. In R.
© Blackwell Publishing Ltd. 2003
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
219
Schmidt, Attention and awareness in foreign language learning (Technical
Report #9). Honolulu: University of Hawai’i, Second Language Teaching
and Curriculum Center. 183–216.
Kasper, G. (1998) Analysing verbal protocols. TESOL Quarterly 32: 358–62.
Kormos, J. (1998) Verbal reports in L2 speech production research. TESOL
Quarterly 32: 353–8.
—— (2000) The role of attention in monitoring second language speech
production. Language Learning 50: 343–84.
Leow, R.P. (1997) Attention, awareness, and foreign language behavior.
Language Learning 47: 467–505.
—— (1998a) The effects of amount and type of exposure on adult learners’ L2
development in SLA. Modern Language Journal 82: 49–68.
—— (1998b) Toward operationalizing the process of attention in second language
acquisition: evidence for Tomlin and Villa’s (1994) fine-grained analysis of
attention. Applied Psycholinguistics 19: 133–59.
—— (1999) The role of attention in second/foreign language classroom research:
methodological issues. In J. Gutiérrez-Rexach and F. Martínez-Gil, Advances
in Hispanic linguistics. Somerville, MA: Cascadilla Press. 60–71.
—— (2000) A study of the role of awareness in foreign language behavior:
aware versus unaware learners. Studies in Second Language Acquisition
22: 557–84.
—— (2001) Do learners notice enhanced forms while interacting with the L2?
An online and offline study of the role of written input enhancement in
L2 reading. Hispania 84: 496–509.
—— and K. Morgan-Short (2002) To think aloud or not to think aloud: the
issue of reactivity in SLA Research methodology. Paper presented at the
Conference on Form and Meaning Connections in SLA, Chicago, IL.
Mangubhai, F. (1991) The processing behaviours of adult second language
learners and their relationship to second language proficiency. Applied
Linguistics 12: 268–98.
Norris, J.M. and L. Ortega (2000) Effectiveness of L2 instruction: a research
synthesis and quantitative meta-analysis. Language Learning 50: 417–
528.
Raimes, A. (1994) Language proficiency, writing ability, and composing strategies:
a study of ESL college student writers. In A. Cumming, Bilingual performance
in reading and writing. Ann Arbor: John Benjamins. 139–72.
Robinson, M. (1991) Introspective methodology in interlanguage pragmatics
research. In G. Kasper, Pragmatics of Japanese as native and
target language (Technical Report #3). Honolulu: University of Hawai’i,
Second Language Teaching and Curriculum Center. 29–84.
Robinson, P. (1995) Attention, memory, and the “noticing” hypothesis. Language
Learning 45: 283–331.
—— (1996) Learning simple and complex second language rules under implicit,
incidental, rule-search, and instructed conditions. Studies in Second
Language Acquisition 18: 27–67.
© Blackwell Publishing Ltd. 2003
220
JOAQUIM CAMPS
—— (1997a) Generalizability and automaticity of second language learning
under implicit, incidental, enhanced, and instructed conditions. Studies in
Second Language Acquisition 19: 223–47.
—— (1997b) Individual differences and the fundamental similarity of implicit
and explicit adult second language learning. Language Learning 47: 45–99.
Rosa, E. and M. O’Neill (1999) Explicitness, intake, and the issue of awareness.
Studies in Second Language Acquisition 21: 511–56.
Schmidt, R. (1990) The role of consciousness in second language learning.
Applied Linguistics 11: 129–56.
—— (1994) Deconstructing consciousness in search of useful definitions for
applied linguistics. AILA Review 11: 11–26.
—— (1995) Consciousness and foreign language learning: a tutorial on the
role of attention and awareness in learning. In R. Schmidt, Attention and
awareness in foreign language learning (Technical Report #9). Honolulu:
University of Hawai’i, Second Language Teaching and Curriculum Center.
1–63.
Seliger, H.W. (1983) The language learner as a linguist: of metaphors and
realities. Applied Linguistics 4: 179–91.
Tanenhaus, M.K. and M.J. Spivey-Knowlton (1996) Eye-tracking. Language
and Cognitive Processes 11.6: 583–8.
VanPatten, B. (1996) Input processing and grammar instruction. Norwood,
NJ: Ablex.
Wickens, C.D. (1989) Attention and skilled performance. In D.H. Holding,
Human skills. New York: John Wiley. 71–105.
[Received 7/9/02; revised 10/12/02]
Joaquim Camps
Department of Romance Languages and Literatures
University of Florida
170 Dauer Hall, Box 117405
Gainesville, Florida 32611-7405
USA
e-mail: [email protected]fl.edu
© Blackwell Publishing Ltd. 2003
CONCURRENT
AND
RETROSPECTIVE VERBAL REPORTS
221
Appendix
Text and multiple-choice task.
Read the text and circle the letter of the items below that would better fill in
the blank in every case.
María tiene mucho trabajo en casa los sábados por la mañana. Primero lava
____(1) y luego la plancha. Después arregla ____(2) porque lo quiere tener en
orden. Ordena ____(3) que tiene en su escritorio y los pone en montones distintos.
Recoge ____(4) del mes pasado que tiene por el suelo y las lleva a reciclar.
Limpia ____(5) que tiene en su escritorio y la cambia de lugar. Al mediodía
almuerza. Toma ____ (6) y lo pone en la mesa de la cocina. Saca ____(7) que
tiene en el refrigerador, lo pone en el plato, y prepara un sándwich. Toma ____
(8) y la pone en el sándwich. También pone tomate y jamón. A María le gusta
comprar ____(9) que hay en el mercado del barrio, porque los venden a buen
precio. Toma también ____(10) y las pone al lado del sándwich. Entonces va a
la sala de estar, toma ____(11) que está al lado de la mesa, la coloca al lado de
la ventana, y se sienta. Mira ____(12) en el balcón, los llama y les da migas de
pan. Después llega la hora de ____(13). María las mira todos los días en el canal
4 de televisión. Se sienta delante de ____(14) que le regaló su madre, y lo
enciende. Le gusta mucho ver a ____ (15) tan importantes que actúan en esos
programas y las admira por su belleza. También le parecen muy guapos ____
(16), y los contempla con emoción. En fin, María pasa una tarde muy agradable
los sábados.
(1)
a. los pantalones
b. la cara
c. la ropa
(5)
a. las fotos
b. la lámpara
c. la mesa
(9)
a. las manzanas
b. los tomates
c. los cuchillos
(13)
a. las tareas
b. las telenovelas
c. los noticieros
(2)
a. la habitación
b. el cuarto
c. el almuerzo
(6)
a. una botella
b. una taza
c. un plato
(10)
a. unas bananas
b. unas cucharas
c. un tenedor
(14)
a. la radio
b. el periódico
c. el televisor
(3)
a. las cosas
b. los teléfonos
c. los libros
(7)
a. la mantequilla
b. el vino
c. el pan
(11)
a. una revista
b. una silla
c. un sillón
(15)
a. las sirvientas
b. las actrices
c. los actores
(4)
a. las revistas
b. los periódicos
c. las blusas
(8)
a. (la) lechuga
b. (la) cerveza
c. (el) atún
(12)
a. los pájaros
b. los dibujos
c. las golondrinas
(16)
a. los actores
b. los libros
c. las actrices
© Blackwell Publishing Ltd. 2003