International Journal of Psychophysiology 76 (2010) 1–8
Contents lists available at ScienceDirect
International Journal of Psychophysiology
j o u r n a l h o m e p a g e : w w w. e l s e v i e r. c o m / l o c a t e / i j p s y c h o
I spy with my little eye: Detection of temporal violations in event sequences and the
pupillary response
Susanne Raisig ⁎, Tinka Welke, Herbert Hagendorf, Elke van der Meer
Department of Psychology, Humboldt-Universität zu Berlin, Unter den Linden 6, 10099 Berlin, Germany
a r t i c l e
i n f o
Article history:
Received 25 June 2009
Received in revised form 11 January 2010
Accepted 12 January 2010
Available online 20 January 2010
Keywords:
Event knowledge
Predictions
Temporal violation
Conflict-monitoring
Pupillary response
Locus coeruleus
Anterior cingulate cortex
a b s t r a c t
Scripts of familiar activities store the temporal order of events. This enables us to generate predictions about
which event will follow another. When an event does not unfold in the chronological order, a mismatch
arises between the predictions and the external sensory input which is perceived as a conflict. The detection
of this mismatch is accomplished by a comparison mechanism (Zacks et al., 2007; Barsalou, 2009). We have
applied pupillometry to investigate the nature of this comparison process. We further tested for individual
differences in the efficiency of the mismatch detection. Participants were presented the title of an event
sequence to trigger predictions about the order in which events would unfold. Subsequently, three script
events were presented one at a time. The events either unfolded in the correct chronological order or
included temporal violations at different points within the event triplet. Violations of the temporal order had
to be detected. As soon as it was detected, the trial had to be terminated. We found that a temporal violation
elicited a large pupillary response in all individuals indicating that the comparison between the predictions
and the external sensory input was accomplished online and worked equally well for all individuals.
However, not all individuals terminated the trial after having detected the violation. Results showed that
efficient individuals who responded adequately had a greater pupillary response than inefficient individuals
suggesting that they invested more cognitive resources. The results are discussed in light of theories of
behavioral performance and conflict-monitoring.
© 2010 Elsevier B.V. All rights reserved.
1. Introduction
The ability to predict and foresee is a skill that often fails on us. The
weather forecast is probably the most prominent example of
predictions going wrong. However, every one of us is excellent at
predicting and foreseeing certain events that will occur in the near
future. We do it all day, every day. For example, when I reach out my
hand to grasp the cup standing on the table, most are able to predict
that I will move the cup to my mouth in order to drink. Crucial for
predictions is the knowledge about the temporal order in which
events unfold over time. Like in the example above, I can foretell
someone's actions based on my event-based knowledge that after
lifting a cup, it is usually brought to one's mouth. Similarly, predictions
are formed about events in complex event sequences. For example,
when I enter a restaurant I can predict that a waiter will show me to a
table: next, he will give me a menu, then he will ask me what I would
like to eat, and so forth. This knowledge is stored in scripts in
long-term memory (Schank and Abelson, 1977). The script structure
has been compared to an event chain that preserves the correct,
chronological temporal order of events, which enables accurate predic-
⁎ Corresponding author. Tel.: + 49 30 2093 9382; fax: + 49 30 2093 9361.
E-mail address: [email protected] (S. Raisig).
0167-8760/$ – see front matter © 2010 Elsevier B.V. All rights reserved.
doi:10.1016/j.ijpsycho.2010.01.006
tions about future events through priming. In the restaurant script, the
event order food primes the following event eat food, which in turn primes
the event pay the bill. In this way, all component events of the sequence
can be retrieved and future-oriented predictions are possible. The study
presented here focuses on these script-based predictions and investigates
cases when predictions are not fulfilled. This happens, for instance, when
an event does not unfold in the correct, chronological order but rather
violates the temporal order. Naturally, such an event will be unpredicted
and unexpected. Consider the following example: in a restaurant, I sit
down at a table and expect to be handed a menu by the waiter. After
having read the menu, I predict that the waiter will ask for my order,
which I will receive after a certain amount of time. If, however, I only sit
down at a table after having received my food, my predictions clearly do
not match the event I am currently encountering.
It is crucial that violations of the temporal order are detected in
order to adjust behavior or re-analyze the situation (in the above case,
for example, one would come to the conclusion after a re-analysis that
one is in a fast-food restaurant where the event sit down at table takes
place after the event receive food). Therefore, it is necessary to
monitor whether anything unexpected and unpredicted has occurred.
An ongoing event must be continuously compared to the predictions
to identify mismatches between them. Zacks et al. (2007) suggest that
upon perceiving an event, a mental representation is formed in
working memory that depicts what is currently going on. They call
2
S. Raisig et al. / International Journal of Psychophysiology 76 (2010) 1–8
this representation an event model. Apart from the sensory input,
predictions are also integrated top-down into the event model from
script structures in long-term memory. When the sensory input (i.e.,
the unfolding event) does not match the predictions within the event
model, an “error detection mechanism” becomes active. According to
Barsalou (1999), who adopts a somewhat different approach, perceptual and motor aspects of an experience are stored as perceptual
symbols in modality-specific areas of the brain. Upon knowledge
retrieval from memory, the necessary aspects of the experience are
simulated or re-enacted which is assumed to take place in working
memory. These simulations produce predictions that are then
compared to the sensory input when events occur (Barsalou, 2009).
Thereby, it is assessed whether events unfold as predicted or not.
Despite the differences, both accounts assume that the comparison
between the external sensory input and the predictions that have
been derived from the internal mental representation is based on an
online comparison process that constantly monitors for mismatches
between predictions and what is actually taking place.
Unpredicted and unexpected events as well as mismatches have been
shown to lead to specific psychophysiological changes, for example,
changes in the pupillary response. In an auditory oddball task where a
target tone had to be detected among distracters, Steinhauer and Zubin
(1982) found an inverse relationship between stimulus probability and
pupil size: the largest pupillary response was evoked by the rare,
unpredictable stimulus, while the highly predictable, frequent stimulus
evoked the least change in pupil size. Analogous results were found by
Gilzenrat, Nieuwenhuis, Jepma, and Cohen (in press). Correctly detected
target tones that occurred only in 20% of the cases elicited significantly
larger pupil dilations than correctly ignored distracter tones that were
highly frequent. Mismatches lead to similar effects in the pupillary
response as unpredicted events: in the Stroop Color–Word task, a
response conflict emerges in incongruent trials because of the mismatch
between the color word and the ink color. Apart from increasing response
times, high conflict trials also elicit greater changes in pupil size than nonconflict, congruent trials (Brown et al., 1999; Siegle et al., 2008) because
the former require additional cognitive resources for detecting, processing and resolving the conflict. This interpretation is consistent with the
assumption that the pupil accurately and reliably reflects cognitive
resource consumption. Research has demonstrated repeatedly that pupil
dilation is positively correlated with task difficulty, that is, it increases
when more cognitive resources are required (e.g., Hess and Polt, 1964;
Beatty, 1982; Kahneman, 1973; Just et al., 2003). The pupillary response
has the advantage that it is an online measure depicting the cognitive
resource consumption during the complete processing stage prior to
response. Therefore, it is very well suited to index the moment-tomoment changes in resource consumption.
The results described above provide evidence that unpredicted
events, mismatches and the conflicts they impose require additional
cognitive resources which can be measured with the pupillary
response. The present study applies this measure to unpredicted
events in event sequences in order to investigate the “error detection
mechanism” that compares the external sensory input and the
predictions that have been derived from the internal mental
representation (cf., Zacks et al., 2007; Barsalou, 2009). In a previous
study, we presented the title of an event sequence followed by three
component events that were presented simultaneously on the
computer screen. Participants had to detect errors in the temporal
order and had to decide whether the temporal order of the triplet was
correct or not (Raisig et al., 2007). Results suggested that each event
was indeed compared to the underlying mental representation.
However, due to the mode of presentation we could not specify the
exact nature of this comparison process. In order to investigate the
comparison process in the present study, the title of the event
sequence (e.g., going to a restaurant) was presented first to activate
the component events in long-term memory as well as their temporal
order and to produce predictions about which event will succeed
another. Subsequently, the three component events were presented
sequentially. This mimicked a continuous flow of events and each
event could be compared to the predictions that had been formed. In
an error detection task participants were asked whether the order of a
presented event triplet was correct or whether there was an error in
the temporal order (i.e., a temporal violation). For example, the three
events (A) enter the restaurant–(B) sit down at table–(C) read the
menu constituted the event sequence going to a restaurant. The
pupillary response was recorded continuously during processing to
depict the changes in cognitive resource consumption that were
associated with the event that was presented. In condition correct/
correct, events unfolded in the chronological order as stored in
long-term memory and therefore predictions were satisfied (see
Table 1). However, when events did not unfold in the chronological
order but rather violated the temporal order, a mismatch occurred.
These temporal violations could occur at different points of the event
triplet (see Table 1).
We assume that the detection of the temporal violation evokes a
significant increase in the pupillary response at the moment it occurs.
More specifically, in conditions false/correct and false/false, where the
temporal violation occurs upon presentation of the second event, the
task-evoked pupillary response should be greater than in conditions
correct/correct and correct/false, where there is no temporal violation
at this point. Upon presentation of the third event, the pupillary
response should be greater in condition correct/false than in condition
correct/correct because of the temporal violation that occurs.
Another important difference to our previous study (Raisig et al., 2007)
is that participants were encouraged to abbreviate a trial as soon as
possible, that is, as soon as they detected a temporal violation. Hence,
conditions false/correct and false/false should be abbreviated after
presentation of the second event. However, because we were using
event triplets we were able to compare cases where the temporal
violation was detected successfully with cases where the task was
continued despite a temporal violation. Specifically, if detection of the
temporal violation was accurate, conditions false/correct and false/false
should be terminated after the second event by an overt behavioral
response. We call these cases false/correct (abbreviated) and false/false
(abbreviated). Because a behavioral response terminated a trial, the
abbreviated conditions will naturally have shorter reaction times. We
assume that performance in the error detection task will be subject to
individual differences. That is, there will be individuals who show good
error detection in that they will efficiently abbreviate most, if not all, of the
possible trials. And there will be bad error detectors who will not
abbreviate the conditions false/correct and false/false. There are two
possible explanations to account for bad error detection: either the
temporal violation is simply not detected due to an inadequate
comparison process. Or the temporal violation is detected but participants
fail to inhibit further processing. With the pupillary response we
attempted to adjudicate between these possibilities. In the former case,
Table 1
Conditions and their temporal violations.
Condition
Triplet
Temporal violations
Correct/correct
A–B–C
Correct/false
A–C–B or
B–C–A
False/correct
B–A–C or
C–A–B
False/false
C–B–A
No temporal violation: the events unfold in
the correct temporal order.
Late temporal violation: the second event fulfills
the predictions; the third event violates
the temporal order and leads to a mismatch
between the predictions and the sensory input.
Early temporal violation: the second event violates
the temporal order and leads to a mismatch
between the predictions and the sensory input; the
third event is chronologically related to the second.
Early temporal violation: the second event violates
the temporal order and leads to a mismatch
between the predictions and the sensory input;
the third event again violates the temporal order.
S. Raisig et al. / International Journal of Psychophysiology 76 (2010) 1–8
the temporal violation should not evoke a pupillary response. If, however,
the second event elicits a large pupillary response but the task is
nevertheless continued, the latter explanation would be supported.
No matter which explanation holds, however, results from the
literature suggest that the individual differences in our task may arise
due to the amount of cognitive resources that are invested into the task.
Van der Meer, Beyer, Horn, Foth, Bornemann, Ries et al. (2010)
compared highly intelligent students (IQ > 120) with students of
average intelligence. The highly intelligent students showed better
task performance accompanied by greater pupil dilation. High-IQ
individuals seemed to dedicate more cognitive resources to the
processing of a high-level cognitive task than their peers as seen in a
greater pupillary response. This result suggests that the pupillary
response can discriminate between individuals' resource consumption.
Therefore, we applied pupillometry to investigate individual differences
in the detection of temporal violations in event sequences. If individual
differences are accounted for by the amount of cognitive resources that
are dedicated to the task, we expect that the pupillary response will be
greater in individuals who show good task performance than in
individuals how do not respond efficiently to the temporal violation.
2. Method
2.1. Participants
Thirty psychology students of Humboldt-Universität zu Berlin
participated in the experiment. They received either course credit or
€10 for participation. All were German native speakers. Seven
participants had to be excluded from the analysis of the pupil data
due to extensive blinking artifacts or other technical problems so that
23 participants remained in the analyses (12 female; mean age: 24.9,
SD = 6.4).
2.2. Stimuli and design
Event triplets consisted of three component events from event
sequences that had been produced and standardized in a script
generation task (Raisig et al., 2009). The events were arranged to
produce four conditions with temporal violations at different points
within the triplet. Table 1 shows the possible arrangements. In
condition correct/correct, events were always arranged in the form of
A–B–C. In condition correct/false events can be arranged either A–C–B
or B–C–A. In condition false/correct there are also two possibilities,
namely, B–A–C or C–A–B. Condition false/false again has only one
possible arrangement, which is C–B–A.
The main variable that we manipulated in a repeated measures
design was the temporal order of events. Fifty percent of the triplets
adhered to the correct temporal order so that the second and third event
fulfilled the script-based predictions when they were presented. Of the
chronologically triplets, 30 were targets while 60 triplets were filler
items. The filler items did not come from the pool of standardized events
sequences. Fillers were designed in a fashion similar to that of correct
triplet except that completely different event sequences were used.
Because fillers were not derived from the standardized pool of events,
they were not included in any analyses. The remaining 50% of the
triplets had an incorrect temporal order. Of these, 30 included one late
temporal violation (condition correct/false), 30 included one early
temporal violation (condition false/correct) and 30 were in the inverse
temporal order, that is, they included two temporal violations (condition
false/false). Reaction times (in milliseconds) and error rates were
recorded as indices of response latency and accuracy, and the pupillary
response (in millimeter deviations from a baseline) was continuously
recorded over the whole processing period as an index of cognitive
resource consumption.
3
2.3. Apparatus
The pupil size of the right eye was recorded with the iView X
Hi-speed system (SensoMotoric Instruments) with a sampling rate of
240 Hz and accuracy above .03 mm. The participant's head was placed
onto a chin rest above which the camera was positioned so that the eye
could be tracked continuously during the experiment. The experiment
was programmed using Presentation for Windows Version 12.1, which
was compatible with the iView X system. Presentation sent triggers
marking the on- and offsets of stimuli as well as participants' response.
Participants responded with the left and right control key on a standard
computer keyboard. Answer-response key assignment was randomized
between participants. Illumination in the lab was kept constant for all
testing sessions (luminance ≈ 300 lx (lux)).
2.4. Procedure
Participants were seated at the table and the camera was prepared
for recording. On the computer screen instructions appeared
explaining the task and asking for quick but accurate response.
Participants were explicitly encouraged to respond as soon as a
decision concerning the correctness of the event triplet could be made
by pressing either the right or left control key (assignment of yes and
no response to the left or right control key was varied between
participants). Practice trials followed in which participants were
informed whether their response was correct or not. After the practice
trials the experimental block began. Each trial started with a fixation
cross in the middle of the screen lasting for 2000 ms. Afterwards the
name of an event sequence appeared for 1000 ms. When it
disappeared, the first event appeared on left of the centre of the
screen. After 1500 ms, the second event appeared in the middle of the
screen next to the first one. If a temporal violation occurred at this
point a decision was possible and participants were instructed to
terminate the task by pressing the respective control key. If no
temporal violation occurred at this point and/or no response was
given within a 1500 ms time window, the trial continued and the
third event was presented on the right to the centre of the screen. The
third event remained on the screen until a response had been made.
After response (whether after the second event or third event) a mask
consisting of Xs replaced the areas on the screen where an event had
been. The mask was presented for 2000 ms and was used as a
refraction period for the pupil. Reaction times and pupil size were
recorded from the presentation of the name of the event sequence
onwards.
2.5. Data analysis
Prior to data analysis conditions were recoded. Conditions false/
correct and false/false were recoded as false/correct (abbr.) and false/
false (abbr.), respectively, when the trial was abbreviated within the
1500 ms time window after presentation of the second event. When a
response was given only after the third event, the conditions kept their
original label of false/correct and false/false, respectively. It could
happen that the third event was presented just at the moment when the
participant was about to press the response key. Such cases were
identified and all reactions to conditions false/correct and false/false
within a 300 ms time window after appearance of the third event were
also coded as false/correct (abbr.) and false/false (abbr.), respectively.
We chose this time window because we consider this time window to
be too short to read the event and embed it into a situational context
(cf., Barsalou et al., 2008). Because of this re-coding we ended up with
6 conditions (without fillers).
Erroneous responses were excluded from all the analyses of reaction
times and pupil data. Reaction times that were greater than +2 SD or
less than −2 SD from the mean were identified and replaced by the
mean reaction time ±2 SD for each condition and participant. The pupil
4
S. Raisig et al. / International Journal of Psychophysiology 76 (2010) 1–8
curve was smoothed using a computer algorithm developed in-house
(box filter-technique). Missing data in the curve due to blinks (sudden
large changes in pupil diameter) were corrected by interpolation. All
data correction was done with Matlab Version 7.4. 13% of trials
(including the filler condition) had to be discarded altogether due to
large artifacts that could not be interpolated. The pupil trace was
averaged stimulus-locked to the presentation of the name of the event
sequence. The pupil size within different time windows was determined
for each participant and compared between conditions to index the
amount of cognitive resources at different processing stages. The time
windows will be specified in the Results section. The pupil size was
expressed as a millimeter deviation from the baseline that was calculated as the mean pupil size during a 200 ms time window prior to
presentation of the activity name (baseline correction).
Reaction times, error rates and pupil size of conditions correct/
correct, correct/false, and the non-abbreviated conditions false/
correct and false/false were submitted to an analysis of variance
(ANOVA). To identify differences between conditions we calculated
repeated contrasts where each condition was compared to the
previous condition. Effect sizes of focused comparisons are indicated
as r-values. The abbreviated conditions false/correct (abbr.) and false/
false (abbr.) were compared separately with a paired t-test. Both data
sets were normally distributed. The Greenhouse–Geisser correction
was applied to all repeated measures with more than one degree of
freedom. Significance level was set at p < .05. Effect sizes are indicated
as r-values for each repeated contrast.
Participants were grouped as efficient or inefficient responders,
respectively, according to their performance in the detection task.
Participants who in conditions false/correct and false/false majorly
responded after the second event (i.e., in more than 70% of the cases)
were classified as efficient responders (n = 7). Participants who majorly
read all three events were classified as inefficient responders (n = 7).
Only those participants were grouped who could be classified clearly as
either one responder type or another. To test whether responders
differed regarding the amount of cognitive resources they dedicated to
the task, the pupil sizes in each condition were compared. Further, to
test whether efficient responders were not only more efficient but also
faster and more accurate in all conditions, reaction times and error rates
were compared between responder groups.
3. Results
3.1. Detection of temporal violations
Table 2 shows the mean reaction times, error rates and pupil sizes
for each condition as well as the total number of curves that were used
to compute the mean pupil size.
3.1.1. Reaction times
There was a main effect of condition (F(3, 66) = 11.13, p < .001). The
longest reaction time occurred in condition correct/false (see Table 2).
Repeated contrasts revealed that it was significantly greater than the
reaction time in condition correct/correct (F(1, 22) = 70.35, p < .01,
r = .9) and the non-abbreviated condition false/correct (F(1, 22) = 25.5,
Table 2
Mean reaction times (in ms), error rates (in %), pupil sizes (in mm-deviation from
baseline) and number of curves in each condition.
Condition
RT
Errors
Pupil
Curves
Correct/correct
Correct/false
False/correct
False/false
False/correct (abbr.)
False/false (abbr.)
5282
5627
5246
5377
3795
3902
5.8
14.4
15.4
4.3
1.5
1.1
.26
.33
.31
.30
.36
.37
610
548
278
370
316
264
p < .01, r = .7). In addition, the difference between the non-abbreviated
conditions false/correct and false/false was significant (F(1, 22) = 4.63,
p < .05, r = .4).
Naturally reaction times were shorter for the abbreviated conditions
false/correct (abbr.) and false/false (abbr.) than for the non-abbreviated
conditions. Although these conditions were equal (i.e., both required a
pairwise comparison of temporally inverse events) the difference
between these conditions reached significance (t(22) = −2.1, p = .049).
3.1.2. Error rates
Table 2 shows the mean error rates in percent. There was a main
effect of condition (F(3, 66) = 9.03, p < .001). Most errors were made
in the condition correct/false and the non-abbreviated condition false/
correct. Repeated contrasts showed that condition correct/false
differed significantly from condition correct/correct (F(1, 22) = 19.5,
p < .001, r = .7) and the non-abbreviated condition false/correct
differed significantly from the non-abbreviated condition false/false
(F(1, 22) = 16.5, p < .01, r = .7). No significant difference was found
between the abbreviated conditions false/correct (abbr.) and false/
false (abbr.) (t(22) = .58).
3.1.3. Pupillary response
Fig. 1 shows the pupil trace over time beginning at the fixation
cross (time point − 1.00). It must be noted beforehand that the pupil
is a rather slow reacting parameter. Data by Beatty (1982) or Zimmer
(1984) have shown that the pupil only starts to show a task-evoked
response 300–500 ms after the stimulus.
Generally, the shape of the pupil curve reflected the course of the
trial. At time point zero when the title of the event sequence was
presented (vertical dashed line) the pupil size decreased reaching its
minimum size after 2 s. After presentation of the first event (first vertical
grey line) the pupil steadily began to increase in all conditions. The slope
was steepest for the conditions false/correct (abbr.) and false/false
(abbr.). After about 3 s the pupil reached a plateau that was highest for
the abbreviated conditions false/correct (abbr.) and false/false (abbr.).
Conditions correct/correct and correct/false reached the lowest
plateaus. After this plateau the pupil resumed to increase in response
to the presentation of the second event (at 2.5 s, vertical grey line). To
this point, the task consisted merely of an event pair that was either
chronologically related (no temporal violation) or not (temporal
violation). In conditions false/correct (abbr.) and false/false (abbr.)
that were terminated at this point, the pupil reached its final peak at
around 4 s ms. In the other conditions that had not been terminated at
this point, the pupil curve reached a second plateau (at about 4.8 s) that
was highest for the non-abbreviated conditions false/correct and false/
false. After this plateau the pupil began to increase again in response to
the presentation of the third event (at 4 s, vertical grey line). It reached
its final peak at about 5.5 s.
To test the assumption that an early temporal violation that occurred
upon presentation of the second event elicited an increased pupillary
response, we determined the pupil sizes within the time window 3.2
and 4.8 s (see Fig. 1 and Table 3). This time window was chosen on
grounds of the shape of the pupil curve: the presentation of the second
event elicited a pupillary response at about 3.2 s that lasted until the
second plateau at about 4.8 s was reached. Within this time window, we
considered only conditions that had not been abbreviated (i.e., conditions correct/correct, correct/false, and the non-abbreviated conditions false/correct, and false/false) because the pupillary response is not
superimposed by decision or motor responses in these conditions. We
collapsed the pupil sizes of conditions correct/correct and correct/false
since these conditions are equal at this moment within the trial (i.e.,
they include no temporal violation at this point). The same was done for
the non-abbreviated conditions false/correct and false/false that both
include a temporal violation at this point. The mean pupil sizes were
then submitted to a paired t-test. We found that the mean pupil size
differed significantly between conditions with a temporal violation and
S. Raisig et al. / International Journal of Psychophysiology 76 (2010) 1–8
5
Fig. 1. Mean pupillary response (in mm from baseline) for each condition. Time 0 indicates the presentation of the event sequence (dashed vertical line). Each grey vertical line
indicates presentation of the events.
conditions without a temporal violation (t(22) = −3.23, p < .001)
whereby the temporal violation evoked a significantly greater pupillary
response (Table 3). Within this time window, the pupil reached its final
peak in the abbreviated conditions false/correct (abbr.) and false/false
(abbr.) (see Fig. 1). Peak dilations between these conditions were not
significantly different (t(22)= −.19, p = .85).
When the third event was presented, the pupil increased further
until it reached its peak. The peak dilations of the conditions correct/
correct, correct/false, and the non-abbreviated conditions false/correct,
and false/false were compared. The main effect of condition failed to
reach significance (F(3, 66) = 2.2, p = .09). Repeated contrasts were
nevertheless performed. They showed that the pupil size of condition
correct/false was significantly greater than the pupil size of condition
correct/correct (F(1, 22) = 10.21, p < .01, r = .56). There was no
significant difference between the non-abbreviated conditions false/
correct and false/false regarding pupil diameter although a difference
had been detected in reaction times.
3.2.1. Reaction times
Efficient responders were always faster in conditions correct/
correct, correct/false and the non-abbreviated conditions false/
correct, and false/false. The main effect of responder group reached
significance (F(1, 12) = 5.99, p < .05). Like in the overall analysis the
condition correct/false produced the longest reaction times (see Table 4).
The condition by responder interaction effect was not significant
(F = 1.05). When comparing reaction times of the abbreviated conditions, we also found a significant difference between the responder
groups (F(1, 12) = 11.3, p < .05) but no difference between conditions
(F = 3.18).
3.2.2. Error rates
Efficient responders made more errors in trials when all three
events were read (conditions correct/correct, correct/false and the
non-abbreviated conditions false/correct, and false/false; see Table 4).
This difference reached significance (F(1, 12) = 7.7, p < .05). The condition by responder interaction effect was not significant (F = 1.24).
3.2. Individual differences
Table 4 shows the mean reaction times, error rates and pupil sizes
for each responder group.
Table 3
Pupil size (in mm-deviation from baseline) at plateau (3.2–4.8 s).
Conditions Temporal relation
of the event pair
1 and 2
3 and 4
Pupil size at plateau
All
Efficient
(n = 23) responders
(n = 7)
No temporal
.16
violation
Temporal violation .24
Table 4
Mean reaction times (in ms), error rates (in %) and peak pupil sizes (in mm-deviation
from baseline) between groups.
Condition
Responder
RT
Errors
Pupil
Correct/correct
Inefficient
Efficient
Inefficient
Efficient
Inefficient
Efficient
Inefficient
Efficient
Inefficient
Efficient
Inefficient
Efficient
5557
5003
6009
5286
5599
4813
5675
5059
3998
3686
4216
3688
3
9
10
24
13
19
6
5
2
1
1
1
.14
.28
.23
.39
.18
.45
.23
.42
.34
.39
.31
.41
Correct/false
False/correct
Inefficient
responders
(n = 7)
.19
.06
.36
.15
False/false
False/correct (abbr.)
False/false (abbr.)
6
S. Raisig et al. / International Journal of Psychophysiology 76 (2010) 1–8
When comparing the abbreviated conditions, we found no difference
between the responder groups (F = 1.6) and no interaction effect
(F = 1.1).
3.2.3. Pupillary response
We compared the mean pupil sizes within the time window 3.2 to
4.8 s between responder groups. As in the overall analysis, we collapsed the mean pupil sizes of conditions correct/correct and correct/
false and of the non-abbreviated false/correct and false/false,
respectively. A temporal violation evoked a greater increase in pupil
size (cf. Table 3). Further, the mean pupil size in this time window
differed between groups whereby efficient responders had a significantly greater pupil size than inefficient responders (F(1, 12) = 8.1,
p < .05). Within this time window, the pupil of the abbreviated
conditions reached its final peak (see Fig. 1). There was no significant
difference in the peak value between the two abbreviated conditions.
There was also no significant differences between the responder
groups (F = 1.24).
When the third event was presented, the pupil increased further
until it reached its peak. The peak dilations of the conditions correct/
correct, correct/false, and the non-abbreviated conditions false/correct
and false/false were compared. There was a main effect of responder
group (F(3, 36) = 7.40, p < .05) whereby the efficient responders always
reached a significantly greater pupil size than the inefficient responders
(see Table 4).
4. Discussion
The current experiment addressed questions concerning the
detection of temporal violations in event sequences that occur when a
non-chronological event is presented. Such an event does not match the
predictions that have been derived from the internal representation
(i.e., script) of the event sequence in long-term memory. We
investigated the comparison process that is thought to monitor for
such mismatches (Zacks et al., 2007; Barsalou, 2009) with regards to the
efficiency of error detection and individual differences. We found that
an event that did not match the predictions elicited a greater pupillary
response than an event that matched the predictions. Importantly, we
could show that the performance in the error detection task was subject
to individual differences. Participants who were good error detectors
and showed efficient task performance also had a greater pupillary
response. In the following we will first briefly discuss the reaction time
data. Subsequently, we will turn to the pupil data discussing how the
data contribute to our knowledge of how temporal violations in event
sequences are detected. Finally, we will venture to relate our findings to
current theories of performance and conflict-monitoring.
4.1. Reaction times
We could replicate the pattern of the reaction time data from our
previous study (Raisig et al., 2007) where the condition correct/false
also produced the longest reaction times. Although the condition
correct/correct also required reading and processing all three events it
has a processing benefit because the correct temporal order of the
events is adhered to and consequently predictions are fulfilled. Hence,
the temporal violation introduced by the third event in condition
correct/false does not satisfy the predictions and leads to the significant
increase in reaction times. The non-abbreviated conditions false/correct
and false/false show similar reaction times to condition correct/correct.
This had also been the case in our former study. We will return to this
point further down in the discussion.
One strikingly different result from our 2007 study was the meaningful difference in reaction times between the non-abbreviated conditions false/correct and false/false. Unexpectedly, we also found a
significant difference between the abbreviated conditions false/correct
(abbr.) and false/false (abbr.) in the current study. This result can be
explained by the distance effect that reaction times decrease when the
temporal distance between event increases (e.g., Nottenburg and
Shoben, 1980). Because the event pairs in the abbreviated condition
false/false (abbr.) always took the form of C–B it had a smaller temporal
distance than the event pairs in the condition false/correct (abbr.) that
had a great temporal distance (20 event pairs took the form C–A). The
greater temporal distance was confirmed by ratings of temporal distance that we obtained post-hoc from new participants. The distance
effect might also account for the difference in the non-abbreviated
conditions.
4.2. Changes in pupillary response due to temporal violations
We will first consider the conditions that were not abbreviated (i.e.,
correct/correct, correct/false, and the non-abbreviated conditions false/
correct and false/false): the waveform of the pupil showed that each event
elicited a change in pupil size regardless of whether the event introduced a
temporal violation or not. This is to be expected, however, because it
reflects the processing of incoming sensory information. Although an
increase in pupil size was apparent in all conditions, the extent of the
increase differed between conditions. In the non-abbreviated conditions
false/correct and false/false the pupil increased substantially upon
presentation of the second event due to the temporal violation it inflicted
as compared to the conditions correct/correct and correct/false where a
chronological event was presented and therefore predictions were
fulfilled at this point. As the trial continued and the third event was
presented, condition correct/false led to the greatest increase in pupil size
due to the temporal violation in this condition. The pupil sizes of the
non-abbreviated conditions false/correct and false/false were not significantly different from the chronological condition correct/correct. These
results allow some conclusions regarding the nature of the comparison
process. Because the pupil measures cognitive resource consumption
continuously throughout processing, it depicts online the moment-tomoment changes in cognitive effort that is related to the detection of
temporal violations. Taking into account that the pupillary response
slightly lags behind the presentation of stimuli (cf. Zimmer, 1984; Beatty,
1982), the pattern of the pupil waveform suggests that temporal
violations and the mismatches they constitute were detected the moment
they occurred. It can be concluded, therefore, that the representation of
the sensory input that depicts the actual state is rapidly and continuously
compared to the predicted state in working memory in order to react to
the mismatch — in our case by terminating the task, in more natural cases
by re-analyzing the situation or adapting one's behavior. Therefore, the
comparison process enables a rapid adaptation to guide ongoing
processing and optimize behavior (cf. Zacks et al., 2007).
The question that arises is how good the online comparison
process actually is? We found that many participants did not show a
behavioral reaction to the temporal violations and the mismatches
arising from them. Rather, they showed inefficient response behavior
and did not abbreviate the conditions false/correct and false/false
although this would have been possible. Other participants, however,
were efficient and abbreviated most trials of conditions false/correct
and false/false. Are the individual differences in the behavioral
performance due to the quality of the comparison process? The
pattern of the pupil curves and the results obtained from the first time
window (3.2–4.8 s) do not speak for such a conclusion. Instead results
suggest that the temporal violation and the mismatch it inflicted were
indeed detected by all individuals alike but only few individuals were
efficient in giving an overt response. Two findings are in favor of this
conclusion: first, in the overall as well as the group analysis the pupil
always increased in response to the temporal violation that occurred
upon presentation of the second event. This indicates that at least the
autonomic pupillary response to the temporal violation was adequate.
Second, in the non-abbreviated conditions false/correct and false/false
the pupil increased only minimally upon presentation of the third
event suggesting that the new sensory input was only processed
S. Raisig et al. / International Journal of Psychophysiology 76 (2010) 1–8
superficially, that is, without processing its temporal properties. The
finding that reaction times were similar between the correct
condition correct/correct and the non-abbreviated conditions false/
correct and false/false also speaks for superficial processing of the
third event in the non-abbreviated conditions. Therefore, it can be
concluded that inefficient responders failed to perform adequately
despite the detection of a temporal violation.
This conclusion leads to the question why some participants failed to
respond adequately? We assumed that individual differences in the error
detection task were accounted for by the amount of cognitive resources
that were dedicated to the task. In line with our assumption we found that
efficient participants had larger pupil sizes in the first time window and
reached greater peak dilations than inefficient responders. Because the
pupillary response is directly related to the amount of cognitive resource
consumption (cf., Just et al., 2003; van der Meer et al., 2010) we can
conclude that efficient participants indeed put more effort into the task
than inefficient responders. This can even be observed on item-level: in
the overall analysis including all participants, abbreviated trials can be
considered as reflecting efficient trials. When looking at the pupil
waveforms of these abbreviated trials, we find that they show a steeper
slope, that is a greater pupillary response, from presentation of the first
event onwards. This suggests that more cognitive effort was put into these
trials by all participants. Of course other factors may also influence how
well and efficient errors are detected and responded to. Our results,
however, show that efficiency in error detection is directly related to the
amount of cognitive resources that were dedicated to a task.
We have argued so far that efficient responders dedicated more
cognitive resources into the task. However, they nevertheless made more
errors than inefficient responders in the non-abbreviated conditions
correct/correct/, correct/false, false/correct, and false/false. When looking
at the distribution of errors among the conditions, we find that errors
occurred particularly in conditions correct/false and false/correct. These
conditions are known to produce increased error rates (Raisig et al., 2007).
In condition correct/false, for example, the temporal order of the first and
the second event is evaluated and no temporal violation is detected. This
increases the response tendency of accepting the item as being correct.
The third event that is presented introduces a temporal violation which is
perceived as mismatch or conflict, respectively. Apart from the conflict
between the representational level (i.e., prediction) and the perceptual
level (i.e., sensory input), an additional conflict occurs on the response
level: the initial response tendency of accepting the item must be
overridden and changed into a rejecting response resulting in longer
reaction times and more errors. It seems that efficient individuals who
were generally good at detecting and reacting to a conflict between the
representational and the perceptual levels were strongly affected by the
conflict on the response level. The same explanation might apply to the
non-abbreviated condition false/correct. However, only few items fell into
that condition in the efficient group because they abbreviated the majority
of items. Hence, the interpretation that efficient responders are especially
affected by the conflict on the response level would have to be tested in
order to gain stronger support for this claim.
4.3. Integration of the results into theories of performance and conflictmonitoring
The efficient responders in our task can be described as being highly
task-engaged: they dedicate many cognitive resources into the task as
seen in the strong task-evoked pupillary response, which ameliorates
their performance. Inefficient responders, on the contrary, show less task
engagement and a lesser task-evoked pupillary response. So far, our
results clearly link performance and task engagement to the task-evoked
pupillary response. Recent findings suggest that the task-evoked pupillary
response reflects the phasic activity of the locus coeruleus (LC), an area in
the brainstem that releases the neurotransmitter norepinephrine (NE)
(Gilzenrat et al., in press). Crucially, the LC–NE system has also been
related to behavioral performance. The framework of Aston-Jones and
7
Cohen (2005) posits that accurate performance and high task engagement
is accompanied by bursts of activation of the LC neurons in response to
motivationally significant stimuli (i.e., phasic mode). Although it is beyond
the scope of this paper to scrutinize the relationship between behavior
and LC activity, we want to relate our results to this framework because it
has been suggested that the LC may be critically involved in the
comparison process that is considered to underlie the detection of errors
in event sequences (Zacks et al., 2007). The close relation between the
behavioral and the psychophysiological levels that we have observed (i.e.,
efficient responding accompanied by increased pupillary response) seems
compatible with the assumption that task engagement and accuracy of
performance are related to LC activity. That is, because the pupil indexes
LC activity (Gilzenrat et al., in press) performance in our task can also be
related to LC phasic activity.
As outlined above, the task-evoked pupillary response is considered an
index of LC phasic activity. However, in the literature the pupillary
response is mostly related to the cognitive workload and the cognitive
resources that are consumed during a task (e.g., Just et al., 2003). We
found that a temporal violation elicited a large pupillary response. We
interpreted that the temporal violation was perceived as a mismatch or
conflict which increased the need for cognitive resources similar to the
response conflict in the Stroop task (e.g., Siegle et al., 2008). A brain area
repeatedly associated with conflict-monitoring and error detection is the
anterior cingulate cortex (ACC) (cf., Botvinick et al., 2001). It has been
suggested that the ACC becomes active when tasks are difficult. Conflicts
make a task cognitively demanding thereby indexing the demand for
cognitive resources and mental effort (Rushworth et al., 2004; Walton et
al., 2003). Hence, the ACC may be “importantly involved in linking mental
effort to the autonomic changes that typically accompany it” (Botvinick et
al., 2004, p. 545).
Considering our results in the light of the framework by Aston-Jones
and Cohen (2005) as well as ACC functioning (Botvinick et al., 2004) our
work is in line with work emerging in this field that assumes a connection
between ACC, LC and pupillometry. It has been speculated, for example,
that the LC system responds to ACC conflict detection (Cohen et al., 2000)
and neuroanatomic evidence suggests that the ACC has projections to the
LC (Rajkowski et al., 2000) where it is thought to influence LC activity
(Aston-Jones and Cohen, 2005). If indeed ACC projects signals to the LC the
strength of the activation of the ACC might influence the strength of the LC
phasic response and consequently directly determine behavioral performance. An imaging study by Hester, Fassbender, and Garavan (2004)
demonstrated that the degree of ACC activation was directly related to
performance in a conflict-monitoring task (Go/No-Go task). Inattentive
individuals (as measured by self-reported absentmindedness) displayed
reduced ACC activation and performed worse on the conflict-monitoring
task than individuals who described themselves as attentive and showed
higher ACC activation. It remains to be investigated whether high ACC
activation in attentive individuals is also accompanied by LC phasic
activity and a large pupillary response, respectively. This would lend some
support to the presumed ACC–LC–pupillometry connection and would
allow us to conclude that efficient individuals in our task had a strong LC
phasic activation due to strong ACC activation. However, the work in this
field is still in its infancy and therefore this interpretation of our results
remains speculative.
What we can conclude from our study without speculation is that
pupillometry is a useful measure in error detection and conflictmonitoring tasks because it depicts the moment-to-moment resource
consumption and therefore gives insight into the time course of the
error detection. More importantly, it allowed us to identify the source of
the individual differences that apparently were due to the amount of
cognitive resources that were dedicated to the task at hand.
Acknowledgements
We would like thank Roman Purkhart for laboratory assistance and
helpful discussion. This research was supported by the German Science
8
S. Raisig et al. / International Journal of Psychophysiology 76 (2010) 1–8
Foundation (DFG) grant ME1362-10 awarded to Elke van der Meer and
Herbert Hagendorf. We thank Sander Nieuwenhuis and an anonymous
reviewer for valuable comments on an earlier version of this paper.
References
Aston-Jones, G., Cohen, J.D., 2005. An integrative theory of locus coerules-norepinephrine
function: adaptive gain and optimal performance. Annual Review of Neuroscience 28
(1), 403–450.
Barsalou, L., 1999. Perceptual symbol systems. Behavioral and Brain Sciences 22, 577–660.
Barsalou, L.W., 2009. Simulation, situated conceptualization, and prediction. Philosophical
Transactions of the Royal Society B: Biological Sciences 364 (1521), 1281–1289.
Barsalou, L.W., Santos, A., Simmons, W.K., Wilson, C.D., 2008. Language and simulation
in conceptual processing. In: de Vega, M., Glenberg, A.M., Graesser, A.C. (Eds.),
Symbols, Embodiment, and Meaning. Oxford University Press, Oxford, pp. 245–284.
Beatty, J., 1982. Task-evoked pupillary responses, processing load, and the structure of
processing resources. Psychological Bulletin 91 (2), 276–292.
Botvinick, M.M., Braver, T.S., Barch, D.M., Carter, C.S., Cohen, J.D., 2001. Conflict monitoring
and cognitive control. Psychological Review 108 (3), 624–652.
Botvinick, M.M., Cohen, J.D., Carter, C.S., 2004. Conflict monitoring and anterior
cingulate cortex: an update. Trends in Cognitive Sciences 8 (12), 539–546.
Brown, G., Kindermann, S.S., Siegle, G.J., Granholm, E., Wong, E.C., Buxton, R.B., 1999. Brain
activation and pupil response during covert performance of the Stroop Color Word
task. Journal of the International Neuropsychological Society 5 (04), 308–319.
Cohen, J.D., Botvinick, M.M., Carter, C.S., 2000. Anterior cingulate and prefrontal cortex:
who's in control? Nature Neuroscience 3, 421–423.
Gilzenrat, M.S., Nieuwenhuis, S., Jepma, M., and Cohen, J.D. (in press). Pupil diameter
tracks changes in control state predicted by the adaptive gain theory of locus
coeruleus function. Cognitive, Affective, & Behavioral Neuroscience.
Hess, E.H., Polt, J.M., 1964. Pupil size in relation to mental activity during simple
problem-solving. Science 143, 1190–1192.
Hester, R., Fassbender, C., Garavan, H., 2004. Individual differences in error processing: a
review and reanalysis of three event-related fMRI studies using the GO/NOGO task.
Cerebral Cortex 14 (9), 986–994.
Just, M.A., Carpenter, P.A., Miyake, A., 2003. Neuroindices of cognitive workload:
neuroimaging, pupillometric and event-related potential studies of brain work.
Theoretical Issues in Ergonomics Science 4, 56–88.
Kahneman, D., 1973. Attention and Effort. Prentice Hall, Englewood Cliffs, NJ.
Nottenburg, G., Shoben, E.J., 1980. Scripts as linear orders. Journal of Experimental
Social Psychology 16 (4), 329–347.
Raisig, S., Welke, T., Hagendorf, H., van der Meer, E., 2007. Investigating dimensional
organization in scripts using the pupillary response. Psychophysiology 44 (6), 864–873.
Raisig, S., Welke, T., Hagendorf, H., van der Meer, E., 2009. Insights into knowledge
representation: the influence of amodal and perceptual variables on event
knowledge retrieval from memory. Cognitive Science 33, 1252–1266.
Rajkowski, J., Lu, W., Zhu, Y., Cohen, J.D., Aston-Jones, G., 2000. Prominent projections
from the anterior cingulate cortex to the locus coeruleus (LC) in rhesus monkey.
Society for Neuroscience Abstracts 26, 838.15.
Rushworth, M.F.S., Walton, M.E., Kennerley, S.W., Bannerman, D.M., 2004. Action sets and
decisions in the medial frontal cortex. Trends in Cognitive Sciences 8 (9), 410–417.
Schank, R.C., Abelson, R.P., 1977. Scripts, Plans, Goals and Understanding: Inquiry into
Human Knowledge Structures. Erlbaum, Hillsdale, NJ.
Siegle, G.J., Ichikawa, N., Steinhauer, S., 2008. Blink before and after you think: blinks
occur prior to and following cognitive load indexed by pupillary responses.
Psychophysiology 45 (5), 679–687.
Steinhauer, S.R., Zubin, J., 1982. Vulnerability to schizophrenia: Information processing
in the pupil and event-related potential. In: Usdin, E., Hanin, I. (Eds.), Biological
Markers in Psychiatry and Neurology. Pergamon Press, Oxford, pp. 371–385.
van der Meer, R., Beyer, R., Horn, J., Foth, M., Bornemann, B., Ries, J., et al., 2010. Resource
allocation and fluid intelligence: insights from pupillometry. Psychophysiology 47,
158–169.
Walton, M.E., Bannerman, D.M., Alterescu, K., Rushworth, M.F.S., 2003. Functional
specialization within medial frontal cortex of the anterior cingulate for evaluating
effort-related decisions. Journal of Neuroscience: The Official Journal of the Society
for Neuroscience 23 (16), 6475–6479.
Zacks, J.M., Speer, N.K., Swallow, K.M., Braver, T.S., Reynolds, J.R., 2007. Event perception: a
mind-brain perspective. Psychological Bulletin 133 (2), 273–293.
Zimmer, K., 1984. Ansätze der psychophysiologischen Indikation von Wissensrepräsentation:
Die Pupillomotorik als sensbiler Indikator semantischer Informationsverarbeitungsaktivität. In: Klix, F. (Ed.), Gedächtnis, Wissen, Wissensnutzung. VEB Deutscher Verlag der
Wissenschaft, Berlin, pp. 137–155.