VISUAL COGNITION, 2006, 14 (4/5/6/7/8), 704 715
Evidence for a systematic component within scan paths
in visual search
Iain D. Gilchrist
Department of Psychology, University of Bristol, UK
Monika Harvey
Department of Psychology, University of Glasgow, UK
We present evidence that scan paths in visual search can include a systematic
component. The task for subjects in the experiment was to search for a target that
was either present or absent. With regular grid-like displays, participants generated
more horizontal saccades than vertical saccades. Disruption of the grid structure in
the display modulated but did not eliminate the systematic component. This is
consistent with the scan path being partly determined by a cognitive strategy. We
discuss the implications of this finding for studies that use refixation to investigate
memory mechanisms in visual search.
Monitoring eye movements during visual search has become an important
method for investigating the processes that mediate between display onset
and the manual response (e.g., Findlay, 1997; Hooge & Erkelens, 1996, 1999;
Zelinsky & Sheinberg, 1997). Models of visual search and particularly ones
that include the programming of saccades, have tended to focus on bottom
up salience driven mechanisms for saccade allocation to items in the display
(e.g., Itti & Koch, 2000). At the centre of most of these models is a salience
map of one form or another. The salience of items is determined by the
strength of the visual input to the system and by the characteristics of the
target. So, for example if the target is a green vertical bar then green items
and vertical items will have enhanced salience. This is the mechanism by
which attention (Wolfe, 1994; Wolfe, Cave, & Franzel, 1989) or saccades
(Findlay & Walker, 1999; Itti & Koch, 2000; Wolfe & Gancarz, 1996) are
deployed to items that are more similar to the target. Over and above these
Please address all correspondence to I. D. Gilchrist, Department of Psychology, University
of Bristol, Bristol BS8 1TN, UK. E-mail: [email protected]
This work was supported by a grant from the EPSRC UK (grant no: GR/M37295). We
would like to thank Casimir Ludwig for help with data collection.
http://www.psypress.com/viscog
# 2006 Psychology Press Ltd
DOI: 10.1080/13506280500193719
SCAN PATHS IN VISUAL SEARCH
705
salience-based processes it is assumed that the location to be fixated next is
determined by a random process (Itti & Koch, 2000; Scinto, Pillalamarri, &
Karsh, 1986). However, four decades ago Williams (1966) suggested that
there was often a bias in the direction of saccades in search and that these
biases reflected a systematic process rather than the action of a random
process (see also Norton & Stark, 1971a, 1971b; Yarbus, 1967). This
observation is of course consistent with introspection. When search becomes
difficult we tend to start in one place and work systematically through the
display. In this paper we have used Williams’ (1966) term ‘‘systematic’’ to
describe this nonrandom component in the scan path. Although there have
been qualitative reports of a systematic component in scanning (e.g., Hooge
& Erkelens, 1996), it has not been studied quantitively. Indeed early attempts
to quantify scan paths in visual search lead to the conclusion that randomwalk models best characterized the allocation of fixations in search (Scinto
et al., 1986). More recently, Motter and Belky (1998) investigated the extent
to which the previous saccade influenced the characteristics of the
subsequent saccade. They found no evidence for either the length or the
direction of the previous saccade having a large influence on the subsequent
saccade, except for a slight bias for the next saccade to avoid the area just
crossed by the saccade. One of the reasons for the lack of evidence for
systematic scanning in visual search may be that although scanning could be
purely systematic, such a process will often operate alongside other saliencebased processes to determine which item is fixated next. If systematic
scanning is not the sole factor determining the structure of the scan path it
may not be easy to detect its presence. In this paper we report an experiment
in which we record eye movements in visual search. The purpose of this
report is to (1) describe a method by which systematic patterns in scan path
structure can be identified, (2) demonstrate that systematic scanning does
occur in visual search, and (3) show that the extent of this behaviour is
modulated by, but relatively robust to, substantial disruption of display
structure.
One of the difficulties of detecting the presence of systematic scanning
behaviour is developing methods by which it can be detected both across
trials and across participants and then analysed using statistical techniques.
Following Williams’ (1966) claim that there is often a directionality to
successive fixations, in the current experiment we analysed the frequency of
saccade directions. Analysis of saccade direction has the potential to reveal
systematic patterns, even when they are combined with other mechanisms to
determine the final scan path.
In the current experiment, across three conditions, we manipulated the
regularity of the display while keeping display size and the display elements
constant and analysed the frequency of saccades in each direction across
conditions.
706
GILCHRIST AND HARVEY
EXPERIMENT
In all three conditions the target was an upright triangle, and the distractor
items were downward pointing triangles and leftward pointing triangles. In
Condition 1 the items were completely regular, less regular in Condition 2,
and even less so in Condition 3.
Method
Participants. Twelve adults (9 female and 3 male, age range 1945 years)
took part in the study. All were from the University of Bristol and were paid,
or received course credit, for their time.
Design and procedure. The experiment was a repeated measures design,
with the order of condition presentation presented in separate blocks and
counterbalanced across the participants. Viewing distance was 57 cm. The
participants were instructed to make a present/absent response using a
button box.
At the beginning of each trial, a small white disc with a black dot in its
centre appeared in the middle of the screen. Online compensation for any
spatial offset in the calculated eye position was made at this point before
each trial.
The display was then presented until a manual response was made and,
when appropriate, this was followed by an error message presented centrally
on the screen. Participants were given verbal instructions describing the task
and asked to respond as quickly and accurately as possible.
Displays were presented on a 17-inch SVGA monitor with 800 /600 pixel
resolution. A chinrest was used to minimise head movements. In all
conditions the target was an upright triangle and the distractor items were
triangles pointing downwards and pointing to the left (see Figure 1), display
size was 25 items. In trials where the target was not present, the target was
replaced with one of the distractor types.
Each of the three blocks contained 98 trials. Participants were given a
break of approximately 3 minutes between blocks of trials. In Condition 1
the 25 items were placed randomly on the junctions of an imaginary 5 /5
grid resulting in a spatially regular display. In Condition 2 the same 25
display items were placed randomly on the junctions of an imaginary 6 /6
grid leaving 12 randomly selected locations blank in each trial. And in
Condition 3, the display items were placed randomly on the junctions of an
imaginary 7 /7 grid, leaving 24 randomly selected locations blank in each
trial. Across all three conditions the overall display size was kept constant
(12 /12 deg). As a result, Condition 1 was the most spatially structured
Figure 1. Example displays: Condition 1 (upper panel), Condition 2 (middle panel), and Condition 3
(lower panel).
707
708
GILCHRIST AND HARVEY
display and Condition 3 was the least structured display. Example displays
can be seen in Figure 1.
Eye movement recording and analysis. Two dimensional, binocular eye
movements were recorded using an SMI Eye-Link eyetracker (SensoMotoric
Instruments GmbH, Berlin, Germany). The Eye-Link system uses an
infrared video technique sampling at 250 Hz, and features a head movement
compensation mechanism. Displays were presented on one PC (subject PC),
while a second PC (operator PC) recorded the eye position data online. Each
block of trials was preceded by a nine-point calibration and validation
procedure.
The eye position data were analysed off line by an automatic saccade
detection procedure. For each participant, the data were analysed only from
the eye that produced the best spatial resolution, which in this experiment
was typically 0.20 deg.
A fixation was defined as having ended when the eye velocity exceeded
30 deg/s. A fixation began after the velocity fell below this value for five
successive samples (20 ms). This rather stringent criterion excludes the
interval of ocular instability just after the saccade and so leads to a more
accurate calculation of fixation location which is an important measure in
the current experiment. However, as a result the fixation durations reported
in this paper may be shorter (by approximately 15 ms) than those typically
reported elsewhere. The absolute direction of all the saccades larger than
1 degree was analysed. Saccades were coded for direction in degrees with
saccades in a vertical direction coded as zero. For each participant, in each
condition, the frequency of saccades in each direction in 10 degree frequency
bins was calculated. As the total number of saccades was large, even for a
single participant, these data could be analysed using standard parametric
tests. The percentage of fixations outside the display area was 3.3% in
Condition 1, 4.5% in Condition 2, and 6.1% in Condition 3. These fixations
were excluded for subsequent analyses.
Results
Errors. The percentages of manual response error trials per condition
were 6.72%, 7.14%, and 6.38%, for Conditions 1, 2, and 3 respectively. Most
errors occurred on target-present trials (13.3%) rather than target-absent
trials (0.68%). Error trials were excluded from later analyses.
Reaction times. Mean reaction times in the three conditions are shown
in Table 1. A repeated measures ANOVA was carried out on these data.
Reaction times were slower for target-absent (4665 ms) than for targetpresent (2549 ms), F (1, 11) /63.7, p B/ .001; but there was no reliable effect
SCAN PATHS IN VISUAL SEARCH
709
TABLE 1
The mean reaction times (in ms), fixation duration (in ms), and number of fixations in
the experiment; values are given for target-present and target-absent trials by
condition, with standard deviations in parentheses
Condition 1
Measure
Condition 2
Condition 3
Present
Absent
Present
Absent
Present
Absent
Mean reaction time (ms)
SD
2443
(755)
4596
(1663)
2752
(849)
4723
(1626)
2453
(650)
4676
(1604)
Mean fixation duration (ms)
SD
225
(29)
217
(23)
221
(22)
213
(20)
221
(21)
214
(19)
Mean number of fixations
SD
10.7
(3.0)
18.5
(6.2)
11.6
(3.2)
18.3
(5.9)
11.0
(2.8)
19.0
(6.10)
of Condition, F (2, 22) B/1. There was also no significant interaction between
condition and target presence, F (2, 22) B/1. Inspection of Table 1 revealed
that numerically there were only very small differences between the
conditions.
Fixation duration and fixation number. The total number of fixations
examined in each condition were 17,196 in Condition 1, 17,619 in Condition
2, and 17,635 in Condition 3. Mean fixation duration and the number of
fixations per trial are shown in Table 1. A repeated measures ANOVAs
revealed a significant effect of target presence on fixation duration, with
fixations in target present trials being longer (222 ms) than fixations in target
absent trials (215 ms), F (1, 11) /12.7, p B/ .01. However, there was no
significant effect of condition, F (2, 22) /2.15, p/ .140, nor a significant
interaction, F (2, 22) B/1. For the number of fixations per trial there was
again a significant effect of target presence in that there were, on average,
more fixations in the target absent trials (18.6 fixations) than the target
present trials (11.1 fixations), F (1, 11) /58.9, p B/ .001. There were no
reliable effects of condition, F (2, 22) B/1. The interaction did not reach
significance but was marginal, F(2, 22) /2.90, ns. p / .08. For fixation
duration and fixation number, there was no evidence for reliable differences
between conditions.
Saccade direction. The distribution of saccade directions in the three
conditions is shown in Figure 2.
A repeated measures ANOVA was carried out on saccade direction
frequency. There was no significant effect of condition, F (2, 22) B/1.
However, there was a significant effect of direction, F (35, 385) /15.5,
710
18
18
18
12
12
12
6
6
6
0
0
0
-6
-6
-6
-12
-12
-12
-18
-18
a
-12
-6
0
6
12
18
-18
-18
b
-12
-6
0
6
12
18
-18
-18
-12
-6
0
6
12
c
Figure 2. The distribution of the angle of movement of each saccade in Condition 1 (panel a), Condition 2 (panel b), and Condition 3 (panel c).
18
SCAN PATHS IN VISUAL SEARCH
711
p B/.001, indicating a reliable increase in the number of saccades generated in
a specific direction consistent with a systematic component in the scan paths.
In addition there was a significant interaction, F (70, 770) /5.75, p B/ .001.
Pairwise comparisons between each of the three conditions showed a
significant interaction in all cases: Conditions 1 and 2, F (35, 385) /4.56,
p B/ .001; Conditions 2 and 3, F (35, 385) /3.25, p B/ .001; Conditions 1 and
3, F (35, 385) /7.73, p B/ .001. These interactions reflect a decreasing
asymmetry in the saccade direction distributions from Conditions 1 to 3.
There was evidence for systematic scanning in all three conditions as
reflected in the main effect of direction. The systematic scanning effect took
the form of more horizontal saccades than other directions. In addition this
effect was strongest for the more structured displays. These effects on
saccade direction were not apparent from simple inspection of single scan
paths, and it certainly was not possible to differentiate the extent of these
effects between conditions by inspection alone.
Discussion
In the current experiment we found strong evidence for systematic scanning
in visual search. The method introduced allowed the detection of properties
of the scanning that were consistent across participants and across
conditions. The extent of systematic scanning was modulated by display
structure: As the display became less regular across conditions so the
asymmetries in the directions of the saccades produced was reduced.
However, there was even evidence for such asymmetries in Condition 3
where the regularity of the display had been significantly disrupted. One
possibility is that the asymmetries in the directions of the saccades produced
here simply reflect low-level oculomotor biases. For example, even the
organization of oculomotor muscles may lead to biases in the number of
saccades in each direction. However, here we have shown that the extent of
these asymmetries differs across different display conditions. If the effect was
a result of low-level oculomotor factors we would expect it to be constant
across condition. In a comparison between the conditions, we found no
reliable differences between reaction time and number of fixations. This
suggests that task difficulty was relatively well matched across conditions.
However, as the distribution of saccade direction shows, the manipulation of
display regularity had a systematic and reliable effect on the nature of the
scanning. The experiment also allows an important additional conclusion to
be drawn. Although systematic scanning was modulated by display
regularity, the results from Conditions 2 and 3 confirm that systematic
scanning occurs even for displays that are not perfectly regular. Indeed the
results suggest that these effects are relatively robust to quite large
712
GILCHRIST AND HARVEY
disruptions of the regularity of the structure as can be seen in Condition 3. If
the regularity of the display structure were reduced further the presence of
any detectable systematic behaviour may be further reduced and even
disappear. The current experiment did not investigate this change in detail.
Instead it demonstrated that systematic scanning does not depend solely
on a completely regular display and is not dependent solely on task difficulty.
However, most naturalist visual scenes do contain an extensive
complex spatial structure (Marr, 1982). In these more visually complex
conditions it may be that this structure shapes systematic scanning. The
form that the systematic scanning took in this experiment was almost
certainly a function of the type of structure that was imposed on the
displays: specifically the display items were placed randomly on the
intersections of a grid. However, previous work by Hooge and Erkelens
(1996) has shown that participants will systematically scan around circular
displays, suggesting that systematic scanning is not a behaviour restricted
only to grid based displays.
Memory processes have an influence on search behaviour at a number of
levels (Shore & Klein, 2000). However, one contentious issue is the extent to
which visual search relies on memory mechanisms to prevent items that have
been inspected from being reinspected. Estimates of the capacity for this
type of memory have ranged from no memory at all (Horowitz & Wolfe,
1998) to a limited memory capacity (Gilchrist & Harvey, 2000), and to a
more extensive memory capacity (Peterson, Kramer, Wang, Irwin, &
McCarley, 2001). One approach to this question has been to record eye
movements in order to get a more direct measure of where attention is
allocated at any one time. The key measure here has been the extent of
refixation of distractors. The logic is that when no refixation occurs then the
items have been remembered. A number of groups have modelled this kind
of data to obtain an estimate of the capacity of the memory store (Gilchrist
& Harvey, 2000; Peterson et al., 2001). This leads to two key, interrelated,
questions. The first is what does it mean when an item is refixated? And the
second is, what does is mean when an item is not refixated? The answer to
neither of these questions is straightforward. What does it mean when an
item is refixated? Refixating a distractor item in visual search appears to
suggest that the subject has forgotten that the item has been visited and so is
returning to reinspect it. However, as Gilchrist and Harvey (2000) pointed
out, refixations can also occur because the participant moved away from the
fixated item before the processing of that item was completed. The refixation
in this case simply reflects a return to the item to complete processing.
Indeed such deadline limiting of fixation duration is part of the influential
model of saccade control developed by Henderson (1992). What does it
mean when an item is not fixated? One possibility is that visual processing in
the periphery has ruled out that item as a possible target. This process of
SCAN PATHS IN VISUAL SEARCH
713
visual guidance is an important determinant of saccadic selection in search
(e.g., Findlay, 1997). However, a number of researches in this field have ruled
out guidance by using display items that are very small, or very similar to
that target (e.g., Peterson et al., 2001), so that it is impossible to distinguish
the target from distractors without fixating. It would appear then that if the
stimuli in the search task are correctly selected, an item not being fixated is
evidence for memory. However, the story may not be that simple. The
evidence presented here suggests that saccades in visual search can be
systematic. If the saccades generated by participants are not random, but
instead have a systematic component, then this itself can reduce refixations.
In the most extreme case, where participants follow a fixed route thorough
the display no refixations will occur. The lack of refixations would not be a
product of remembering specifically which items had been visited but instead
would reflect the consequence of following a predictable route. If such a
mechanism does influence scan paths then a lack of refixations in a task may
not be a hallmark of memory in search. In turn this suggests that when the
display has a structure that can drive systematic scanning it is possible that
systematic scanning can substitute for memory. If participants follow
broadly the same scan path on each trial this would allow them to eliminate
part of the display simply because it occurred earlier in the search. In the
current experiment there is a decrease in the extent of strategic scanning
across conditions. If the above argument is correct, then we would expect a
concurrent increase in task difficulty because participants were less able to
rely on systematic scanning to support efficient search. This increase in task
difficulty should be reflected in an increase in the number of fixations and an
increase in the overall response time: We found neither in the current
experiment. There are a number of possible explanations for this apparent
anomaly. Various authors have suggested that memory capacity in search is
quite large (e.g., Peterson et al., 2001); however, the deployment of this
memory might in itself require effort (Gilchrist, North, & Hood, 2001). In
the current experiment what may have occurred is a tradeoff between
memory deployment and systematic behaviour resulting in approximately
equivalent overall search performance. Another possibly is that, in the less
regular displays, participants were still engaging in the same amount of
systematic scanning but that the particular form of scanning adopted varied
either across subjects or even trial-to-trial. If the form of the systematic
scanning is inconsistent then the method presented in this paper is unable to
detect its presence*instead the method presented here detects systematic
scanning that is consistent across trials and participants. Further work
would clearly be needed to disentangle these possibilities. However, whatever
the explanation, the current experiment demonstrates the presence of
systematic scanning in search. The presence of systematic scanning does
not rule out memory processes as being important in search. However, what
714
GILCHRIST AND HARVEY
is clear is that a number of mechanisms appear to structure scan paths in
more difficult visual search. Understanding the relationships between these
mechanisms and developing methods to detect and quantify their effects is
an important step in understanding the search process. As a step towards
this, the present paper presents a method for detecting and statistically
analysing systematic components in scan paths.
REFERENCES
Findlay, J. M. (1997). Saccade target selection during visual search. Vision Research , 37 , 617 631.
Findlay, J. M., & Walker, R. (1999). A model of saccade generation based on parallel processing
and competitive inhibition. Behavioral and Brain Science, 22 , 661 721.
Gilchrist, I. D., & Harvey, M. (2000). Refixation frequency and memory mechanisms in visual
search. Current Biology, 10 , 1209 1212.
Gilchrist, I. D., North, A., & Hood, B. (2001). Is visual search really like foraging? Perception ,
30 , 1459 1464.
Henderson, J. M. (1992). Visual attention and eye movement control during reading and picture
viewing. In K. Rayner (Ed.), Eye movements and visual cognition (pp. 261 283). Berlin:
Springer-Verlag.
Hooge, L. T. C., & Erkelens, C. J. (1996). Control of fixation duration in a simple search task.
Perception and Psychophysics, 58 , 969 976.
Hooge, I. T. C., & Erkelens, C. J. (1999). Peripheral vision and oculomotor control during visual
search. Vision Research , 39 , 1567 1575.
Horowitz, T. S., & Wolfe, J. M. (1998). Visual search has no memory. Nature, 394 , 575 577.
Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of
visual attention. Vision Research , 40 , 1489 1506.
Marr, D. (1982). Vision . San Francisco: W. H. Freeman.
Motter, B. C., & Belky, E. J. (1998). The guidance of eye movements during active visual search.
Vision Research , 38 , 1805 1815.
Norton, D., & Stark, L. (1971a). Eye movements in visual perception. Scientific American , 224 ,
34 43.
Norton, D., & Stark, L. (1971b). Scanpaths in saccadic eye movements while viewing and
recognising patterns. Vision Research , 11 , 929 942.
Peterson, M. S., Kramer, A. F., Wang, R. X. F., Irwin, D. E., & McCarley, J. S. (2001). Visual
search has memory. Psychological Science, 12 , 287 292.
Scinto, L. F. M., Pillalamarri, R., & Karsh, R. (1986). Cognitive strategies for visual search.
Acta Psychologica , 62 , 263 292.
Shore, D. I., & Klein, R. M. (2000). On the manifestations of memory in visual search. Spatial
Vision , 14 , 59 75.
Williams, L. G. (1966). The effect of target specification on objects fixated during visual search.
Perception and Psychophysics, 1 , 315 318.
Wolfe, J. M. (1994). Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin
and Review, 1 , 202 238.
Wolfe, J. M., Cave, K. R., & Franzel, S. L. (1989). Guided Search: An alternative to the feature
integration model for visual search. Journal of Experimental Psychology: Human Perception
and Performance, 15 , 419 433.
SCAN PATHS IN VISUAL SEARCH
715
Wolfe, J. M., & Gancarz, G. (1996). Guided Search 3.0: A model of visual search catches up with
Jay Enoch 40 years later. In V. Lakshminarayanan (Ed.), Basic and clinical applications of
vision science (pp. 189 192). Berkeley, CA: Kluwer Academic Publishers.
Yarbus, A. L. (1967). Eye movements and vision . New York: Plenum Press.
Zelinsky, G. J., & Sheinberg, D. L. (1997). Eye movements during parallel-serial visual search.
Journal of Experimental Psychology: Human Perception and Performance, 23 , 244 262.