On the relationship between eye-movements and activation:
Active vs. passive tasks during ambiguous pronoun resolution
Eiling
1University
Yee1,
Daphna
of Pennsylvania,
Heller2
2University
Background
of Toronto, 3Brown University
• “Automatic”:
• Activation of referent drives shift in attention to it
• Goal-directed:
• To guide action (e.g., hand movement)
• To verify interpretation: Visual information provides feedback to support interpretation
Predictions for task differences:
• “Automatic” looking will occur regardless of task
• Goal-directed looking:
• Action-based tasks (“pick up”, etc)
Zebra was
planting a
vegetable garden
with Elephant.
He carefully
watered the young
plants
Moderate:
Zebra and
Elephant were
planting a
vegetable garden.
Zebra wanted
Elephant to water
the young plants.
Weak:
• Positive relationship ought to hold because eyes guide hand movements
• Passive listening tasks (pronouns do not easily lend themselves to active tasks)
• If looking used to verify interpretation, it is unnecessary when verification unnecessary
• So, looking may increase as a function of activation up to a point, but when referent
already highly active, it will decrease.
The animals were
planting a
vegetable garden.
Zebra and
Elephant watered
the young plants.
How do goals influence the relationship between allocation of
visual attention and referential certainty?
%Zebra - %Elephant
in analysis window
%Zebra - %Elephant
in analysis window
*
*
weak mod strong
average verb
offset + 200 ms
• Positive
relationship
between
referential
certainty and
looking time
analysis
window
probabilty of fixating
• Reasons for looking in VWP?
Results: Passive task, new looks
Strong:
probabilty of fixating
(i.e., people look at things as they are referred to: more looks = higher certainty)
ms from pronoun onset
BEEP!
Instructions:
• You will see a picture and hear a story about it
• Listen carefully to the story and be prepared to answer some questions about it
• Eye-movements
recorded as participants view pictures and listen to stories.
Strong
bias:
Zebra
was
planting
vegetableprobe
garden
with Elephant.
• After
each
story:aquestions
ultimate
anaphora resolution + general comprehension
He• carefully
watered
young
plants
3 conditions,
21 the
stories
/ condition
determined by offline question (e.g., Who decided to add
Moderate
bias:
more varieties?)
Zebra and Elephant were planting a vegetable garden.
Zebra wanted Elephant to water the young plants.
Weak bias:
The animals were planting a vegetable garden.
Zebra and Elephant watered the young plants.
weak
moderate strong
He carefully watered …
Zebra wanted …
They carefully watered …
• Presumably
because listeners
don’t need visual
support to
facilitate/verify
interpretation
Results: Passive task, all looks
*
… he …
Strong
Zebra was planting …
Zebra and Elephant …
… he …
… he …
weak mod strong
Positive
relationship
holds
He carefully watered …
… he …
Moderate
*
Weak
The animals were …
• Positive
relationship holds
up to a point, but
then fails
analysis
window
%Zebra - %Elephant
200-800 ms
analysis window
Moderate
Zebra and Elephant …
weak mod strong
(Heller, Yee & Sedivy
Sedivy,, CUNY 2007)
Strong
Zebra was planting …
average
verb
average
verb
offset
+ 200
offset
+ 200
msms
Later, he decided to add more varieties
Results: Active task, all looks
• 3 condtions, 21 stories/condition determined by offline question (e.g., Who decided to add more varieties?)
• 42 filler stories
• 42 participants/experiment
*
*
ms from pronoun onset
Later, he decided to add more varieties
Methods
Methods
Critical sentence (+ final sentence)
Later, he decided to add more varieties.
They looked forward to eating the vegetables in a few months.
Questions:
• Who decided to add more varieties?
Active Task
• Did the animals plan to eat the vegetables?
• Touch last mentioned
Offline responses
animal upon hearing beep
(Zebra bias)
• In critical trials, beep
occurs after verb that
follows ambiguous
pronoun (e.g., directly
after “he decided”)
CUNY 2009
Sedivy3
Results: Active task, new looks
Visual World eyetracking Paradigm (VWP) is frequently used to study
real-time reference resolution
• Assumption: Visual attention is positively correlated with referential certainty
Zebra:Elephant ratio
and Julie
P220
Zebra wanted …
%Zebra - %Elephant
200-800 ms
analysis window
… he …
weak mod strong
Weak
The animals were …
They carefully watered …
… he …
Positive
relationship
fails
Conclusions
• Practical implications: Task matters
• In passive listening, looks may not reflect activation at high levels of certainty
• Active tasks are preferable because they force tighter link between certainty and visual attention:
• Goal directed looks are positively correlated with certainty
Passive Listening
• No task
• No beeps
• Theoretical implications: Visual attention is goal-directed
• Feedback from visual scene used verify interpretation
• If “goal” is rapid comprehension with no action, when a referent is sufficiently foregrounded, it becomes “given”
information and requires no additional visual attention -- we “move on” to process & predict other aspects of the
language/scene
• Reduced visual attention at high certainty may be due to same mechanism causing reduced referential
expressions (e.g., pronoun production, shortening), briefer gaze duration during picture naming, priming