The time-course of prediction in
incremental sentence processing:
Evidence from anticipatory
eye movements
Yuki Kamide et al.
Predictability and Anticipation – Information Theory meets Visual World
22/05/2015 – Saskia Reifers
Topic of Paper
●
Three eye-tracking experiments
●
●
'visual-world' paradigm
Support for an incremental processor
●
Thematic dependencies
●
●
Can they be evaluated in advanced of linguistic input?
Contextual information and combinatory information can
be applied as soon as appropriate lexical sources are
encountered
Topic of Paper
General Research Question
●
Do constraints that a verb places on its potential objects
come into effect as soon as the verb is encountered?
or
●
Is the application of such constraints delayed until the
object is encountered?
→
investigation of predictive processing
Topic of Paper
Predictive Processing
●
Monitor eye movements towards objects before they are
referred to in the linguistic input
Visual World Paradigm
● Visual scene and auditory presentation of the
linguistic stimulus
→
anticipatory eye movements
Topic of Paper
Former Research
●
Altmann and Kamide (1999)
●
Anticipation of upcoming material on the basis of selectional
restrictions
●
Semantic requirements of a noun to be used as
argument for a verb
drink – object needs to be drinkable, liquid
●
Experiments with a verb in simple subject-verb-object
sentence were made
→ present paper builds on these experiments
Experiment 1
3-place verbs
The woman will spread the butter on the bread
Experiment 1
3-place verbs
The woman will slide the butter to the man
Experiment 1
Experimental Material
●
Two sentential conditions
Inanimate Goal
The woman will spread the butter on the bread.
Animate Goal
The woman will slide the butter to the man.
Experiment 1
Research Questions
1. Research Question:
● Can information associated with a 3-place verb be used to
anticipate properties of the second post-verbal argument
prior to its onset?
●
Anticipation at the verb or during the first post-verbal
argument?
Experiment 1
Research Questions
2. Research Question:
● Are anticipatory eye movements restricted to moments in
time when the linguistic input does not explicitly refer to
any object in the visual scene?
●
reflect only what is referred to in the here-and-now
●
relect only what might be referred to next when there is
nothing being referred to in the here-and now
Experiment 1
Method
Subjects
Procedure
●
64 subjects
●
Presented a visual scene
●
Native speakers of English
●
Listening to sentences
●
Students from the
University of York
Stimuli
●
18 experimental pictures each paired with two sentential
conditions
●
●
Five objects: Agent, Theme, Inanimate Goal, Animate Goal,
Distractor
24 filler items, fixed-random order, two lists
Experiment 1
Results
●
Two Regions of interest
●
Appropriate object
●
●
●
Bread for the Inanimate condition
Man for the Animate condition
Inappropriate object
●
●
Bread for the Animate condition
Man for the Inanimate condition
Experiment 1
Results
Percentage of trials for each region of interest with looks to
the appropriate target, the inappropriate target, and the Agent
Experiment 1
Results
●
Asymmetry in anticipatory looks during the butter
●
Looks to the man
●
●
●
Slide condition: 24,3%
Spread condition: 16,8%
Looks to the bread
●
●
Slide condition: 24,3%
Spread condition: 27,4%
Explanation: Bread could be a Theme
Experiment 1
Discussion
●
More anticipatory eye movements towards the appropriate
Goal object than towards the inappropriate Goal object
●
During the post-verbal referring expression
Problematic
●
●
The woman will slide the man the butter.
Anticipatory eye movements can be obtained during
reference to some other object in the scene
Experiment 1
Open Questions
●
Are the observed effects driven by lexical information
associated with the verb alone?
or
●
Are such effects due to the combination of distinct sources
of information?
Example
●
The woman will spread the butter on the bread.
●
The woman will spread the plaster on the wall.
→
Experiment 2
Experiment 2
Example
The man will ride the motorbike.
Experiment 2
Example
The girl will ride the carousel.
Experiment 2
Research Questions
1. Research Question:
● Can information about the Agent be combined with the
selectional restrictions of the verb to predict the Theme?
Combinatory information
●
semantics of the Agent in combination with the verb's
selectional restrictions over its post-verbal arguments
Experiment 2
Experimental Material
Sentential Conditions
● The man will ride the motorbike.
●
The girl will ride the carousel.
→
Combinatory information should guide anticipatory
eye movements
BUT
●
Could be driven by low-level associations between the
Agent and the Theme Object
●
The cat will eat the mouse.
Experiment 2
Experimental Material
Additional Sentential Conditions
● The man will taste the beer.
●
The girl will taste the sweets.
●
Independence of anticipatory eye movements
●
●
Observation of same movements in the additional sentential
conditions
Looks to motorbike during the man should be the same in
both conditions
●
The man will ride the motorbike.
●
The man will taste the beer.
Experiment 2
Method
Subjects
Procedure
●
64 subjects
●
Presented a visual scene
●
Native speakers of English
●
Listening to sentences
●
Students from the
University of York
Stimuli
●
24 scenes each paired with four sentential conditions
●
Fixed-random order
●
24 unrelated filler items
Experiment 2
Hypothesis
●
Observation of looks towards the motorbike at the verb
●
by the combination of lexical information associated with
both the verb and its preceding subject
Combinatory Effects
●
Compare looks towards the motorbike after
Only verb-based information
● The man will ride
→
●
●
The girl will ride
●
No difference
Compare looks towards the motorbike after
● The man will ride
Only Agent-based information
→
●
The man will taste
●
No difference
Experiment 2
Hypothesis
Observation of looks towards the motorbike at the verb
●
●
by the combination of lexical information associated with
both the verb and its preceding subject
Agent Effects
●
Compare looks towards the motorbike after
●
●
The man will taste
The girl will taste
→ Agent-only effect
●
more looks to the motorbike in the first condition
Experiment 2
Hypothesis
Observation of looks towards the motorbike at the verb
●
●
by the combination of lexical information associated with
both the verb and its preceding subject
Verb Effects
●
Compare looks towards the motorbike after
●
●
The girl will ride
The girl will taste
→ selectional restrictions associated with the verb
●
More looks to the motorbike in the first condition
Experiment 2
Results
●
Two regions of interest
Experiment 2
Results
Percentage of trials for each region of interest with looks to the
motorbike in each of the four different sentential conditions
Experiment 2
Results
Combinatory Effects
● More looks towards the motorbike (both regions)
●
●
in the The man will ride condition than in the The girl will ride
condition
in the The man will ride condition than in the The man will
taste condition
Agent Effects
● No difference between the two conditions in both regions
Verb Effects
● No difference between the two conditions in first region
●
Second region: More looks towards the motorbike in the
The girl will ride condition than in the The girl will taste
condition
Experiment 2
Discussion
●
Combinatory looks were found at the verb (region 1)
●
No selectional restrictions by the verb alone responsible
●
No association between Agent and Theme responsible
BUT
● Region 2: effect of the verb's selectional restrictions
●
More looks at The girl will ride than The girl will taste
condition
Verb effect?
→ Contrast effect
●
Late look towards contrasting objects in the visual scene
●
To confirm selection of Theme
Experiment 2
Open Questions
●
Can a verb's argument only be predicted at or after the
verb?
●
Predictive processes shown so far seem to be verb-specific
●
Is the category 'verb' important or the strength of the
constraints that accompany lexical items of that type?
●
Are the observed movements due to the syntactic
combination of the verb with its subject?
or
●
Are the observed movements due to the combination of
semantic information independently of any syntactic
processing?
Experiment 3
Motivation
Japanese language
●
All arguments of the verb appear prior to the verb
●
Each argument is case-marked (post-nominal particle)
→ if verbs are the driving force behind predictive processing,
there should not be any opportunity in Japanese
constructions for predictive processing of forthcoming
arguments
Experiment 3
Example
weitoresu-ga kyaku-ni
tanosigeni hanbaagaa-o
waitress-nom customer-dat merrily
hakobu.
hamburger-acc bring.
The waitress will merrily bring the hamburger to the customer.
Experiment 3
Example
weitoresu-ga kyaku-o
tanosigeni karakau.
waitress-nom customer-acc merrily
tease.
The waitress will merrily tease the customer.
Experiment 3
Research Questions
1. Research Question:
● Will the processor anticipate that the third argument will
refer to the one object in the scene that could plausibly be
referred to?
●
after the dative condition (NP1-nom NP2-dat) NP-acc is
obligatory (only very few exceptions)
●
For the accusative condition two alternative ways are possible
●
●
No further arguments (monotransitive construction)
3-NP construction, scene contains no plausible object that
could be referred to as Goal
Experiment 3
Experimental Material
Sentential Conditions
Animate noun as second argument
●
Waitress-nom customer-dat merrily hamburger-acc bring.
●
Waitress-nom customer-acc merrily tease.
Inanimate noun as second argument
●
man-nom wall-dat carefully poster-acc hung.
●
man-nom wall-acc carefully wipe.
Foil sentences
● NP1-nom NP2-acc adverb NP3-dat verb
Experiment 3
Method
Subjects
Procedure
●
22 subjects
●
Presented a visual scene
●
Native speakers of Japanese
●
Listening to sentences
●
Students from the University
of York
Stimuli
●
16 sets of experimental items (8 animate, 8 inanimate)
●
16 foil sentences, 16 filler items
●
Fixed-random order, two subject groups, two lists
Experiment 3
Results
●
One regions of interest
Experiment 3
Results
Percentage of trials for each region of interest with looks to the
hamburger in each of the two different sentential conditions
Experiment 3
Discussion
●
Anticipatory eye movements towards the potential referent
of NP3
●
Looks to the most plausible object in scene
●
After encountering the sequence 'NP1 NP2'
→ predictive processing possible without grammatical
head
●
Syntactic information was used
● Case marking
General Discussion
●
Experiments support approach for an incremental
processor
●
Verb-based information anticipates post-verbal arguments
●
Not only the following argument (e.g. Goal)
●
Combinatory information anticipates a subsequent argument
(e.g. agent + verb → theme)
●
Head-final constructions anticipate further arguments in the
absence of their head
General Discussion
●
Two kinds of anticipation
'prediction' – a projection 'forward in time'
●
Processor predicts a post-verbal argument / linguistic
input
●
●
Grammatical information + semantic information
Projected structure is evaluated against visual / mental
context
General Discussion
●
Two kinds of anticipation
'integration' – current evidence
●
●
Not language-specific
Thematic organization of objects in the visual context
●
●
Subset of objects will take part in the event being
described visually
Find roles for those objects during linguistic input
General Discussion
●
Two kinds of anticipation
'prediction' – a projection 'forward in time'
'integration' – current evidence
The man will ride …
●
Does the linguistic structure trigger a predictive process?
●
Syntactic reference
or
●
Does the visual context suggest a plausible argument?
Thank you!