Cognitive Load in the Visual World: The Facilitatory Effect of Gaze
Mirjana Sekicki & Maria Staudte - Saarland University, Germany
Questions
1. Does following a gaze cue influence the cognitive load required for
• We used the Index of Cognitive Activity (ICA: Marshall, 2000; Demberg
processing the corresponding linguistic referent?
& Sayeed, 2016) - a novel measure that allowed us to collect both:
No−gaze Condition
No−gaze Condition
2. Is considering
the
gaze
cue
costly?
Is
there
a
distribution
of
cognitive
load
•
eye-movements
to
detect
patterns
of
anticipation;
order ice−cream
order water
spill ice−cream
spill
order
water
ice−cream
order water
spill ice−cream
spill water
between the cue and the referent?
• cognitive load - to measure the cost of linguistic processing online,
3. Can a gaze cue have a disruptive effect on processing the linguistic referent?
as influenced by the gaze cue.
1
0.4
Der Mann verschüttet gleich das Wasser.
0.2
0.4
0.6
0.2
the water
das Eis.
the ice-cream
water
ice−cream
ice−cream
0.6
distractor2
face
face
22
<100ms>
21
Mean ICA values
ref.gaze water
2
Verb
Adverb
Region
ref.gaze sausage
Object
ref.gaze water
verb
gaze
noun
water
20
Mean ICA values
sausage
21
0.6
distractor1
distractor2
the sausage
5000
4500
17
16
15
15
sausage12
12
21
Mean ICA values
5000
4500
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
2000
1500
16
13
20
0.0
17
Subject
Verb
Region
Adverb
Object
2nd half
No gaze.
water
sausage
Subject
Verb
Region
Adverb
Object
Adverb
Object
21 Referent gaze.
water
sausage
20
19
19
18
18
17
16
15
17
16
15
14
14
13
13
12
12
Subject
Verb
Region
Adverb
Object
Subject
Verb
Region
congruent gaze (tea)
incongruent gaze (juice)
verb
gaze
noun
ICA
no−gaze
gaze
Gaze:
no gaze
congruent
21
incongruent
22
Objects:
tea
juice
distractor1
distractor2
face
0.4
Mean ICA values
5000
4800
4600
4400
4200
4000
3800
3600
3400
3200
3000
2800
2600
2400
2200
2000
1800
1600
1400
1200
1000
5000
4800
4600
4400
4200
Proportion Of Fixations
19
18
17
16
15
14
0.2
13
12
Subject
Region
Adverb
Object
1000
1200
1400
1600
1800
2000
2200
2400
2600
2800
3000
3200
3400
3600
3800
4000
4200
4400
4600
4800
5000
1000
1200
1400
1600
1800
2000
2200
2400
2600
2800
3000
3200
3400
3600
3800
4000
4200
4400
4600
4800
5000
1000
1200
1400
1600
1800
2000
2200
2400
2600
2800
3000
3200
3400
3600
3800
4000
4200
4400
4600
4800
5000
0.0
Verb
Trial Time (ms)
Answers
• The gaze cue is followed and inspires prediction for the upcoming referent, which leads to it being processed with less cognitive effort.
• This holds even when the cue (and the corresponding referent) are mismatching with the previous linguistic context.
• The perception of the gaze cue itself proved not to be costly unless mismatching with the linguistic context.
• Congruent gaze cue helps reduce the cognitive load on the subsequent referent, while incongruent does not induce additional cost on processing the
referent noun (fitting the linguistic context).
the 30th CUNY Conference - MIT, Cambridge MA
March 2017
5000
4500
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
18
20
4000
3800
3600
3400
3200
3000
2800
2600
2400
2200
2000
1800
1600
1400
1200
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
2000
18
0.6
soonTrial Timethe
tea
(ms)
1000
5000
4800
4600
4400
4200
4000
3800
3600
3400
3200
3000
2800
2600
2400
2200
2000
1800
1500
1000
19
13
face
gaze
21 Referent gaze.
water
sausage
20
No gaze.
water
sausage
distractor2
Objects:
tea
juice
distractor1
distractor2
face
0.0
1600
Object
Fixations
Der Mann verschüttet gleich den Tee.
1400
Adverb
1st half
0.2
1200
Region
The Fitting - Incongruent Study
0.4
1000
Verb
19
distractor1
Trial Time (ms)
3
spills
Subject
water
1000
no−gaze
• gaze not always congruent with the
referent noun: Gaze (no-gaze vs.
referent vs. competitor gaze)
• referent noun (& gaze) always fitting
the linguistic context
the man
15
14
Objects:
0.4
5000
4500
4000
3500
3000
2500
2000
incongruent gaze (juice)
0.6
Proportion Of Fixations
1500
1000
5000
4500
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
congruent gaze (tea)
16
14
0.2
2000
4000
relevant windows:
600ms from cue onset
600ms from noun middle
Mean ICA values
no−gaze water
Objects:
the water
Trial Time (ms)
18
ICA
no−gaze sausage
The man spills soon the tea.
19
17
Fixations
Proportion Of Fixations
soon
20
The Mismatching - Congruent Study
die Wurst.
1500
5000
18
17
Der Mann verschüttet gleich das Wasser.
1000
4500
19
face
5000
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
2000
1500
20
Mean ICA values
• gaze always congruent with the
referent noun: Gaze (no-gaze vs.
referent gaze)
• referent noun (& gaze) not always
fitting the context: Fit (fitting vs.
mismatching)
gaze
23 Referent gaze.Conditions:
spill water
spill ice−cream
22
order water
order ice−cream
21
Mean ICA values
“…soon…”
Subject
4500
face
ICA
gaze
No gaze. Conditions:
spill water
spill ice−cream
order water
order ice−cream
15
4000
distractor2
Trial Time (ms)
16
3500
distractor1
0.0
1000
5000
4500
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
1000
23
<1000ms>
3000
ice−cream
Trial Time (ms)
“…the water.”
2500
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
2000
1500
1000
Trial Time (ms)
<300ms>
spills
Obje
0.4
water
0.2
“The man spills …”
the man
verb
gaze
noun
spill water
Objects:
0.0
ref.gaze sausage
spill ice−cream
distractor1
distractor2
0.4
<1000ms>
no−gaze water
order water
Objects:
distractor1
1500
1000
2000
1500
2500
2000
3000
2500
3500
3000
4000
3500
4500
4000
5000
4500
1000
4500
Trial Time (ms)
gaze sausage
order ice−cream
0.0
4000
3500
3000
2500
2000
1500
1000
5000
4500
4000
3500
2500
2000
1500
1000
5000
4500
4000
3500
3000
2500
2000
1500
1000
orders
3000
gleich das Wasser.
das Eis.
0.0
verb noun
spill water
0.2
5000
Der Mann bestellt
spill ice−cream
Referent Gaze Condition
water
5000
soon
order water
Objects:
2000
spills
order ice−cream
1500
the man
Fixations
No−gaze Condition
Proportion Of Fixations
Proportion Of Fixations
Proportion Of Fixations
• gaze always congruent with the
referent noun: Gaze (no-gaze vs.
referent gaze)
• verb more/less restrictive: Constraint
(restrictive vs. non-restrictive)
• referent noun fitting the context:
Plausibility (plausible vs. possible)
The Fitting - Congruent Study
0.6
Proportion Of Fixations
0.6