A step forward in the analysis of visual world
eye-tracking data
Aki-Juhani
Vincent Porretta1 ,
Jacolien van Rij3 , and Juhani Järvikivi1
Kyröläinen2 ,
1 University
2 University
of Alberta,
of Turku, 3 University of Tübingen
[email protected]
28 August 2015
1 / 27
Overview
1
Background on the Visual World Paradigm
2 / 27
Overview
1
Background on the Visual World Paradigm
2
The present experiment
2 / 27
Overview
1
Background on the Visual World Paradigm
2
The present experiment
3
Analyzing Visual World data
Analysis of Variance (ANOVA)
Linear mixed-effects models (LMER)
Generalized additive mixed models (GAMM)
2 / 27
Overview
1
Background on the Visual World Paradigm
2
The present experiment
3
Analyzing Visual World data
Analysis of Variance (ANOVA)
Linear mixed-effects models (LMER)
Generalized additive mixed models (GAMM)
4
Discussion
2 / 27
Visual World Paradigm (VWP)
3 / 27
Visual World Paradigm (VWP)
Records eye gaze location on the screen while presenting an
acoustic signal
3 / 27
Visual World Paradigm (VWP)
Records eye gaze location on the screen while presenting an
acoustic signal
Primarily used to study:
I
I
the time-course of spoken language processing (Allopenna et al.,
1998)
the role of visual context in language comprehension (Tanenhaus
et al., 1995)
3 / 27
Visual World Paradigm (VWP)
Records eye gaze location on the screen while presenting an
acoustic signal
Primarily used to study:
I
I
the time-course of spoken language processing (Allopenna et al.,
1998)
the role of visual context in language comprehension (Tanenhaus
et al., 1995)
Can be used to investigate:
I
I
anticipatory/predictive processing (Kamide et al., 2003)
language processing and development in different populations
(Järvikivi et al., 2014)
3 / 27
Visual World Paradigm (VWP)
Records eye gaze location on the screen while presenting an
acoustic signal
Primarily used to study:
I
I
the time-course of spoken language processing (Allopenna et al.,
1998)
the role of visual context in language comprehension (Tanenhaus
et al., 1995)
Can be used to investigate:
I
I
anticipatory/predictive processing (Kamide et al., 2003)
language processing and development in different populations
(Järvikivi et al., 2014)
Underlying hypothesis: People subconsciously move their gaze to
the visual representation of the word they are processing
3 / 27
A typical experiment
Figure adapted from (Huettig, Rommers, & Meyer, 2011)
4 / 27
A typical experiment
Figure adapted from (Huettig et al., 2011)
Eye gaze time-locked to auditory stimulus
I
I
"The boy will [critical word]..."
"Click on the [critical word]."
4 / 27
A typical experiment
Figure adapted from (Huettig et al., 2011)
Eye gaze time-locked to auditory stimulus
I
I
"The boy will [critical word]..."
"Click on the [critical word]."
Typically with a short preview of visual display (< 1s)
4 / 27
Typical visualization of VWP data
Grand Average
●
●
● ●
●
●
●
●
0.6
●
●
●
●
●
●
Onset of critical item
Proportion Looks
●
●
Item
●
●
0.4
●
Interest Area 1
●
Interest Area 2
●
Interest Area 3
●
Interest Area 4
●
●
●
●
●
●
●
●
0.2
●
●
0.0
●
● ●
● ●
● ● ● ● ●
0
● ● ● ● ● ●
●
● ●
● ●
● ● ●
● ●
●
●
●
● ●
● ●
●
● ●
●
●
● ●
● ●
● ●
● ● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ●
●
● ● ● ●
● ● ●
●
●
●
● ●
● ●
●
●
● ● ● ● ● ●
●
●
●
●
●
●
●
●
●
●
● ● ● ● ● ● ● ● ● ●
● ●
● ●
● ● ● ● ● ● ● ● ● ● ● ●
● ●
● ●
● ● ● ● ● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
200
400
600
800
1000
Time
5 / 27
The present experiment
6 / 27
The present experiment
An investigation of the effect of foreign accentedness on lexical
activation
I
How does accent strength influence lexical processing?
6 / 27
The present experiment
An investigation of the effect of foreign accentedness on lexical
activation
I
How does accent strength influence lexical processing?
The visual world (word) paradigm provides a measure when in
time listeners successfully decode the acoustic signal
I
Looks to the target word during listening
6 / 27
Subjects and Stimuli
7 / 27
Subjects and Stimuli
48 Native English-speaking participants
7 / 27
Subjects and Stimuli
48 Native English-speaking participants
Items
I
I
I
I
From the NU Wildcat Corpus of Native and Foreign-Accented
English (Van Engen et al., 2010)
40 monosyllabic English words
Four talkers (1 native English, 3 native Chinese)
All tokens rated for accentedness (Porretta et al., 2015)
7 / 27
Procedure
8 / 27
Procedure
Listen and find
I
Participants heard a token and had to find the written word among
four options (McQueen & Viebahn, 2007)
8 / 27
Procedure
Listen and find
I
Participants heard a token and had to find the written word among
four options (McQueen & Viebahn, 2007)
The quadruplet consisted of (Allopenna et al., 1998):
I
I
I
I
the target word
an onset competitor (i.e., word sharing the onset)
a rhyme competitor (i.e., rhyming word)
a distractor (i.e., phonologically/semantically unrelated word)
8 / 27
Procedure
Listen and find
I
Participants heard a token and had to find the written word among
four options (McQueen & Viebahn, 2007)
The quadruplet consisted of (Allopenna et al., 1998):
I
I
I
I
the target word
an onset competitor (i.e., word sharing the onset)
a rhyme competitor (i.e., rhyming word)
a distractor (i.e., phonologically/semantically unrelated word)
8 / 27
The data
Grand Average
●
●
● ●
●
●
●
●
0.6
●
●
●
●
●
●
Proportion Looks
●
●
Word
●
●
0.4
●
Target
●
Rhyme Competitor
●
Onset Competitor
●
Distractor
●
●
●
●
●
●
●
●
0.2
●
●
0.0
●
● ●
● ●
● ● ● ● ●
0
● ● ● ● ● ●
●
● ●
● ●
● ● ●
● ●
●
●
●
● ●
● ●
●
● ●
●
●
● ●
● ●
● ●
● ● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ●
●
● ● ● ●
● ● ●
●
●
●
● ●
● ●
●
●
● ● ● ● ● ●
●
●
●
●
●
●
●
●
●
●
● ● ● ● ● ● ● ● ● ●
● ●
● ●
● ● ● ● ● ● ● ● ● ● ● ●
● ●
● ●
● ● ● ● ● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
200
400
600
800
1000
Time
9 / 27
The data
Grand Average
●
●
● ●
●
●
●
●
0.6
●
●
●
●
●
●
Proportion Looks
●
●
Word
●
●
0.4
●
Target
●
Rhyme Competitor
●
Onset Competitor
●
Distractor
●
●
●
●
●
●
●
●
0.2
●
●
0.0
●
● ●
● ●
● ● ● ● ●
0
● ● ● ● ● ●
●
● ●
● ●
● ● ●
● ●
●
●
●
● ●
● ●
●
● ●
●
●
● ●
● ●
● ●
● ● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ●
●
● ● ● ●
● ● ●
●
●
●
● ●
● ●
●
●
● ● ● ● ● ●
●
●
●
●
●
●
●
●
●
●
● ● ● ● ● ● ● ● ● ●
● ●
● ●
● ● ● ● ● ● ● ● ● ● ● ●
● ●
● ●
● ● ● ● ● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
200
400
600
800
1000
Time
How are looks to the target influenced by foreign accent strength?
9 / 27
ANOVA
10 / 27
ANOVA
Factorial predictor
Empirical logit transformation
I
Proportions are inherently bound between 0 and 1
Selection of a time window over which to average
Investigate the average difference in activation between High and
Low accent words
10 / 27
ANOVA (200-700ms)
11 / 27
ANOVA (200-700ms)
Median split on Accentedness
I
Within-subjects factor
Empirical logit looks to target
Average over 500 ms window (200–700 ms)
Carried out in R
11 / 27
ANOVA (200-700ms)
12 / 27
ANOVA (200-700ms)
Significant effect of Accent [F(1) = 9.075, p < 0.005] on target looks
−1.9
−2.0
Empirical Logit Looks
●
−2.1
−2.2
●
−2.3
−2.4
Low
High
Accent Level
12 / 27
ANOVA (200-700ms)
Significant effect of Accent [F(1) = 9.075, p < 0.005] on target looks
−1.9
−2.0
Empirical Logit Looks
●
−2.1
−2.2
●
−2.3
−2.4
Low
High
Accent Level
What is the effect along the continuum of accentedness?
12 / 27
LMER
13 / 27
LMER
Continuous predictor
Empirical logit transformation
Selection of a time window over which to average
Inclusion of random effects and control variables
Investigate the average activation along the accentedness
continuum within the time window
13 / 27
LMER (200-700ms)
14 / 27
LMER (200-700ms)
Accentedness rating as continuous measure
Empirical logit looks to target
Average over 500 ms window (200–700 ms)
Random effects for Subject and Item
Trial as a control variable
Carried out in R with package lme4
14 / 27
LMER (200-700ms)
15 / 27
LMER (200-700ms)
−2.0
−2.2
−2.4
−2.6
Empirical Logit Looks
−1.8
Significant effect of Accentedness (β = -0.1146, SE = 0.0467, t =
-2.455) on target looks.
2
4
6
8
Accentedness
15 / 27
LMER (200-700ms)
−2.0
−2.2
−2.4
−2.6
Empirical Logit Looks
−1.8
Significant effect of Accentedness (β = -0.1146, SE = 0.0467, t =
-2.455) on target looks.
2
4
6
8
Accentedness
Is this effect really linear?
15 / 27
LMER (200-700ms)
−2.0
−2.2
−2.4
−2.6
Empirical Logit Looks
−1.8
Significant effect of Accentedness (β = -0.1146, SE = 0.0467, t =
-2.455) on target looks.
2
4
6
8
Accentedness
Is this effect really linear? How does the effect change through time?
15 / 27
GAMM
16 / 27
GAMM
Allows complex interactions between continuous variables
Suited to handle non-linear data
Empirical logit transformation
No need to average over an arbitrary time window
Can account for autocorrelation in time series data
I
Inherent correlation between timepoint samples
Inclusion of random effects and control variables
Investigate activation through time along the continuum of
accentedness
16 / 27
GAMM
17 / 27
GAMM
Interaction of Time and Accentedness rating (both as continuous
predictors)
Empirical logit looks to target
Time window from 200 ms to 700 ms
Random effect for Event (combination of subject and item)
Trial as a control variable
Carried out in R with package mgcv
17 / 27
GAMM
Significant effect of Time [EDF = 6.221, F = 729.214, p < 0.0001]
1.0
0.5
0.0
−0.5
−1.0
Empirical Logit Looks (centered)
1.5
Main effect of Time
200
300
400
500
600
700
Time
18 / 27
GAMM
Significant effect of Accentedness [EDF = 2.849, F = 8.633, p <
0.0001]
1.0
0.5
0.0
−0.5
−1.0
Empirical Logit Looks (centered)
1.5
Main effect of Accentedness
2
4
6
8
Accentedness
19 / 27
GAMM
Significant interaction between Time and Accentedness [EDF =
12.684, F = 14.002, p < 0.0001]
Significantly improves model [χ2 = 85.581, df = 3, p < 0.0001]
Interaction of Time and Accentedness
−3
0.612
−2
−2.5
8
−1.5
−1.3305
−1
4
5
0.
2
Accentedness
6
−3.273
0
−3
−0.5
200
300
400
500
Time
600
700
20 / 27
Discussion
Generalized additive mixed models
21 / 27
Discussion
Generalized additive mixed models
Inclusion of random effects (as with LMER)
Can account for design-induced variability, such as trial effects (as
with LMER)
21 / 27
Discussion
Generalized additive mixed models
Inclusion of random effects (as with LMER)
Can account for design-induced variability, such as trial effects (as
with LMER)
No assumption of linearity
I
I
(Possibly non-linear) functional form estimated based on the data
Results in improved fit of continuous predictors
Effectively/efficiently handle time-series data
I
I
I
No need to arbitrarily select analysis window for averaging
Can account for autocorrelation in time
R package: itsadug (van Rij et al., 2015)
21 / 27
Discussion
Generalized additive mixed models
22 / 27
Discussion
Generalized additive mixed models
Some caveats
22 / 27
Discussion
Generalized additive mixed models
Some caveats
I
I
I
I
Direction and shape of effects can only be interpreted visually
GAMMs do not handle interactions among multiple factors
gracefully
Computationally intensive
Logistic regression is an alternative to empirical logit
transformation (however, autocorrelation parameter cannot be
included)
22 / 27
Take-home message
Generalized additive mixed models
23 / 27
Take-home message
Generalized additive mixed models
Significant advantages over other statistical methods
Provide a new way to examine visual world data as a non-linear
time-series
Allow us to answer new and interesting questions
I
I
Investigate the effect of a continuous predictor over time
Understand when in time (in greater detail) effects take place
without averaging the time window
23 / 27
Thank you.
Questions?
24 / 27
References I
Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998).
Tracking the time course of spoken word recognition using eye
movements: Evidence for continuous mapping models. Journal
of Memory and Language, 38(4), 419–439.
Huettig, F., Rommers, J., & Meyer, A. S. (2011). Using the visual
world paradigm to study language processing: A review and
critical evaluation. Acta Psychologica, 137(2), 151–171.
Järvikivi, J., Pyykkönen-Klauck, P., Schimke, S., Colonna, S., &
Hemforth, B. (2014). Information structure cues for 4-year-olds
and adults: Tracking eye movements to visually presented
anaphoric referents. Language, Cognition and Neuroscience,
29(7), 877–892.
25 / 27
References II
Kamide, Y., Altmann, G., & Haywood, S. (2003). The time-course of
prediction in incremental sentence processing: Evidence from
anticipatory eye movements, 49, 133–159. Journal of Memory
and Language, 49(1), 133–156.
McQueen, J. M. & Viebahn, M. C. (2007). Tracking recognition of
spoken words by tracking looks to printed words. The
Quarterly Journal of Experimental Psychology, 60(5), 661–671.
Porretta, V., Kyröläinen, A.-J., & Tucker, B. V. (2015). Perceived
foreign accentedness: Acoustic distances and lexical properties.
Attention, Perception, & Psychophysics, 1–14.
Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., &
Sedivy, J. E. (1995). Integration of visual and linguistic
information in spoken language comprehension. Science, 268,
1632–1634.
26 / 27
References III
Van Engen, K. J., Baese-Berk, M., Baker, R. E., Choi, A., Kim, M., &
Bradlow, A. R. (2010). The Wildcat corpus of native-and
foreign-accented English: Communicative efficiency across
conversational dyads with varying language alignment profiles.
Language and Speech, 53(4), 510–540.
van Rij, J., Wieling, M., Baayen, R. H., & van Rijn, H. (2015). itsadug:
interpreting time series and autocorrelated data using gamm. R
package version 0.93.
27 / 27