Background
Experiment
Results
Discussion
References
What big eyes you have:
Pupillary response to reduced speech
Vincent Porretta,
Yoichi Mukai, and Benjamin V. Tucker
University of Alberta
[email protected]
20 October 2016
The Tenth International Conference on the Mental Lexicon
Ottawa, ON
1 / 23
Background
Experiment
Results
Discussion
References
Reduced speech
• Casual, non-careful speech common in spontaneous
productions
2 / 23
Background
Experiment
Results
Discussion
References
Reduced speech
• Casual, non-careful speech common in spontaneous
productions
• Sounds and/or syllables may disappear
• Consonants may appear with a different manner of
articulation or voicing
• Vowels may change quality or appear more consonant-like
2 / 23
Background
Experiment
Results
Discussion
References
Reduced speech
• Casual, non-careful speech common in spontaneous
productions
• Sounds and/or syllables may disappear
• Consonants may appear with a different manner of
articulation or voicing
• Vowels may change quality or appear more consonant-like
• Reduction processes introduce additional variation into the
speech signal
2 / 23
Background
Experiment
Results
Discussion
References
Reduced speech
• Casual, non-careful speech common in spontaneous
productions
• Sounds and/or syllables may disappear
• Consonants may appear with a different manner of
articulation or voicing
• Vowels may change quality or appear more consonant-like
• Reduction processes introduce additional variation into the
speech signal
• Listeners must handle these reductions in order to
comprehend the intended message
2 / 23
Background
Experiment
Results
Discussion
References
Processing reduced forms
• A number of studies have investigated the processing of
reduced speech (e.g., Connine, Ranbom, & Patterson,
2008; McLennan, Luce, & Charles-Luce, 2003; Ranbom
& Connine, 2007)
3 / 23
Background
Experiment
Results
Discussion
References
Processing reduced forms
• A number of studies have investigated the processing of
reduced speech (e.g., Connine et al., 2008; McLennan
et al., 2003; Ranbom & Connine, 2007)
• Tucker (2011) conducted an auditory lexical decision
experiment using words containing reduced and unreduced
word-medial /d/(flap) and /g/
3 / 23
Background
Experiment
Results
Discussion
References
Processing reduced forms
• A number of studies have investigated the processing of
reduced speech (e.g., Connine et al., 2008; McLennan
et al., 2003; Ranbom & Connine, 2007)
• Tucker (2011) conducted an auditory lexical decision
experiment using words containing reduced and unreduced
word-medial /d/(flap) and /g/
• Reaction times showed that words with reduced variants
take longer to process than those with unreduced variants
3 / 23
Background
Experiment
Results
Discussion
References
Processing reduced forms
• A number of studies have investigated the processing of
reduced speech (e.g., Connine et al., 2008; McLennan
et al., 2003; Ranbom & Connine, 2007)
• Tucker (2011) conducted an auditory lexical decision
experiment using words containing reduced and unreduced
word-medial /d/(flap) and /g/
• Reaction times showed that words with reduced variants
take longer to process than those with unreduced variants
• This indicates that reduced forms are more difficult to
process than unreduced forms
3 / 23
Background
Experiment
Results
Discussion
References
Processing reduced forms
• A number of studies have investigated the processing of
reduced speech (e.g., Connine et al., 2008; McLennan
et al., 2003; Ranbom & Connine, 2007)
• Tucker (2011) conducted an auditory lexical decision
experiment using words containing reduced and unreduced
word-medial /d/(flap) and /g/
• Reaction times showed that words with reduced variants
take longer to process than those with unreduced variants
• This indicates that reduced forms are more difficult to
process than unreduced forms
• No difference was found between /d/(flap) and /g/ overall
3 / 23
Background
Experiment
Results
Discussion
References
Pupillometry
• Pupillometry is the measurement of pupil dilation
4 / 23
Background
Experiment
Results
Discussion
References
Pupillometry
• Pupillometry is the measurement of pupil dilation
• Early studies used this as a measure of emotional arousal
(see Beatty, 1982)
4 / 23
Background
Experiment
Results
Discussion
References
Pupillometry
• Pupillometry is the measurement of pupil dilation
• Early studies used this as a measure of emotional arousal
(see Beatty, 1982)
• The response is sensitive to linguistic variables such as
lexical frequency (Kuchinke et al., 2007)
4 / 23
Background
Experiment
Results
Discussion
References
Pupillometry
• Pupillometry is the measurement of pupil dilation
• Early studies used this as a measure of emotional arousal
(see Beatty, 1982)
• The response is sensitive to linguistic variables such as
lexical frequency (Kuchinke et al., 2007)
• Pupillary response is larger for auditory stimuli than for
visual stimuli (Klingner et al., 2010)
4 / 23
Background
Experiment
Results
Discussion
References
Pupillometry and speech
• Pupil dilation can be used to measure cognitive processing
load for speech perception (Beatty, 1982)
5 / 23
Background
Experiment
Results
Discussion
References
Pupillometry and speech
• Pupil dilation can be used to measure cognitive processing
load for speech perception (Beatty, 1982)
• The response has been shown to vary with:
• Listening effort (Zekveld et al., 2010; Zekveld & Kramer,
2014)
• Syntactic ambiguity (Engelhardt et al., 2010)
• Speech rate (Koch & Janse, 2016)
• Detection of mispronunciations (Tamási et al., in press)
5 / 23
Background
Experiment
Results
Discussion
References
This study
• The current study examines spoken word processing (as
measured by pupil dilation) of words containing reduced
and unreduced consonants
6 / 23
Background
Experiment
Results
Discussion
References
This study
• The current study examines spoken word processing (as
measured by pupil dilation) of words containing reduced
and unreduced consonants
• Questions:
• Is the processing load indexed by pupil dilation sensitive to
differences in reduction?
6 / 23
Background
Experiment
Results
Discussion
References
This study
• The current study examines spoken word processing (as
measured by pupil dilation) of words containing reduced
and unreduced consonants
• Questions:
• Is the processing load indexed by pupil dilation sensitive to
differences in reduction?
• If so, do the results correspond to patterns in reaction
times?
6 / 23
Background
Experiment
Results
Discussion
References
This study
• The current study examines spoken word processing (as
measured by pupil dilation) of words containing reduced
and unreduced consonants
• Questions:
• Is the processing load indexed by pupil dilation sensitive to
differences in reduction?
• If so, do the results correspond to patterns in reaction
times?
• When (if at all) do these differences emerge in time?
6 / 23
Background
Experiment
Results
Discussion
References
This study
• The current study examines spoken word processing (as
measured by pupil dilation) of words containing reduced
and unreduced consonants
• Questions:
• Is the processing load indexed by pupil dilation sensitive to
differences in reduction?
• If so, do the results correspond to patterns in reaction
times?
• When (if at all) do these differences emerge in time?
• Do dilation and time course reveal differences between /d/
and /g/ due to flapping, not previously observed in
behavioral results?
6 / 23
Background
Experiment
Results
Discussion
References
The experiment
• Task:
• Listen-and-repeat (similar to Zekveld et al., 2010)
• Presentation of stimulus (via headphones)
• 500 ms pure tone beep prompting response
• Participant’s spoken repetition of the stimulus
7 / 23
Background
Experiment
Results
Discussion
References
The experiment
• Task:
• Listen-and-repeat (similar to Zekveld et al., 2010)
• Presentation of stimulus (via headphones)
• 500 ms pure tone beep prompting response
• Participant’s spoken repetition of the stimulus
• Participants:
• Native speakers of North American (Western Canadian)
English (n = 39)
7 / 23
Background
Experiment
Results
Discussion
References
The experiment
• Task:
• Listen-and-repeat (similar to Zekveld et al., 2010)
• Presentation of stimulus (via headphones)
• 500 ms pure tone beep prompting response
• Participant’s spoken repetition of the stimulus
• Participants:
• Native speakers of North American (Western Canadian)
English (n = 39)
• Data:
• Gaze and pupil size data via Eyelink II eye-tracker (250 Hz)
• Spoken responses recorded on digital recorder via
head-mounted microphone
7 / 23
Background
Experiment
Results
Discussion
References
Materials
• Same materials as used in Tucker (2011)
• Naturally produced disyllabic words (n = 80) containing
word-medial /d/ and /g/.
• 40 /d/ (e.g., ‘ready’ /ôEdi/)
• 40 /g/ (e.g., ‘baggy’ /bægi/)
8 / 23
Background
Experiment
Results
Discussion
References
Materials
• Same materials as used in Tucker (2011)
• Naturally produced disyllabic words (n = 80) containing
word-medial /d/ and /g/.
• 40 /d/ (e.g., ‘ready’ /ôEdi/)
• 40 /g/ (e.g., ‘baggy’ /bægi/)
• Good examples of reduced and unreduced forms were
chosen based on intensity difference between the closure
and surrounding vowels
• Counterbalanced lists
8 / 23
Background
Experiment
Results
Discussion
References
Materials
• “Puddle” - Unreduced B (Left), Reduced B (Right)
9 / 23
Background
Experiment
Results
Discussion
References
Materials
• “Puddle” - Unreduced B (Left), Reduced B (Right)
• Durations:
• Average duration (overall): 370 ms
• Unreduced: 369 ms
• Reduced: 372 ms
9 / 23
Background
Experiment
Results
Discussion
References
Analysis
• Pupil dilation
• Production and production latency (analyses pending)
10 / 23
Background
Experiment
Results
Discussion
References
Analysis
• Pupil dilation
• Production and production latency (analyses pending)
• Pupil data pre-processing:
• Blinks semi-automatically cleaned
• Signal downsampled to 50 Hz
• Signal baseline normalized (500 ms pre-stimulus period)
10 / 23
Background
Experiment
Results
Discussion
References
Conditional Averages
0.10
Pupil Dilation
0.05
0.00
0.10
0.05
0.00
Reduced
Unreduced
●●●●●●●●●● ●
●● ●●
●●●●
●
●●
●●●●●●●●●
●●
●●
●●●
●
●●
●●
●●●●●●●●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●
●
●●●● ●●●●●● ●
●●●●●●
●●
●●
●●●
●● ●
●●●
●●
●●●
●●
●●
●●●
●●
●
●●
●●●
●
●●
●●
●●●
●
●●
●
●●●
●
●● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●● ●●●●●●●●
●●●●
●●●●● ●
●●
●
●●●●
●● ●
●●
● ●●●●●
●
●●●●
●●●
●●
●
●●
●
●●
●●●●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
500
1000
1500
●
● ●● ●●●●●●●●●●●●●● ●●●●
● ●● ●●
●●
●●●
●●●●
●
●●
●
●●
●●●
●●●●
●
●
●●
●●
●●
● ●● ●●
●●
● ●●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
2000
500
1000
1500
/d/
/g/
2000
Time
11 / 23
Background
Experiment
Results
Discussion
References
Modelling
• Performed in R version 3.3.1 (R Development Core Team,
2016)
• Generalized additive mixed-effects modeling using the
package mgcv version 1.8-15 (Wood, 2016)
• Model comparison and plotting using the package itsadug
version 2.2 (van Rij, Hollebrandse, & Hendriks, 2016)
12 / 23
Background
Experiment
Results
Discussion
References
Model
• Response variable:
• Baselined Pupil Dilation (200–2000 ms)
13 / 23
Background
Experiment
Results
Discussion
References
Model
• Response variable:
• Baselined Pupil Dilation (200–2000 ms)
• Input variables
• Random smooths for Time by Subject
• Random smooths for Time by Item
• Smooth for Time by Phoneme-by-Reduction (4-level factor)
• Tensor product for Time by Lexical frequency
• Tensor product for Time by Trial
• Smooth for Gaze coordinates (X, Y)
• Smooth for Baseline dilation
• List
• Rho (autocorrelation parameter)
13 / 23
Background
Experiment
Results
Discussion
References
Comparison of reduction by phoneme
Reduced vs. Unreduced over Time by phoneme. Difference plots with 99%
confidence intervals.
14 / 23
Background
Experiment
Results
Discussion
References
Comparison of reduction by phoneme
U
500
1000
Time
1500
2000
0.04
Reduced /d/ − Unreduced /d/
−0.02
R
−0.05 0.05
Pupil Dilation
/d/: Reduced and Unreduced
Pupil Dilation Difference
Reduced vs. Unreduced over Time by phoneme. Difference plots with 99%
confidence intervals.
1309−2000 ms
500
1000
1500
2000
Time
14 / 23
Background
Experiment
Results
Discussion
References
Comparison of reduction by phoneme
U
1000
1500
2000
0.04
1309−2000 ms
500
1000
1500
/g/: Reduced and Unreduced
Reduced /g/ − Unreduced /g/
U
500
1000
Time
1500
2000
−0.02
0.05
R
2000
0.04
Time
Pupil Dilation Difference
Time
−0.05
Pupil Dilation
500
Reduced /d/ − Unreduced /d/
−0.02
R
−0.05 0.05
Pupil Dilation
/d/: Reduced and Unreduced
Pupil Dilation Difference
Reduced vs. Unreduced over Time by phoneme. Difference plots with 99%
confidence intervals.
1091−2000 ms
500
1000
1500
2000
Time
14 / 23
Background
Experiment
Results
Discussion
References
Comparison of phonemes by reduction
/d/ vs. /g/ over Time by reduction. Difference plots with 99% confidence
intervals
15 / 23
Background
Experiment
Results
Discussion
References
Comparison of phonemes by reduction
500
1000
Time
1500
2000
0.04
Reduced /d/ − Reduced /g/
−0.02
/d/
/g/
−0.05 0.05
Pupil Dilation
Reduced /d/ and /g/
Pupil Dilation Difference
/d/ vs. /g/ over Time by reduction. Difference plots with 99% confidence
intervals
500
1000
1500
2000
Time
15 / 23
Background
Experiment
Results
Discussion
References
Comparison of phonemes by reduction
1000
1500
2000
0.04
500
1000
1500
2000
Unreduced /d/ and /g/
Unreduced /d/ − Unreduced /g/
/g/
500
1000
Time
1500
2000
−0.02
0.05
/d/
0.04
Time
Pupil Dilation Difference
Time
−0.05
Pupil Dilation
500
Reduced /d/ − Reduced /g/
−0.02
/d/
/g/
−0.05 0.05
Pupil Dilation
Reduced /d/ and /g/
Pupil Dilation Difference
/d/ vs. /g/ over Time by reduction. Difference plots with 99% confidence
intervals
1600−1745 ms
500
1000
1500
2000
Time
15 / 23
Background
Experiment
Results
Discussion
References
Effect of reduction between phonemes
• Are the timing and size of difference between /d/ and /g/
significant?
16 / 23
Background
Experiment
Results
Discussion
References
Effect of reduction between phonemes
• Are the timing and size of difference between /d/ and /g/
significant?
• Post hoc analysis
• Calculated difference between Reduced and Unreduced by
word over time
16 / 23
Background
Experiment
Results
Discussion
References
Effect of reduction between phonemes
• Are the timing and size of difference between /d/ and /g/
significant?
• Post hoc analysis
• Calculated difference between Reduced and Unreduced by
word over time
• Modelled the difference curve by phoneme to determine
if/when they diverge from each other
16 / 23
Background
Experiment
Results
Discussion
References
Effect of reduction between phonemes
• Are the timing and size of difference between /d/ and /g/
significant?
• Post hoc analysis
• Calculated difference between Reduced and Unreduced by
word over time
• Modelled the difference curve by phoneme to determine
if/when they diverge from each other
• No significant difference between the curves
16 / 23
Background
Experiment
Results
Discussion
References
Reduction
• Dilation
• Results indicate that reduced forms (of both /d/ and /g/)
elicit greater pupillary response
17 / 23
Background
Experiment
Results
Discussion
References
Reduction
• Dilation
• Results indicate that reduced forms (of both /d/ and /g/)
elicit greater pupillary response
• This mirrors reaction time results obtained by Tucker
(2011), indicating an increased processing load is incurred
for reduced forms
17 / 23
Background
Experiment
Results
Discussion
References
Reduction
• Dilation
• Results indicate that reduced forms (of both /d/ and /g/)
elicit greater pupillary response
• This mirrors reaction time results obtained by Tucker
(2011), indicating an increased processing load is incurred
for reduced forms
• Timing
• Difference between reduced and unreduced forms arises
after 1000 ms (well after the average offset of the stimuli)
17 / 23
Background
Experiment
Results
Discussion
References
Reduction
• Dilation
• Results indicate that reduced forms (of both /d/ and /g/)
elicit greater pupillary response
• This mirrors reaction time results obtained by Tucker
(2011), indicating an increased processing load is incurred
for reduced forms
• Timing
• Difference between reduced and unreduced forms arises
after 1000 ms (well after the average offset of the stimuli)
• This persists through the remainder of the time course
indicating a long lasting effect
17 / 23
Background
Experiment
Results
Discussion
References
Reduction by phoneme
• No (or very little) difference found between /d/ and /g/
within reduced or unreduced forms (similar to Tucker,
2011)
18 / 23
Background
Experiment
Results
Discussion
References
Reduction by phoneme
• No (or very little) difference found between /d/ and /g/
within reduced or unreduced forms (similar to Tucker,
2011)
• Post hoc analysis did not support greater dilation or timing
differences of the effect of reduction between phonemes
18 / 23
Background
Experiment
Results
Discussion
References
Reduction by phoneme
• No (or very little) difference found between /d/ and /g/
within reduced or unreduced forms (similar to Tucker,
2011)
• Post hoc analysis did not support greater dilation or timing
differences of the effect of reduction between phonemes
• While word-medial /d/ is realized as a flap, dilation and
timing do not appear sensitive to this flapping process.
18 / 23
Background
Experiment
Results
Discussion
References
Take-home messages
• Pupillometry is a tool that can inform us about the
processing of reduced speech
19 / 23
Background
Experiment
Results
Discussion
References
Take-home messages
• Pupillometry is a tool that can inform us about the
processing of reduced speech
• Processing load increased for reduced forms
19 / 23
Background
Experiment
Results
Discussion
References
Take-home messages
• Pupillometry is a tool that can inform us about the
processing of reduced speech
• Processing load increased for reduced forms
• Little evidence of differences between phonemes
19 / 23
Background
Experiment
Results
Discussion
References
Take-home messages
• Pupillometry is a tool that can inform us about the
processing of reduced speech
• Processing load increased for reduced forms
• Little evidence of differences between phonemes
• Opens possibility of using pupillometry for studying
reduction processing in a broader (e.g., sentential) context
19 / 23
Background
Experiment
Results
Discussion
References
Take-home messages
• Pupillometry is a tool that can inform us about the
processing of reduced speech
• Processing load increased for reduced forms
• Little evidence of differences between phonemes
• Opens possibility of using pupillometry for studying
reduction processing in a broader (e.g., sentential) context
Thank you.
Questions?
19 / 23
Background
Experiment
Results
Discussion
References
References I
Beatty, (1982). Task-evoked pupillary responses, processing
load, and the structure of processing resources.
Psychological Bulletin, 91(2), 276–292.
Connine, Ranbom, & Patterson, (2008). Processing variant
forms in spoken word recognition: The role of variant
frequency. Perception & Psychophysics, 70(3),
403–411.
Engelhardt, Ferreira, & Patsenko, (2010). Pupillometry reveals
processing load during spoken language comprehension.
The Quarterly Journal of Experimental Psychology,
63(4), 639–645.
20 / 23
Background
Experiment
Results
Discussion
References
References II
Klingner, Tversky, & Hanrahan, (2010). Effects of visual and
verbal presentation on cognitive load in vigilance,
memory, and arithmetic tasks. Psychophysiology, 48(3),
323–332.
Koch, & Janse, (2016). Speech rate effects on the processing
of conversational speech across the adult life span. The
Journal of the Acoustical Society of America, 139(4),
1618–1636.
Kuchinke, Võ, Hofmann, & Jacobs, (2007). Pupillary responses
during lexical decisions vary with word frequency but not
emotional valence. International Journal of
Psychophysiology, 65(2), 132–140.
McLennan, Luce, & Charles-Luce, (2003). Representation of
lexical form. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 29(4), 539–553.
21 / 23
Background
Experiment
Results
Discussion
References
References III
R Development Core Team. (2016). R: A language and
environment for statistical computing. Version 3.3.0. R
Foundation for Statistical Computing. Vienna, Austria.
Ranbom, & Connine, (2007). Lexical representation of
phonological variation in spoken word recognition.
Journal of Memory and Language, 57(2), 273–298.
Tamási, McKean, Gafos, Fritzsche, & Höhle, (in press).
Pupillometry registers toddlers’ sensitivity to degrees of
mispronunciation. Journal of Experimental Child
Psychology.
Tucker, (2011). The effect of reduction on the processing of
flaps and /g/ in isolated words. Journal of Phonetics,
39(3), 312–318.
22 / 23
Background
Experiment
Results
Discussion
References
References IV
van Rij, Hollebrandse, & Hendriks, (2016). Children’s eye gaze
reveals their use of discourse context in object pronoun
resolution. In Holler, Goeb, & Suckow (Eds.), Empirical
perspectives on anaphora resolution.
Wood, (2016). mgcv: Mixed GAM computation vehicle with
GCV/AIC/REML smoothness estimation. R package
version 1.7-29.
Zekveld, & Kramer, (2014). Cognitive processing load across a
wide range of listening conditions: Insights from
pupillometry. Psychophysiology, 51(3), 277–284.
Zekveld, Kramer, & Festen, (2010). Pupil response as an
indication of effortful listening: The influence of sentence
intelligibility. Ear and Hearing, 31(4), 480–490.
23 / 23