Auditory Perceptual Simulation: Effects
on Language Processing and
Comprehension
Peiyun Zhou
University of Illinois Urbana-Champaign
Beckman Institute for advanced Science and Technology
LING 575 Topics in Computational Linguistics
Feb 9th, 2017
1
22222222222
2
Harry Potter says:
“Give it here, Malfoy, or I’ll kick you off your broom!”
3
“Give it here, Malfoy, or I’ll kick you off your broom!”
4
Auditory Perceptual Simulation
Auditory Perceptual Simulation (APS) refers to
the phenomenon when readers mentally
simulate characteristics of either the voices of
the characters depicted in texts or the voices of
other speakers (including their own) while they
read silently.
(Hubbard, 2010; Stites, Luke, & Christianson, 2013; Yao &
Scheepers, 2011; Zhou & Christianson, 2016a; 2016b)
5
Background
6
Background
Eye movements of readers are modulated by the speech rate
ascribed to direct quotations during silent reading
(Kosslyn & Matt, 1977; Stites, Luke, & Christianson, 2012; Yao & Scheepers, 2011)
e.g. John walked into the room and said quickly,
“I found my car keys”
Faster reading speed
e.g. John walked into the room and said slowly,
“I found my car keys”
Slower reading speed
7
Auditory Perceptual Simulation (APS) Paradigm
(Zhou & Christianson 2016a)
“xiaofu”
+
“The bird ate the worm was small.”
The bird ate the worm. The bird was small. (Y/N)
8
Background
•
•
Readers’ can perceptually simulate different speakers’s
voices during silent reading (native vs. non-native)
APS affected reading comprehension
a. “The bird that ate the worm was small.” (SRC-plausible)
b. “The worm that the bird ate was small.” (ORC-plausible)
c. “The worm that ate the bird was small.” (SRC-implausible)
d. “The bird that the worm ate was small.” (ORC-implausible)
(Zhou & Christianson 2016a; 2016b)
9
Background
•
Imagine a faster native speech led to faster online silent
reading speed compared to imagine a slower non-native
speech
•
Compared to normal silent reading, APS of either native or
non-native speech
•
•
•
Increase online reading speed
Generate better comprehension (e.g. 20% increase in accuracy)
Deepen sentence processing
10
(Zhou & Christianson 2016a; 2016b)
Event-RelatedPotentials(ERPs)
11
N400 Effects
-
Negative
+
Postive
13
N400 Effects
- Negative
+ Postive
14
P600 Effects
-
+
15
Research questions
1. How similar is perceptual simulation to listening?
2. How does perceptual simulation affect readers’ processing
of grammatical errors in sentences?
3. Does it depend on whether the simulated voice is a native
or a non-native speaker? Hanulíková et al 2012)
16
Native Speech
(Edward,1977
Non-Native Speech
Gass & Varonis, 1984; Lippi-Green,1997; Vornik et al, 2003)
17
“Elephants climb trees.”
(Lev-Ari & Keysar, 2010)
18
“Elephants climb trees.”
(Lev-Ari & Keysar, 2010)
19
Hanulíková et al 2012
20
Study Design
Two ERP experiments
1. E1 examines no APS during silent reading
2. E2 explores APS of native and non-native speech
during silent reading
21
Prediction
APS = “Listening”: APS of native speech should
elicit P600 when reading grammatical errors, but
APS of non-native speech should not (Hanulíková et al
2012; van Goch, & Weber, 2012; Zhou & Christianson, 2016).
22
E1 Materials
Grammaticality
•
•
Error types
Correct
•
Subject Verb Disagreement
Incorrect
•
Pronoun Case Mismatch
1.a The carpenters chat when they sand the wood. (Grammatically correct)
1.b The carpenters chats when they sand the wood. (Subject verb Disagreement)
1.c The carpenters chat when them sand the wood. (Pronoun case mismatch)
23
Motivation
1. Pronoun case mismatch is
more salient than subject-verb
disagreement.
2. Native Chinese speakers
often make subject-verb
agreement errors; there is no
agreement of any kind in
Chinese.
Coulson, King, and Kutas (1998)
24
E1 Paradigm
+
Presentation rate:
The
350ms/ word
50ms/blank screen
Carpenters
500ms/ fixation cross
1000ms/ end of the sentence
………
Paraphrase
(Yes/No)
25
No APS Pronoun
No APS Verb
—Ungrammatical
—Gramatical
26
E1 Summary
•
Both types of errors generate P600 effects
•
Pronouns > Verbs
27
Experiment 2
Grammaticality
•
Correct vs. Incorrect
Error types
•
Subject verb disagreement vs. Pronoun mismatch
Speech
•
Native English speech vs. Non-native speech
28
E2 APS paradigm
Familiarize readers with two voices
—One faster native English speaker
—One slower non-native English speaker (with Chinese accent)
Native speaker’s speech rate (2 min.50 s) was faster than
non-natives (4 min.15 s.)
Readers heard each speaker’s voice (~500 word text)
while viewing their photos
--Once before reading, once in middle of experiment.
29
E2 Procedure
SOA= 400ms/ word
“xiaofu”
+
“The
……..
Paraphrase (Yes/No)
30
Native Speaker
31
Non-Native Speaker
32
One Speaker’s Photo
33
Sentence
“The carpenters chat when they sand the wood.”
34
Paraphrase Verification
The bird ate the worm. The worm was small. (Yes / No)
35
Social Attractiveness Survey
•
An example
Accent of the native English speaker
(1=no foreign accent at all, 7=very strong foreign accent)
1 2 3 4 5 6 7
•
We ensured that participants perceived the differences in
speech rate and accent between the two speakers
--Native speaker rated as faster 6.8 vs. 1.2 (on 7-point Likert scale)
--Non-native speaker rated as having stronger accent 6.7 vs. 1.1
36
Native APS Pronoun
No APS Pronoun
—Ungrammatical —Gramatical
37
No APS Verb
Native APS Verb
—Ungrammatical —Gramatical
38
(uv)
Grammaticality effects in No-APS and Native APS
39
(uv)
Grammaticality effects in No-APS and Native APS
40
No APS vs. Native APS
Summary
Native APS does not make the grammaticality effects
bigger.
How about the Non-native APS?
41
Non-Native APS Pronoun
Native APS Pronoun
Native APS
—Ungrammatical —Gramatical
42
Non-Native APS
- - -Ungrammatical - - -Gramatical
Native APS Verb
Native APS
—Ungrammatical —Gramatical
Non-native APS Verb
43
Non-Native APS
- - -Ungrammatical - - -Gramatical
(uv)
Grammaticality effects between no-APS, native APS,
and non-native APS
45
(uv)
Grammaticality effects between no-APS, native APS,
and non-native APS
46
Hanulíková et al
Non-native APS Verb
47
Non-Native APS
- - -Ungrammatical - - -Gramatical
Summary
• APS ≈ listening
• See additional evidence from fMRI
(Yao, Berlin, & Scheepers, 2011)
48
Thank you!
Thank you!
UIUCEdPsychPsycholinguisticsLab
UIUCLanguageandBrianLab
Dr.JosephToscano
KeqiWei
KatharineTyndall
ShaolingyunGuo
YoungJaeLee
Dr.KielChristianson
Dr.SusanGarnsey
50
Individual Differences
E1
E2
225
words/min
267
words/min
NDRT
Comprehension
77.9%
81.5%
Auditory
Imagery
71/98
72/98
NDRT
Reading Speed