Listening effort, and the cognitive demands
imposed by noisy and ambiguous speech
Ingrid Johnsrude
School of Communication Sciences and Disorders, and Department of Psychology
Brain and Mind Institute
Western University
London ON Canada
“listening effort” or “effortful listening” or “ease of
listening” on PubMed: 1980-2016
Number of papers (per million)
45
40
35
30
25
20
15
10
5
0
1980 2001 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Year
2000 2005
“auditory scene analysis”: 41
“hearing impairment” or deafness: 1160
hypertens
What is “listening effort”?
Interaction between processing demands and cognitive abilities.
There is not ONE type of listening effort; but MANY.
Johnsrude, IS & Rodd JM (2015). Factors that increase the processing demands when
listening to speech. In Neurobiology of Language (G Hickok and S Small, eds), Elsevier.
What are “processing demands”?
E.g., Linguistic demands:
1. Words with multiple meanings
What are “processing demands”?
E.g., Linguistic demands:
1. Words with multiple meanings
“The shell was fired towards the tank”
”The star had many fans who came to her gigs”
“A spade was not the suit that the card shark wanted”
Maintaining multiple meanings in mind; meaning selection;
reinterpretation…
2. Using context
“I want to eat Grandpa Bunny’s hearing aids”
What are “processing demands”?
E.g., Linguistic demands:
1. Words with multiple meanings
“The shell was fired towards the tank”
”The star had many fans who came to her gigs”
“A spade was not the suit that the card shark wanted”
Maintaining multiple meanings in mind; meaning selection;
reinterpretation…
2. Using context
Visual perception, selective attention, inhibition, episodic
memory…
3. Complexity (syntactic and otherwise)
Linguistic hierarchical structure; mapping pronouns onto
antecedents; metaphor; visual imagery....
Different demands recruit
different processes Background
noise
In Noise Clear
• High-ambiguity sentences
“the shell was fired towards the tank”
at least 2 ambiguous words
• Low-ambiguity sentences
“her secrets were written in her diary”
Johnsrude, Rodd, & Davis (in prep)
Clear
High-ambiguity sentences are less
intelligible in noise
High-ambiguity
Low-ambiguity
Word report percent correct
100
98
96
94
92
90
88
86
84
82
80
Clear
Johnsrude, Rodd, & Davis (in prep)
-2 dB SNR
Ambiguity and noise
recruit different networks
that are largely disjoint
High vs low ambiguity sentences
Noisy sentences vs clear sentences
Both effects present
Amb x
High Amb > Noise > Clear
Clarity
interaction Low Amb
Johnsrude, Rodd, & Davis (in prep)
Listening to sentences (must!) recruit
different processes compared to single
words…
People typically remember degraded speech worse than clear speech
even after intelligibility differences are controlled
Rabbitt, 1966,1968, 1990; Pichora-Fuller et al, 1995, 1996; Brown and Pichora-Fuller 2000; Murphy,
Craik et al, 2000, McCoy, Tun,et al, 2005; Heinrich et al 2008, 2010, 2011; Surprenant, 1999; Piquado
et al, 2010; Cousins et al 2014.
“Effortfulness hypothesis”
The extra effort required to achieve perceptual success (when speech is
degraded, or masked, or the listener has hearing loss) requires processing
resources that might otherwise be available for encoding the speech content in
memory. Rabbitt, 1968; Wingfield, Tun, & McCoy, 2005.
Memory for degraded sentences
3 Speech Types
2.
3.
Clear speech
High-intelligibility NV
speech (6-band)
Even higher intelligibility
12-band NV speech
27 participants in pilot
18 participants in fMRI
90
% understood (gist)
1.
100
80
70
60
50
40
30
20
10
0
Clear
Ritz et al, in preparation
NV12
NV6
Memory for degraded sentences
3 Speech Types X 2 Attention conditions
1.
Clear speech
1.
Attend speech
2.
High-intelligibility NV
speech (6-band)
2.
Track highlighted dots
(ignore speech)
3.
Even higher intelligibility
12-band NV speech
27 participants in pilot
18 participants in fMRI
Ritz et al, in preparation
Results
Attend Speech
Pilot (n=27)
Attend MOT
p = .00019
p = .0087
1.6
Sensitivity d’
1.4
1.2
1
0.8
0.6
0.4
0.2
0
Clear
NV12
NV6
Speech Type
Ritz et al, in preparation
Results
Attend Speech
Pilot (n=27)
Attend MOT
p = .00019
p = .0087
1.6
Sensitivity d’
1.4
1.2
1
0.8
0.6
0.4
0.2
0
Clear
NV12
NV6
Speech Type
Ritz et al, in preparation
Results
Attend Speech
Pilot (n=27)
p = .00019
fMRI (n=18)
p = .018
p = .0087
1.6
p = .043
1.6
1.4
1.4
1.2
Sensitivity d’
Attend MOT
1.2
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
Clear
NV12
NV6
0
Speech Type
Ritz et al, in preparation
Clear
NV12
NV6
What is “listening effort”?
Interaction between processing demands and cognitive abilities.
Different utterances make different demands.
Recruit different cognitive processes; different brain areas.
These have different “downstream” effects (i.e., on memory)
Affordances
What is “listening effort”?
Interaction between processing demands and cognitive abilities.
Every person has their own
unique constellation of
cognitive strengths
and weaknesses
How do we ‘carve
up’ cognition?
How do we ‘carve up’ cognition?
E.g., Fluid and crystallized intelligence, executive
functions (inhibiting, shifting, updating); memory
(episodic/procedural/semantic/WM )
What is “listening effort”?
Weak
Strong
Johnsrude, IS & Rodd JM (2015). Factors that increase the processing demands when
listening to speech. In Neurobiology of Language (G Hickok and S Small, eds), Elsevier.
Weak
What is “listening effort”?
Strong
Weaker
cognitive ability
Stronger
cognitive ability
Listening effort
Interaction between:
processing demands and
cognitive abilities (individual differences)
Johnsrude, IS & Rodd JM (2015). Factors that increase the processing demands when
listening to speech. In Neurobiology of Language (G Hickok and S Small, eds), Elsevier.
Conclusion: Listening effort
Subjectively, listening effort may ‘feel’ unitary, and we assess it with
unidimensional methods (pupillometry, questionnaires; LIFG activity).
BUT
It is due to different demands placed on different cognitive abilities.
“Affordances”
Individuals differ in their abilities (cognitive profiles)
The subjective feeling of effort is due to the demands made by an utterance,
AND to the individual’s ability to meet that particular demand.
Many different (neuroanatomically dissociable) kinds of ‘listening effort’.