Brain Products Press Release
July 2012, Volume 43
User Research
When processing the melody of speech is atypical: ERP insights into emotional prosody
processing in Williams syndrome
by Ana Patrícia Teixeira Pinheiro 1,2
1
Neuropsychophysiology Laboratory, School of Psychology, University of Minho, Braga, Portugal
2
Cognitive Neuroscience Laboratory, Department of Psychiatry, Harvard Medical School, Boston, USA
Williams syndrome is a genetic neurodevelopmental disorder
resulting from a submicroscopic deletion of approximately
1.6 Mb including 24-28 genes on the long arm of chromosome
7 (7q11.23) (Ewart et al., 1993). This syndrome is characterized
by an uneven cognitive profile and intellectual disability
(Martens, Wilson, & Reutens, 2008). In spite of early claims
proposing the modular preservation of language in WS
coexisting with severe deficits in visuospatial abilities, a
large body of recent research has questioned these claims
and demonstrated that language abilities of individuals with
Williams syndrome are in line with their general cognitive
abilities. Abnormalities have been shown in specific language
subcomponents, including deficits in the recognition of
emotion in speech stimuli, i.e. emotional prosody (Catterall
et al., 2006; Plesa-Skwerer et al., 2006, 2007).
Auditory emotional processing: the example of emotional
prosody
Of note, emotional prosody represents the non-verbal vocal
expression of emotion (e.g. Wildgruber, Ackermann, Kreifelts
& Ethofer, 2006). At the perceptual and physical levels,
emotional prosody is conveyed primarily by means of pitch,
intensity, duration and timbre (Juslin & Laukka, 2003; Scherer
& Oshinsky, 1977; Schirmer & Kotz, 2006). Schirmer and
Kotz argue that three main stages are involved in
indexing emotional information from an acoustic signal.
The first stage occurs around 100 ms, when the sensory
Fig. 1: stages of emotional prosody
processing of acoustic cues takes place. The second stage
of emotional prosody processing occurs around 200 ms,
corresponding to the extraction of emotional significance
from acoustic cues (see also Paulmann & Kotz, 2008;
Paulmann, Seifert, & Kotz, 2010). The third stage (around
www.brainproducts.com
400 ms) is associated with higherorder cognitive processes that allow
emotional
significance
to
be
Ana Pinheiro
assigned to the output of sensory
processes,
including the integration of the emotional
significance of acoustic cues with semantic information.
Importantly, event-related potential (ERP) studies on emotional
prosody processing have indicated the role of the P200
component as an index of the detection of emotional
salience from speech stimuli, i.e. the second stage
of emotional prosody processing (Paulmann & Kotz,
2008; Paulmann et al., 2010). However, a recent study
demonstrated that the sensitivity to emotional salience can
occur as early as 100 ms and is indexed by N100 amplitude
modulated by the emotional valence of non-verbal
vocalizations (Liu et al., 2012).
Impaired assignment of emotional salience to a speech signal:
the example of Williams syndrome
The ability to successfully recognize emotional prosody
in speech stimuli is one of the cornerstones of effective
functioning in social environment. Even though the
engaging style of conversation frequently reported in
Williams syndrome (WS) suggests that social communication
is a relative strength in this disorder (e.g. Gonçalves et al.,
2010), behavioral studies on emotional prosody recognition
in WS demonstrate that emotional prosody recognition is
disrupted in WS when compared with typically developing
controls (Catterall et al., 2006; Plesa-Skwerer et al., 2006,
2007). However, individuals with WS seem to perform better
than participants with learning or intellectual disabilities on
the recognition of emotional tone of voice in filtered speech
(Plesa-Skwerer et al., 2006), suggesting that sensitivity to
non-linguistic affective information may be less affected in WS.
Notably, no event related potentials (ERP) studies of prosody
processing have been conducted in WS. While reaction time
measures and error data are indirect measures of all processes
that took place before the response was made, ERPs probe
the course of neurocognitive processes before a response
is made or even in its absence. Together they make it possible
to arrive at a more complete understanding of all cognitive
processes involved and their putative abnormalities.
It is not clear how lexical and supra-segmental features of
speech signal may interact to convey emotion or how such
processes differ from the processes of extracting emotional
Brain Products Press Release
information from a speech signal that carries suprasegmental information alone, in typically developing
controls and WS. In our ERP study we examined the temporal
course of emotional prosody processing in WS, and
investigated the role of semantic information in different
stages of vocal emotional processing in WS. The study design,
subjects and results have been described in detail previously
(Pinheiro et al., 2011), in Research in Developmental
Disabilities.
We hypothesized that all stages of vocal emotional processing
would be impaired in WS, i.e. abnormalities in N100 and P200
components and reduced accuracy in recognition of emotional
prosody.
Twelve participants with WS (M ± SD = 17.3 ± 6.5 years) and a
group of typically developing subjects individually matched
for chronological age and gender were presented with
neutral, positive and negative sentences in two conditions:
(1) sentences with intelligible semantic information (‘semantic
content condition’ – SCC); (2) sentences with unintelligible
semantic information (‘pure prosody’ condition – PPC). Five
of these participants (2 typically developing and 3 WS
individuals) were excluded from final statistical analyses due
to excessive artifacts.
Fig. 2: SSC-PPS
Stimuli were 216 auditory sentences (108 SCC sentences and
108 PPC sentences). In each sentence condition, 36 sentences
had a happy intonation, 36 an angry intonation, and 36 a
neutral intonation. Of note, in the PPC, the lexical-semantic
information was absent due to the systematic distortion of
the acoustic signal (as described in Pinheiro et al., 2011 and
in Pinheiro et al., in press). However, even though all lexical
www.brainproducts.com
July 2012, Volume 43
and semantic information was suppressed by substitution of
phones using Praat (http://www.fon.hum.uva.nl/praat/) and
MBROLA (http://mambo.ucsc.edu/psl/mbrola/) software, the
prosodic modulations of the original sentences were kept.
In order to minimize cognitive demands, sentence conditions
were not randomized within each experimental block. We
first presented all sentences with intelligible semantic
information (SCC), followed by all transformed sentences
(PPC). We posited that switching between the two conditions
in the same block could enhance cognitive demands,
introducing a confounding variable in the results (as pointed out
by Kotz et al., 2003) and in addition that switching between
the two conditions within the same block could create a
possible influence of normal speech on prosodic speech.
After listening to each sentence, participants were asked
to decide the emotional intonation underlying each sentence
they have heard. While the participants listened to the
sentences and decided the associated vocal emotion,
the electroencephalogram (EEG) was recorded using
QuickAmp EEG recording system (Brain Products, Munich,
Germany), with 22 Ag-AgCl electrodes. Averages were
computed using a 200-ms prestimulus baseline, timelocked to the sentence onset and spanning the length of
a sentence (1500 ms after the onset of the sentence). This
approach was adopted following all existing ERP studies of
emotional prosody processing using non-spliced sentences
(Paulmann & Kotz, 2008b; Paulmann et al., 2009) in order
to assure the comparability of our results with previous
studies. Although prosody is a supra-segmental feature of
a speech signal not locked particularly to any given word,
in the case of our design, the first word carried most of
the prosodic information related to the emotional significance
of an utterance. Time locking to the onset of the sentence,
and thus to this first word, and extending the epoch across
the entire sentence insured that all possible significant shifts
in prosody and their underlying neural events were captured.
After careful visual inspection of grand averages, three
peaks were identified and selected for analysis: N100 (SSC:
100-200 ms; PPS: 100-160 ms), P200 (SSC: 200-320 ms;
PPS: 160-260 ms) and N300 (SSC: 320-450 ms; PPS: 280380 ms). Different latency windows were selected for SSC and
PPS after a careful inspection of grand average waveforms
that indicated earlier ERP effects for PPS relative to
SSC. Interestingly, in this study we observed a third ERP
component (N300) that was not reported in the study of
Paulmann and Kotz (2008) and Paulmann et al. (2010). This
may be due to linguistic differences in the stimuli used in
our experiment and in those previous studies (European
Portuguese in the current study, and German in the study of
Paulmann and Kotz and Paulmann et al.). The EEG data
were analyzed using the software package Brain Analyzer
Brain Products Press Release
July 2012, Volume 43
1.05.0005 (Brain Products, Munich, Germany).
al. (1993). Hemizygosity at the elastin locus in a developmental disorder,
ERP results indicated group differences at the level of N100,
P200, and N300, suggesting differential effects of sentence
condition and emotion. When compared with typically
developing controls, WS individuals showed reduced N100
for SCC sentences, indicating a differential sensory processing
of speech carrying simultaneously semantic information
(SCC) and speech with no intelligible semantic information
(PPC), and suggesting that semantic information present in
non-distorted speech signal may interact with early sensory
processes making it more difficult for WS individuals to process
‘normal speech’ relative to ‘pure prosody’ speech signal.
Williams syndrome. Nature Genetics, 5(1), 11-16.
Also, more positive P200 was observed for SCC sentences,
in particular for emotional intonations (happy and angry),
in WS relative to typically developing controls, suggesting
abnormal detection of emotional salience in vocal stimuli,
and indicating that these abnormalities are dependent on the
presence of semantic information.
Gonçalves, O. F., Pinheiro, A. P., Sampaio, A., Sousa, N., Férnandez,
M., & Henriques, M. (2010). The narrative profile in Williams Syndrome:
There is more to storytelling than just telling a story. The British Journal of
Developmental Disabilities, 56(111), 89-109.
Juslin, P. N., & Laukka, P. (2003). Communication of emotions in vocal
expression and music performance: different channels, same code? Psychol
Bull, 129(5), 770-814.
Liu, T., Pinheiro, A., Zhao, Z., Nestor, P. G., McCarley, R. W., & Niznikiewicz,
M. A. (2012). Emotional cues during simultaneous face and voice processing:
electrophysiological insights. PLoS One, 7(2), e31001.
Martens, M. A., Wilson, S. J., & Reutens, D. C. (2008). Research Review:
Williams syndrome: a critical review of the cognitive, behavioral, and
neuroanatomical phenotype. Journal of Child Psychology and Psychiatry,
49(6), 576-608.
Neville, H. J., Mills, D. L., Bellugi, U., & Hillsdale, N. J. E., 1994. (1994).
The reduced N300 for both types of sentence conditions
(SCC and PPC) in WS relative to typically developing controls
also suggested abnormal processing of the emotional
significance of the acoustic signal, irrespective of the semantic
status of the sentences.
Eff_ects of altered auditory sensitivity and age of language acquisition
Finally, behavioral results demonstrated higher error rates in
the WS group relative to typically developing controls but only
for angry prosody.
Paulmann, S., & Kotz, S. A. (2008). Early emotional prosody perception
on the development of language-relevant neural systems: Preliminary
studies of Williams syndrome. In S. Broman & J. Grafman (Eds.), Cognitive
Deficits in Developmental Disorders: Implications for Brain Function (6783). Hillsdale, NJ: Erlbaum.
based on different speaker voices. Neuroreport, 19(2), 209-213.
Pinheiro, A. P., Galdo-Alvarez, S., Rauber, A., Sampaio, A., Niznikiewicz,
M., & Goncalves, O. F. (2011). Abnormal processing of emotional prosody
Conclusions
This study was able to demonstrate that abnormalities in ERP
measures of early auditory processing in WS (e.g. Neville et
al., 1994; Pinheiro et al., 2010) are also present during the
processing of emotional prosody information, spanning the
three stages of vocal emotional processing (Schirmer & Kotz,
2006): extracting sensory information from the acoustic
signal (N100), detection of emotionally salient acoustic
cues (P200), and cognitive evaluation of the emotional
significance of stimuli (N300 and error rates).
Together, these findings suggest an interaction between
disrupted lower-level sensory processes and higher-order
cognitive processes in bringing about emotional prosody
processing abnormalities in WS. They further suggest that
the abnormal integration of perceptual and categorical
information with the conceptual knowledge of negative
emotion (angry prosody) may play an important role in
inefficient negative emotion recognition.
in Williams syndrome: an event-related potentials study. Research in
Developmental Disabilities, 32(1), 133-147.
Pinheiro, A. P., Galdo-Alvarez, S., Sampaio, A., Niznikiewicz, M., &
Goncalves, O. F. (2010). Electrophysiological correlates of semantic
processing in Williams syndrome. Research in Developmental Disabilities,
31(6), 1412-1425.
Pinheiro, A.P., del Re, E., Mezin, J., Nestor, P.G., Rauber, A., McCarley,
R.W., Goncalves, O.F., & Niznikiewicz, M. (in press). Sensory-based and
higher-order operations contribute to abnormal emotional prosody in
schizophrenia:
an
electrophysiological
investigation.
Psychological
Medicine.
Plesa-Skwerer, D., Faja, S., Schofield, C., Verbalis, A., & Tager-Flusberg, H.
(2006). Perceiving facial and vocal expressions of emotion in individuals
with Williams syndrome. American Journal of Mental Retardation, 111(1),
15-26.
Plesa-Skwerer, D., Schofield, C., Verbalis, A., Faja, S., & Tager-Flusberg,
H. (2007). Receptive prosody in adolescents and adults with Williams
syndrome. Language and Cognitive Processes, 22(2), 247-271.
Scherer, K. R., & Oshinsky, J. S. (1977). Cue utilization in emotion
References:
attribution from auditory stimuli. Motivation and Emotion, 1(4),
Catterall, C., Howard, S., Stojanovik, V., Szczerbinski, M., & Wells, B.
331-346.
(2006). Investigating prosodic ability in Williams syndrome. Clinical
Schirmer, A., & Kotz, S. A. (2006). Beyond the right hemisphere: brain
Linguistics and Phonetics, 20(7-8), 531-538.
Ewart, A. K., Morris, C. A., Atkinson, D., Jin, W., Sternes, K., Spallone, P., et
www.brainproducts.com
mechanisms mediating vocal emotional processing. Trends in Cognitive
Sciences, 10(1), 24-30.