Cognition and Emotion
ISSN: 0269-9931 (Print) 1464-0600 (Online) Journal homepage: http://www.tandfonline.com/loi/pcem20
On the malleability of the meaning of contexts: the
influence of another person’s emotion expressions
on situation perception
Ursula Hess & Shlomo Hareli
To cite this article: Ursula Hess & Shlomo Hareli (2016): On the malleability of the meaning
of contexts: the influence of another person’s emotion expressions on situation perception,
Cognition and Emotion, DOI: 10.1080/02699931.2016.1269725
To link to this article: http://dx.doi.org/10.1080/02699931.2016.1269725
View supplementary material
Published online: 21 Dec 2016.
Submit your article to this journal
Article views: 32
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=pcem20
Download by: [Université du Québec à Montréal]
Date: 15 January 2017, At: 05:08
COGNITION AND EMOTION, 2016
http://dx.doi.org/10.1080/02699931.2016.1269725
BRIEF ARTICLE
On the malleability of the meaning of contexts: the influence of another
person’s emotion expressions on situation perception
Ursula Hessa and Shlomo Harelib
a
Department of Psychology, Humboldt University of Berlin, Berlin, Germany; bDepartment of Business Administration, University
of Haifa, Haifa, Israel
ABSTRACT
ARTICLE HISTORY
Research on the relationship between context and facial expressions generally
assumes a unidirectional effect of context on expressions. However, according to
the model of the meaning of emotion expressions in context (MEEC) the effect
should be bidirectional. The present research tested the effect of emotion
expression on the interpretation of scenes. A total of 380 participants either (a)
rated facial expressions with regard to the likely appraisal of the eliciting situation
by the emoter, (b) appraised the scenes alone or (c) appraised scenes shown
together with the expressions they supposedly elicited. The findings strongly
supported the MEEC. When a scene was combined with an expression signalling a
situation that is undesirable, or high in locus of control or sudden, the participants
appraised the scene correspondingly. Thus, the meaning of scenes is malleable and
affected by the way that people are seen to react to them.
Received 4 May 2016
Revised 28 November 2016
Accepted 30 November 2016
Traditionally, research on the understanding of
emotion expressions has been conducted using carefully controlled “context-free” expressions (Hess &
Hareli, 2015). Apart from the fact that it is dubious
whether expressions can ever be fully devoid of
context given that the face already provides the
social group membership of the expresser as context
information (Hess, Adams, & Kleck, 2009; Hess &
Hareli, 2015), the importance of considering context
explicitly when studying emotion expressions has
been increasingly emphasised (e.g. Barrett, Mesquita,
& Gendron, 2011).
This context has been provided by a variety of
means (see also, Matsumoto & Hwang, 2010), such
as combining facial expressive stimuli with pictures
or stories describing the purported emotion elicitor
(Carroll & Russell, 1996; Noh & Isaacowitz, 2013;
Szczurek, Monin, & Gross, 2012). Social group information can be provided by adding information such
as a Niqab or a surgical mask to the facial information
(Fischer, Gillebaart, Rotteveel, Becker, & Vliek, 2012;
Hareli, David, & Hess, 2013; Kret & de Gelder, 2012)
Facial expression; emotion;
context
or by providing group labels (Thibault, Bourgeois, &
Hess, 2006). Alternatively, information from the face
has been contrasted with information from another
nonverbal channel – for example the body (Aviezer
et al., 2008). That is, there is a wide range of information that can be considered context. However,
what all of these have in common is the notion that
the meaning of context, especially situational
context, remains stable and it is the meaning of the
facial expression that is malleable. We submit that
this is not the case, rather the process of evaluating
emotion expressions in context, also allows for and
even demands a reevaluation of the context in light
of the expression.
This notion is based on the model of the meaning
of emotion expressions in context (MEEC; Hess &
Hareli, 2016). In this model, expressions are perceived
within a situational context (the real world) and then
interpreted within an interpretive context (the perceived world). Importantly, the model emphasises
the role of social or reengineered appraisals (Hareli &
Hess, 2010; Manstead & Fischer, 2001), that is, the
CONTACT Ursula Hess
[email protected]
Supplemental data for this article can be accessed at: http://dx.doi.org/10.1080/02699931.2016.1269725.
© 2016 Informa UK Limited, trading as Taylor & Francis Group
KEYWORDS
2
U. HESS AND S. HARELI
reconstructed appraisals of a situation that can be
deduced from the emotional reaction of another
person. This information may be used to adjust one’s
own reactions in light of different goals such as that
of maintaining a given relationship (e.g. Evers,
Fischer, Rodriguez Mosquera, & Manstead, 2005), but
it can also be used as information about the context
itself. If the information provided by the context
and the social appraisal of the expression are sufficiently congruent, then the emotion can be labelled
readily. In case of a mismatch the perceiver has to reevaluate the match explicitly. One outcome of this
process can be to change the meaning of the
expression as suggested by research on the malleability of expression perception (e.g. Aviezer et al., 2008;
Righart & De Gelder, 2008). Another outcome would
be to reevaluate the situation. That is, according to
the MEEC congruence is recreated by reinterpreting
either the situation or the facial expression or both.
That emotion expressions can influence the
interpretation of objects presented at the same time
or shortly afterwards can be adduced from research
on gaze-cuing. When eye-gaze is combined with
facial expressions, differential effects as a function of
emotion expressions have been found for (kitchen or
garage) objects (Bayliss, Frischen, Fenske, & Tipper,
2007) or for other faces (Mumenthaler & Sander,
2012, 2015) that the gaze was directed to (but not
for those that were not gazed at). These findings
suggest that when a person seems to pay attention
to an object or face, the facial expression of that
person gains informative value that influences subsequent ratings. However, in these studies the
environmental objects were either carefully chosen
to be neutral (Bayliss et al., 2007) or were also faces
(Mumenthaler & Sander, 2012, 2015) rather than
scenes.
The present research aimed to study the malleability of the meaning of potentially emotion eliciting
events depicted in scenes, that is, on whether participants would reinterpret the scene in line with a purported facial reaction to that scene.
Methods
Participants
For the main experiment, 318 (168 men, 1 other) participants with a mean age of 38 years (SD = 12), and
for the facial expression rating study, 62 participants
(37 men) with a mean age of 36 years (SD = 12) were
recruited via Amazon Mturk, completed the survey
and passed control questions probing for appropriate
diligence. To determine sample size, we aimed for a
minimum of 13 judges per cell, which for emotion
judgments typically assures acceptable interclass correlations. Data collection was stopped when this goal
was reached.
Materials
Facial expressions
Facial expressions of two positive and two negative
high arousal emotions: happiness, pride, disgust and
fear by two men and two women were taken from
the Amsterdam Dynamic Facial Expression Set
(ADFES) (Van der Schalk, Hawk, Fischer, & Doosje,
2011) resulting in a total of 4 (emotion) × 2 (gender) ×
2 (identities) = 16 faces. The ADFES provides images
for happiness and pride but no other positive
emotion expression. The rationale for the choice of
the negative emotion was a desire to match emotions
across valence and arousal. Disgust and fear are both
high arousal emotions (e.g. Russell, 1980) and hence
“match” the sole positive options best in that regard.
Specifically, we did not want to select sadness as the
only low arousal emotion (Russell, 1980). Anger was
not chosen because anger is a negative approach
emotion (Carver & Harmon-Jones, 2009).
Scenes
Four photos each showing emotion eliciting scenes
for happiness, pride, disgust and fear were taken
from the International Affective Picture System (IAPS,
Bradley & Lang, 2007) and the internet1 resulting in
a total of 16 scenes.
Procedure and dependent variables: main
experiment
Each participant completed a single trial. Following
the general design by Szczurek et al. (2012), participants saw a scene randomly combined with one of
the emotion expressions and were told that the
expression occurred in reaction to the scene (see
Figure 1 for an example). In the control condition, participants saw only the scene without any expression.
Each participant completed only one trial. Participants
in the experimental condition were first asked to rate
the degree to which the person showed happiness,
pride, disgust and fear on a series of scales anchored
with 0 – not at all and 6 – very intensely. In the
COGNITION AND EMOTION
3
Figure 1. Example for the arrangement of faces and scenes.
control condition they were asked to indicate the likelihood that the scene would elicit these emotions. This
served as a manipulation check for the meaning of the
scenes.
Participants were then asked to rate the scenes on
a series of 15 questions based on a short version of the
appraisal section of the GRID questionnaire (Fontaine,
Scherer, & Soriano, 2013). These questions cover all
appraisal dimensions (see supplementary materials
for the list of questions used in the main experiment).
Participants were asked whether the “situation that
caused the emotion” involved behaviour that, for
example, violated laws or socially accepted rules of
behaviour. The corresponding question for participants who saw only the context was “Did the situation
shown in the photo involve behavior that violated laws
or socially accepted rules of behavior?” All scales were
anchored with 0 – not at all and 6 – very much.
Because some of the items correlated substantially,
we conducted a principal component analysis that
yielded four factors, based on the scree plot, which
explained 61% of the variance. As factor 3 and 4 had
considerable cross-loadings and the resulting scales
had low internal consistencies, these two factors
were combined. This yielded three factors: negative
consequences (5 items, α = .73), internal locus of
control (6 items, α = .73) and suddenness (2 items,
α = .63). Two items did not load sufficiently on any
factor and were excluded.
They then were asked to reverse engineer the appraisals underlying the expression by answering the same
appraisal questions as above with the instruction to
assess to what degree the situation that caused the
emotion had occurred suddenly, was caused by the
expresser themselves, was pleasant, etc. The appraisals were combined into the same scales as for the
main experiment.
Results
Manipulation check
Planned contrasts were conducted to assess whether
the context scenes were considered to be likely to
elicit the target emotion when shown without an
expression (for means, standard deviations and
F-values, see Table 1). The fear and pride scenes
were rated as significantly more likely to elicit fear
and pride, respectively, than any other scene.
Disgust scenes were rated as more likely than happy
and pride scenes to elicit disgust, however, fear
scenes were also considered somewhat likely to
elicit disgust, most likely because the two snakes
used as stimuli can also be seen as disgusting.
Happy scenes were rated as most likely to elicit happiness, yet, pride scenes were also expected to elicit
happiness. This makes intuitive sense as most pride
events, such as winning a medal, also induce happiness. Thus, overall, the scenes were found to elicit
the expected emotions.
Procedure and dependent variables: facial
expression ratings
In addition, a second group of participants (N = 62, see
above) each rated one of the facial expressions, using
the same emotion scales as in the main experiment.
Facial expression ratings
To assess the reverse engineered appraisals of the presumed situations that elicited the facial expressions,
4
U. HESS AND S. HARELI
Table 1. Mean ratings, standard deviations and F-values for the perceived likelihood that the scenes elicit the target emotions as a function of
scene content.
Disgust
Fear
Joy
Pride
Scene
Rated Emotion
Mean
SD
Mean
SD
Mean
SD
Mean
SD
F
DF
p
Joy
Pride
Disgust
Fear
1.14
0.29
4.43
3.07
2.03
0.13
2.21
1.94
0.60
0.40
3.73
5.40
1.35
0.19
1.75
1.30
5.13
2.67
0.60
0.47
0.83
0.36
1.30
0.26
5.87
5.87
0.27
0.07
0.35
0.09
0.46
1.13
65.30
140.87
27.49
55.78
3,55
3,55
3,55
3,55
<0.001
<0.001
<0.001
<0.001
h2p
.78
.89
.60
.75
Note: Numbers in bold are significantly (p < .05) different from the remaining conditions.
one-way analyses of variance on the three appraisal
scales were conducted and followed up by post hoc
tests (for means and standard errors see upper panel
of Figure 2). Significant main effects of expression
emerged for suddenness, F(3,58) = 6.41, p = .001,
h2p = .25, internal locus of control (ILC), F(3,58) =
36.45, p < .001, h2p = .65, and undesirability of the situation, F(3,58) = 61.15, p < .001, h2p = .76. Post hoc tests
(p < .05) showed that for facial expressions of disgust
and fear, the eliciting situations were rated as higher
in undesirability and lower in internal locus of control
than situations that elicit happiness and pride. Situations that elicit fear were considered to be more
sudden than situations that elicit happiness and
pride, and those that elicit disgust as more sudden
than those that elicit pride. In sum, participants were
able to deduce the likely appraisals of the situation
by the emoter from the expression shown.
Hypothesis testing
To assess whether the social appraisal of the
expression shown in response to the scenes affected
the meaning of the scenes, a series of univariate analyses of variance with planned contrasts comparing
the control condition with the expression conditions
were conducted for each scene category (for means
and standard errors see Figure 2, for F-values see
Table 2, for means, standard deviations and confidence intervals see supplementary materials).
All ratings of scenes were affected by the emotion
expressions that they supposedly elicited. Overall, happiness and pride scenes were more affected than
disgust and fear scenes. The effect of the expressions
was very systematic and matched the reverse engineered appraisals based on the expression ratings. Compared to the control condition, disgust expressions
presented together with scenes increased appraisals
of undesirability and lowered appraisals of internal
locus of control for happy and pride scenes in line
with the higher ratings of reverse engineered undesirability and higher ratings of locus and control when the
expressions were appraised without context.
Similarly, fear expressions, which were appraised as
high in suddenness, increased appraisals of suddenness for all scenes except fear scenes; for happy and
pride scenes they also increased appraisals of undesirability and decreased appraisals of internal locus of
control in line with the appraisal of the faces alone.
Happy expressions by contrast increased appraisals
of internal locus of control for disgust and fear scenes,
as well as decreased appraisals of suddenness for fear
scenes, also in line with the appraisal of the faces
alone. Pride expressions only affected fear scenes
and did so in the same way as happy expressions.
Thus, overall, whenever an expression affected the
meaning of a scene it did so in line with the appraisal
of the face alone and in the same direction for all
scenes. This suggests that participants relied on the
social or reengineered appraisal of the accompanying
expression when appraising the scene.
Discussion
The present results provide strong evidence that the
perception of scenes is influenced by another
person’s facial expressions that were supposedly elicited by these scenes. Specifically, the reverse
Table 2. F-values, p-values and explained variance for appraisals of the scenes as a function of emotion expression: main experiment.
Undesirability of the situation
Scenes
Disgust
Fear
Happiness
Pride
F
1.37
1.56
4.20
15.70
DF
4,74
4,69
4,74
4,81
p
0.254
0.194
0.004
0.001
Internal locus of control
h2p
.07
.08
.19
.44
F
3.57
22.61
5.69
33.48
DF
4,74
4,69
4,74
4,81
p
0.010
0.001
0.001
0.001
Suddenness
h2p
.16
.57
.24
.62
F
4.80
9.89
2.39
6.72
DF
4,74
4,69
4,74
4,81
p
0.002
0.001
0.059
0.001
h2p
.21
.36
.11
.25
COGNITION AND EMOTION
5
Figure 2. Mean ratings and standard errors for appraisals of the scenes as a function of emotion expression: main experiment. Note: Stars indicate
means that are significantly different (p < .05) from non-starred means; ILC: internal locus of control.
6
U. HESS AND S. HARELI
engineered appraisals based on the facial expression
alone matched closely the effect of this expression
on the perception of a scene. When a scene was combined with an expression for which the reverse engineered appraisals assume an eliciting event that is
undesirable, or high in locus of control or sudden,
the participants appraised the scene correspondingly,
even when this was not the case when the scene was
shown alone. That is, participants relied on the
emotional facial expressions of others when evaluating the scene, even though they do not know these
others. As such, this process is not the same as adult
social referencing (Parkinson, Phiri, & Simons, 2012),
but may rather be one of the processes underlying
social referencing.
These findings strongly support the MEEC and the
notion that when participants are aware of both facial
expressions and an eliciting context they engage both
in an appraisal of the situation and in the social appraisal of the expression. When these two do not match,
they adjust their perceptions accordingly. However,
this adjustment also has limits. For example, the perception of disgust scenes was not as strongly affected
as the perception of happy and pride scenes. Thus,
disgust scenes were always rated as very undesirable
irrespective of the expression that accompanied
them, whereas happy and pride scenes were rated
as significantly more undesirable when accompanied
by disgust or fear expressions. This suggests that participants discounted the non-matching happy and
pride expressions when at the same time faced with
a disgusting scene. In fact, individuals who smile
when faced with disgusting stimuli are perceived as
affectively deviant to some degree (Szczurek et al.,
2012) and hence their expressive input is likely discounted. This suggests that disgust scenes might be
overall less malleable.
One factor that affects malleability is whether the
discrepancy between the social appraisal of the
expression and the appraisal of the scene can be
readily reconciled. The MEEC supposed that adjustment to the interpretation of either scenes or faces
are only required when the discrepancy is difficult to
reconcile. Cultural memes are one means of such
reconciliation. In this vein, in a recent study Hess
and Hareli (in press) found that when asked to
explain discrepant information from both sources,
people often recur to cultural memes. For example,
when a picture of a cute kitten is combined with an
emotional fear reaction, the cultural meme of fear of
cats is invoked. Thus, when a ready meme is available,
no further processing of either scene or facial information is required. In the same vein, if as suggested
by (Szczurek et al., 2012) individuals who do not
react with disgust to disgust scenes are perceived as
“deviant” sources of information, their input may be
discounted and no further reconciliation is required.
Future research should explore these limits of the
influence of social appraisals on context perception.
In sum, the present research provides strong evidence for the notion that not only does the context
in which facial expressions are shown impact the perception of these expressions (Aviezer et al., 2008;
Righart & De Gelder, 2008), but conversely, the social
appraisal of these facial expressions does in turn
impact on the perception of the context in which
they are shown. That is, neither facial expressions nor
context are immutable, instead both are malleable.
Note
1. 7380, 1945, 3250, 7360, 1051, 1113, 1302, 1930, 1920,
2655, 2791, 1463, for pride hands holding medals or trophies were shown.
Disclosure statement
No potential conflict of interest was reported by the authors.
References
Aviezer, H., Hassin, R., Ryan, J., Grady, C., Susskind, J., Anderson,
A., … Bentin, S. (2008). Angry, disgusted, or afraid? Studies
on the malleability of emotion perception. Psychological
Science, 19, 724–732.
Barrett, L. F., Mesquita, B., & Gendron, M. (2011). Context in
emotion perception. Current Directions in Psychological
Science, 20(5), 286–290.
Bayliss, A. P., Frischen, A., Fenske, M. J., & Tipper, S. P. (2007).
Affective evaluations of objects are influenced by observed
gaze direction and emotional expression. Cognition, 104(3),
644–653. doi:10.1016/j.cognition.2006.07.012
Bradley, M. M., & Lang, P. J. (2007). The International Affective
Picture System (IAPS) in the study of emotion and attention.
In J. A. Coan & J. J. B. Allen (Eds.), Handbook of emotion elicitation and assessment (pp. 29–46). New York, NY: Oxford
University Press.
Carroll, J. M., & Russell, J. A. (1996). Do facial expressions signal
specific emotions? Judging emotion from the face in context.
Journal of Personality and Social Psychology, 70, 205–218.
Carver, C. S., & Harmon-Jones, E. (2009). Anger is an approachrelated affect: Evidence and implications. Psychological
Bulletin, 135, 183–204.
Evers, C., Fischer, A. H., Rodriguez Mosquera, P. M., & A. S. R.
Manstead. (2005). Anger and social appraisal: A “spicy” sex
difference? Emotion, 5(3), 258–266. doi:10.1037/1528-3542.5.
3.258
COGNITION AND EMOTION
Fischer, A. H., Gillebaart, M., Rotteveel, M., Becker, D., & Vliek, M.
(2012). Veiled emotions: The effect of covered faces on
emotion perception and attitudes. Social Psychological and
Personality Science, 3, 266–273. doi:10.1177/1948550611418534
Fontaine, J. R. J., Scherer, K. R., & Soriano, C. (2013). Components of
emotional meaning: A sourcebook. Oxford: Oxford University
Press.
Hareli, S., David, S., & Hess, U. (2013). Competent and warm but
unemotional: The influence of occupational stereotypes on
the attribution of emotions. Journal of Nonverbal Behavior,
37, 307–317. doi:10.1007/s10919-013-0157-x
Hareli, S., & Hess, U. (2010). What emotional reactions can tell us
about the nature of others: An appraisal perspective on
person perception. Cognition and Emotion, 24, 128–140.
Hess, U., Adams, R. B., Jr., & Kleck, R. E. (2009). The face is not an
empty canvas: How facial expressions interact with facial
appearance. Philosophical Transactions of the Royal Society
London B, 364, 3497–3504.
Hess, U., & Hareli, S. (2015). The role of social context for the
interpretation of emotional facial expressions. In M. K.
Mandal & A. Awasthi (Eds.), Understanding facial expressions
in communication (pp. 119–141). New York, NY: Springer.
Hess, U., & Hareli, S. (2016). The impact of context on the perception of emotions. In C. Abell & J. Smith (Eds.), The expression of
emotion: Philosophical, psychological, and legal perspectives
(pp. 199–218). Cambridge: Cambridge University Press.
Hess, U., & Hareli, S. (in press). The social signal value of emotions:
The role of contextual factors in social inferences drawn from
emotion displays. In J. Russell & J.-M. Fernandez-Dols (Eds.),
The psychology of facial expression.
Kret, M. E., & de Gelder, B. (2012). Islamic headdress influences
how emotion is recognized from the eyes. Frontiers in
Psychology, 3, 110. doi:10.3389/fpsyg.2012.00110
Manstead, A. S. R., & Fischer, A. H. (2001). Social appraisal: The
social world as object of and influence on appraisal processes.
7
In K. R. Scherer, A. Schorr, & T. Johnstone (Eds.), Appraisal processes in emotion: Theory, methods, research (pp. 221–232).
New York, NY: Oxford University Press.
Matsumoto, D., & Hwang, H. S. (2010). Judging faces in
context. Social and Personality Psychology Compass, 4(6),
393–402.
Mumenthaler, C., & Sander, D. (2012). Social appraisal influences
recognition of emotions. Journal of Personality and Social
Psychology, 102, 1118–1135.
Mumenthaler, C., & Sander, D. (2015). Automatic integration of
social information in emotion recognition. Journal of
Experimental Psychology: General, 144(2), 392–399.
Noh, S. R., & Isaacowitz, D. M. (2013). Emotional faces in context:
Age differences in recognition accuracy and scanning patterns [Press release].
Parkinson, B., Phiri, N., & Simons, G. (2012). Bursting with anxiety:
Adult social referencing in an interpersonal balloon analogue
risk task (BART). Emotion, 12(4), 817–826. doi:10.1037/
a0026434
Righart, R., & De Gelder, B. (2008). Recognition of facial
expressions is influenced by emotional scene gist. Cognitive,
Affective, & Behavioral Neuroscience, 8(3), 264–272.
Russell, J. A. (1980). A circumplex model of affect. Journal of
Personality and Social Psychology, 39, 1161–1178.
Szczurek, L., Monin, B., & Gross, J. J. (2012). The stranger effect:
The rejection of affective deviants. Psychological Science,
23(10), 1105–1111. doi:10.1177/0956797612445314
Thibault, P., Bourgeois, P., & Hess, U. (2006). The effect of groupidentification on emotion recognition: The case of cats and
basketball players. Journal of Experimental Social Psychology,
42, 676–683.
Van der Schalk, J., Hawk, S. T., Fischer, A. H., & Doosje, B. (2011).
Moving faces, looking places: Validation of the Amsterdam
Dynamic Facial Expression Set (ADFES). Emotion, 11(4),
907–920.