An Adaptable Fuzzy Emotion Model for Emotion Recognition
Natascha Esau
Lisa Kleinjohann
C-LAB
Fuerstenallee 11
D-33102 Paderborn, Germany
e-mail: {nesau, lisa, bernd}@c-lab.de
Abstract
Existing emotion recognition applications usually distinguish between a small number of emotions. However this set of so called basic emotions varies from
one application to another depending on their according needs. In order to support such differing application needs an adaptable emotion model based on
the fuzzy hypercube is presented. In addition to existing models it supports also the recognition of derived
emotions which are combinations of basic emotions.
We show the application of this model by a prosody
based fuzzy emotion recognition system.
Keywords: Fuzzy emotion model, fuzzy hypercube,
fuzzy emotion recognition, basic emotion.
1
Introduction
Emotions are an evident part of interactions between
human beeings. But also for interactions of humans
with computer systems emotions play a major role,
since humans can never entirely switch off their emotions. During the last years interest in emotions increased considerably in various domains of computer
based systems. Examples are robots or virtual agents
that show emotions or human-computer interfaces that
consider human emotions in their interaction capabilities. In Japan an entire stream called KANSEI information processing [10] deals with subjective human
feelings when interacting with IT systems. These few
examples already reveal two major tasks of emotion
processing in IT systems, the recognition of human
emotions and the (re)production of artificial emotions.
Whereas robots or virtual agents often show emotions
themselves, for many other IT systems the recognition
of human emotions and appropriate reactions suffice
Bernd Kleinjohann
to improve the system’s performance or acceptance.
Imagine for instance a user who is angry, because the
IT system does not behave in the expected way or she
tried several times to accomplish a task without success. In such a situation it would be very helpful and
increase the system acceptance, if an IT system could
recognize this emotion and react accordingly. Another
example is speech recognition. According to investigations at the MIT a doubling of the word error rate to
about 32% was observed when people talk in an angry
way [4]. In such a case an appropriate system reaction
would be to redirect the user to a human operator or
to give hints how she could decrease the error rate.
Depending on the intended application domain different emotions are relevant for emotion recognition. According to the observation described above,
the speech recognition system Mercury distinguishes
only two classes of emotions: frustration and neutral. For other applications like personal robots or entertainment robots certainly the recognition of some
more emotions like happiness or sadness would be
interesting to react with according robot behavior.
Since also psychologists have not yet agreed upon a
set of basic emotions (see Section 2) it is not likely
to identify a set of emotions, that is appropriate for
all computer based emotion recognition (CBER) systems. Therefore, in this paper we propose an emotion
model for emotion recognition that is easily adaptable
to the selected set of basic emotions for the CBER
problem (see Section 3). However humans do not only
feel some basic emotions in their pure form but also
some more complex or derived emotions [16]. An
example is for instance curiosity which is according
to experiments by Plutchik a combination of acceptance and surprise. Accounting for this observation
our emotion model does not only support basic emo-
tions but also supports the representation of such derived emotions or blends.
Furthermore different intensities or degrees of emotions can be observed [16]. For modelling of different
degrees of membership to different classes in a classification system in many application domains, among
them also emotion recognition, fuzzy logic is a very
useful approach. Therefore we developed our adaptable emotion model using the fuzzy hypercube as basis (see Section 3). We show the applicability of our
approach by a system for emotion recognition from
prosody of natural speech in Section 4. Afterwards
we compare our approach with related work and give
a short conclusion.
Table 1: Basic emotions distinguished by psychologists
Psychologist Basic Emotions
Plutchik
Ekman,
Friesen,
Ellsworth
Frijda
Izard
James
Mowrer
Oatley and
JohnsonLaird
2 Emotion Models in Psychology
Psychologists have tried to explain the nature of human emotions for decades or even centuries. Nevertheless no unique established emotion model exists.
However emotion is now usually seen as a dynamic
process that involves several modalities like motoric
expression, physiological arousal and subjective feeling [13, 9]. For computer based emotion recognition (CBER), however models that help with the classification of emotions are more important. Among
these two major types of emotion models can be distinguished (also mixtures of these types are found):
models that rely on basic emotions and emotion models that classify emotions according to different dimensions like valence, potency, arousal, intensity etc.
The first one has a major advantage for CBER, since it
considerably decreases recognition complexity due to
a small number of basic emotions to which the CBER
can be restricted. A well known earlier model of basic
emotions is the work of Plutchik [16]. He uses basic
emotions as a kind of building block for derived emotions, so called secondary emotions. Plutchik even
distinguishes ternary emotions that are combinations
of secondary derived emotions. His model like many
others also describes the concept of emotion intensity, that represents the strength by which an emotion is felt. Although emotion models exist for several decades, even now there is no general agreement
among psychologists how many basic emotions exist
and what they are. This is shown in table 1 which is
an excerpt from [15].
Due to the variety of basic emotions described in liter-
Acceptance, anger, anticipaton, disgust, joy, fear, sadness, surprise
Anger, disgust, fear, joy, sadness,
surprise
Desire, happiness, interest, surprise,
wonder, sorrow
Anger, contempt, disgust, distress,
fear, guilt, interest, joy, shame, surprise
Fear, grief, love, rage
Pain, pleasure
Anger, disgust, anxiety, happiness,
sadness
ature it seems reasonable to develop an emotion model
for emotion recognition that is easily adaptable to the
selected set of basic emotions for the CBER problem.
3
Fuzzy Emotion Model
As already stated, according to psychologists like
Plutchik humans do not only feel a single basic emotion but have more complex emotional states, where
more than one basic emotion is involved with varying
strength or intensity. Therefore we propose a fuzzy
classification of emotional states using fuzzy hypercubes [12]. Furthermore we assume that the intensity of an emotion can be mapped to the interval [0, 1].
First we define a fuzzy set corresponding to an emotional state and then show how it is represented in a
fuzzy emotion hypercube.
Fuzzy set for emotional state. Let BE be a finite
base set of n basic emotions e1 , e2 , . . . en and
{µF Ej : BE → [0, 1], j = 1, 2, . . .} an infinite set of
fuzzy membership functions. Then each
F Ej := {(ei , µF Ej (ei ) | ei ∈ BE}, j = 1, 2, . . . defines a fuzzy set corresponding to one emotional state
Ej .
Fuzzy emotion hypercube. If BE, µF Ej and F Ej
are defined as described above, we shall use the mem-
bership vector
(µF Ej (e1 ), µF Ej (e2 ), . . . , µF Ej (en )) =: (µF Ej (ei ))
to denote a point in an n-dimensional hypercube.
Each axis of the hypercube corresponds to one basic
emotion ei . Thus a membership vector (µF Ej (ei ))
corresponds to one emotional state Ej and can be interpreted psychologically as vector of emotion intensities (Iei ) := (Ie1 , Ie2 , . . . , Ien ).
The number of distinguished emotions depends on
the psychological theory or in the case of computer
based emotion recognition on the intended application. If for instance the three basic emotions happiness h, anger a and surprise s shall be distinguished,
a three dimensional unit cube as depicted in Figure 1
is needed for modelling emotional states.
E2
(0,0,1)
Surprise
E1
Anger
(0,1,0)
(0,0,0)
Happiness (1,0,0)
Figure 1: Fuzzy unit cube for three emotions happiness, suprise and anger
Figure 2 shows how the unit cube could be further divided in order to represent basic emotions and their
mixtures. In the subcubes denoted by a single emotion the membership function of this emotion takes
values in the interval [0.5, 1.0] whereas the membership values for the other emotions respectively their
intensities are below 0.5. Therfore it is reasonable
to associate the subcube with this basic emotion. In
the subcubes denoted with a sum of emotions (e.g.
Surprise + Happiness) memberships of these emotions are in the interval [0.5, 1.0] whereas the membership of the third emotion is below 0.5. Hence
a derived emotion from these two basic emotions
(e.g. surprise and happiness) is assumed. The subcube where the membership values of all basic emotions are between 0.5 and 1.0 is denoted by the sum
Surprise + Anger + Happiness.
If a general n-dimensional emotion hypercube is regarded certainly not all combinations of up to n emotions make sense. However, whether a combination
is reasonable or not is certainly a psychological question. If combinations that do not make sense are recognized by a CBER this could for instance indicate an
error.
4 Application
The corners in the unit cube describe dual memberships (0 or 1) for all emotions, vertices desribe
dual memberships for two emotions and the third one
varies from 0 to 1. For example, the point E1 =
(1.0, 0.2, 0.3) corresponding to the fuzzy set
F E1 = {(h, 1.0), (a, 0.2), (s, 0.3)} represents a
happy emotional state. The point E2 = (0.2, 1.0, 0.9)
corresponding to
F E2 = {(h, 0.2), (a, 1.0), (s, 0.9)} certainly represents an emotional state for a derived emotion from
anger and suprise. The point (0, 0, 0) represents the
entirely neutral state where no emotion is present.
Surprise
Surprise
+
Happin.
Surprise Surprise
+ Anger
+
Anger + Happin.
Anger
Neutral
Happin.
Anger
+
Happin.
Figure 2: Subdivisions of unit cube representing basic
and derived emotions
This section deals with the application of our adaptable emotion model for the fuzzy rule based emotion
recognition system PROSBER [2].
4.1 Overview of PROSBER
PROSBER recognizes emotions from the prosody of
natural speech. It takes single sentences as input and
classifies them into the emotion categories happiness,
sadness, anger and fear. Furthermore a neutral emotional state is distinguished. PROSBER automatically
generates the fuzzy models for emotion recognition.
Accordingly two working modes are distinguished,
training and recognition, as depicted in Figure 3.
During the training the training samples with wellknown emotion values are used to create the fuzzy
models for the individual emotions. For that purpose
sequences of acoustic parameters like fundmental frequency or jitter are extracted. PROSBER extracts
about twenty parameters that have shown their relevance for emotion recognition in psychological stud-
Emotions of
training samples
Frames
Speech
signal
Preprocessing
Parameter
sequences
Parameter
extraction
Feature
vectors
Feature
calculation
Fuzzy model generation
Membership
functions
generation
Feature
selection
Fuzzy
rule
construction
Training
Membership
functions
Frames
Speech
signal
Preprocessing
Parameter
sequences
Parameter
extraction
Fuzzy
rules
Feature
vectors
Feature
calculation
Emotion
Fuzzy
classification
Recognition
Figure 3: Architecture of PROSBER
ies or in other speech based emotion recognition systems. The sequences of these acoustic parameters are
summarized by statistical analysis steps performed by
the feature calculation. The fuzzy model generation is
based on a fuzzy grid approach [11]. It performs the
following three steps on the training database. First
the membership functions for every feature are generated. Afterwards for each emotion up to six most significant features are selected and then the fuzzy rule
system for each emotion is generated. These fuzzy
models are used in the emotion recognition process to
classify unknown audio data. A detailed description
of PROSBER can be found in [2].
4.2 Fuzzy Emotion Recognition in PROSBER
In order to describe the application of our emotion
model we shall now have a closer look at the fuzzy
classification. In principle it is structured as depicted
in Figure 4.
Feature
vectors
Emotion 1
intensity
Emotion 1
…..
…..
ture fj and emotion intensity Iei by five triangular
membership functions verylow, low, medium, high
and veryhigh as schematically depicted in Figure 5.
However, the actual start and end coordinates as well
as the maximum coordinates are generated automatically during the training phase. This representation
is simple enough to support real time emotion recognition, yet allows to distinguish degrees to which a
feature or emotion is present in the current input sentence. Furthermore, it is in line with psychologists’
approaches who often use two up to ten levels for
characterizing psychological phenomena like emotion
intensities.
1.0
very
very
low low med. high high
0.8
0.5
0.2
0
0.25 0.5 0.75
1.0
Figure 5: Membership functions for features and emotions
The rule set for the emotion ei is generated by a fuzzy
grid approach [11]. Since this approach uses only
the AN D connector it generates 5K+1 rules of the
following form:
IF f1 IS verylow AND ... AND fK IS verylow THEN
Iei IS veryhigh
IF f1 IS verylow AND ... AND fK IS low THEN Iei
IS veryhigh
IF f1 IS verylow AND ... AND fK IS medium THEN
Iei IS medium
...
Emotion
Emotion n
Rule Base
Emotion n
Fuzzification
Fuzzy Inference
Max
Defuzzification
Emotion n
intensity
Figure 4: Principle structure of fuzzy classification
For each basic emotion ei , i = 1, ..., n, a separate rule set is generated by an adapted fuzzy grid
method. Each rule takes the fuzzified features fj , j =
1, ..., K, K ≤ 6, as input and produces a fuzzy emotion value Iei as output. We represent each fea-
The number of rules could be reduced, if the OR connector or rule pruning could be used. However, both
features are not yet supported by the fuzzy library we
use. For defuzzification of emotion values we use
the center of gravity (COG) method. By projecting
the COG to the x-axis we calculate the corresponding
emotion intensity. Hence a four dimensional vector
(Ih , Is , Ia , If ) = (µE (h), µE (s), µE (a), µE (f ))
containing the intensities of the four emotions happiness h, sadness s, anger a, and fear f is generated.
This vector represents the membership values for each
emotion and hence determines a point in the four dimensional emotion hypercube.
Presently PROSBER recognizes a single basic emotion. In order to select this emotion we determine the
emotion erec ∈ {h, s, a, f } with maximum intensity
Ierec = max{Ih , Is , Ia , If }. If the maximum cannot
be determined unambiguously, since two or more intensity values are maximal, that emotion is selected
which was recognized for the previous sentence. The
neutral emotional state is identified by a hypercube
part near to the origin as depicted in Figure 2. We
plan to extend PROSBER for recognition of combined
emotions as described above.
5
Related Work
Up to now a variety of emotion models have been
described in literature. They are mainly dedicated
to computer based emotion (re)production or simulation in different application domains. A broad
application domain are virtual agents, that show
(pseudo)emotional behavior in their communication
with humans [3, 8, 5, 7]. They rely on a dimensional model of emotions based on the event-appraisal
emotion model of Ortony et al. [14]. They usually distinguish the dimensions pleasure, arousal and
dominance and try to maintain their dynamics over
time. The PETEEI system [7] for simulation of a
pet’s evolving emotional intelligence similarly to our
system uses fuzzy sets for emotion representation.
However this system is different from our approach
since it associates certain types of events with positive or negative feelings in order to react with according emotions whereas our approach is dedicated to
emotion recognition from certain features (of speech,
facial expression etc.). Emotion recognition as investigated in our approach is to some extent covered
by Kismet [6]. However Kismet recognizes intentions rather than emotions. The emotional system developed for AIBO and SDR [1] like our model uses
basic emotions. Since it is intended for production
of emotional behavior it uses the dimensions pleasure and arousal as mentioned above. But the dominance dimension is substituted by a confidence dimension representing the certainty of recognized external stimuli. The models described above only deal
with single emotions and do not allow to represent
combinations or blends of emotions like our approach.
The Cathexis model [17] supports this feature and
is also adaptable to different sets of basic emotions
or emotion families as they are called there by supporting the coexistence of several active so called
emotion proto-specialists representing different emotion families. However Cathexis is also dedicated to
the (re)production of emotional behavior in synthetic
agents whereas our adaptable emotion model is intended for emotion recognition.
6 Conclusion and Outlook
This paper presented an adaptable emotion model for
emotion recognition. It uses the concept of an ndimensional fuzzy hypercube to represent emotional
states made up of n basic emotions. In contrast to
other approaches this allows not only the representation and recognition of a fixed set of basic emotions but also supports the handling of derived emotions. We showed the application of this model using the fuzzy prosody based emotion recognition system PROSBER. As a first step we proposed a division of the unit hypercube in equally sized subcubes
to distinguish basic emotions and their combinations
or blends. An interesting point for further investigation is whether this subdivision corresponds to human
recognition. This could for instance be done using a
learning approach that automatically finds such subdivisions and compares them with human interpretations of corresponding emotional states.
References
[1] R. C. Arkin, M. Fujita, T. Takagi, and R. Hasegawa.
An ethological and emotional basis for human-robot
interaction. In Robotics and Autonomous Systems,
vol. 42, no. 3, pages 191–201. Elsevier Science, 2003.
[2] A. Austermann, N. Esau, L. Kleinjohann, and
B. Kleinjohann. Prosody based emotion recognition
for mexi. In Proceedings of IEEE/RSJ Int. Conference on Intelligent Robots and Systems (IROS 2005),
Edmonton, Alberta, Canada, August 2005.
[3] J. Bates. The role of emotion in believable agents.
Communication of the ACM, 37(7), pages 122–125,
1992.
[4] A. Boozer. Characterization of Emotional Speech
in Human-Computer-Dialogues. M.Sc. Thesis. MIT
Press, 2003.
[5] C. Breazeal. Affective interaction between humans
and robots. In Proc. of ECAL 01, pages 582–591,
Prague, 2001.
[6] C. Breazeal and L. Aryananda. Recognition of affective communicative intent in robot-directed speech. In
Autonomous Robots 12, pages 83–104. Kluwer Academic Publishers, 2002.
[7] M. S. El-Nasr, J. Yen, and T. Ioerger. Flame - a
fuzzy logic adaptive model of emotions. In Automous
Agents and Multi-agent Systems 3, pages 219–257,
2000.
[8] C. Elliot. The Affective Reasoner: A Process model of
emotions in a multi-agent system. Ph.D. Thesis. Institute for the Learning Sciences, Evanston, IL: Northwestern University, 1992.
[9] N. H. Frijda. Neural Networks and Fuzzy Systems;
A Dynamical Systems Approach to Machine Intelligence. Cambridge University Press, 1986.
[10] S. Hashimoto. Kansei as the third target of information processing and related topics. In Proceedings
of Intl. Workshop on Kansei Technology of Emotion,
pages 101–104, 1997.
[11] H. Ishibuchi and T. Nakashima. A study on generating fuzzy classification rules using histogramms. In
Knowledge based Intelligent electronic Systems, Bd.
1. Prentice Hall, 1998.
[12] B. Kosko. Neural Networks and Fuzzy Systems; A Dynamical Systems Approach to Machine Intelligence.
Prentice Hall, Englewood Cliffs, NJ, 1992.
[13] R. S. Lazarus. Emotion and Adaptation. Oxford University Press, 1991.
[14] A. Ortony, G. Clore, and A. Collins. The Cognitive
Structure of Emotions. Cambridge University Press,
1988.
[15] A. Ortony and W. Turner. What’s basic about basic
emotions? Psychological Review, pages 315–331,
1990.
[16] R. Plutchik. The Emotions. University Press of America, Inc., revised edition, 1990.
[17] J. Velasquez. Modeling emotions and other motivations in synthetic agents. In Proceedings of the AAAI
Conference, pages 10–15. Providence, RI, 1997.