Simulating the Emotion Dynamics of a
Multimodal Conversational Agent
Christian Becker, Stefan Kopp, Ipke Wachsmuth
AG WBS, University of Bielefeld, Germany
Video
Introduction
Related work
Emotion system
Conclusion
The agent Max
Kinematic skeleton with 53 degrees of freedom
• Body animated by model-based animation
• Cognitive architecture based on a BDI framework
• Text-to-speech system (German only):
– TXT2PHO (IKP Uni Bonn) and MBROLA
– Phoneme transcription as basis for automatic generation of
visemes
• Facial animation based on 21 simulated muscles
– Coordinated control of face muscles based on Action Units
(Ekman/Friesen)
– Six muscle sets defined for emotional expressions:
angry
Introduction
bored
fearful
happy
Related work
sad
surprised
Emotion system
Conclusion
Two interaction scenarios
An embodied conversational agent situated in virtual reality
(1) Multimodal Assembly eXpert
•
•
•
•
SFB 360 research scenario
Located in a 3side cave-like projection
User input via speech and gestures
Action selection influenced by the
concurrent emotion system
(2) Presentation agent in the HNF museum
• Multimodal presentation module
extended by eliza-like, smalltalk abilities
• User interaction via keyboard
• User perception via video camera
• Proactive behaviors triggered by the
concurrent emotion system
Introduction
Related work
Emotion system
Conclusion
Why not OCC?
Ortony, Clore and Collins (1988): An appraisal theory of emotions
Positive aspects of the OCC model:
+ covers most situations an emotional agent might have to deal with
+ offers a structure to determine the intensity of emotion types
Untackled questions within the OCC model:
- the OCC model contains no history
- the interactions of different emotion categories are not described
- the course of emotions over time is not considered
and last but not least:
The non-cognitive emergence of emotions is neglected by this model!
(E.g.: The experience of relaxation when sitting in front of a warm oven.)
Introduction
Related work
Emotion system
Conclusion
The emotion system – an overview
Simulating the dynamics of emotional impulses
1. Dynamic component:
Conceptual linkage of
emotions and moods
2. Categorization:
Mapping onto weighted
emotion categories in
PAD space by using a
distance metric
Input
Output
- Valence of emotions: short-time system state
- Valence of moods: longer lasting system state
- (P)leasure ranges from joy (+P) to reluctance (-P)
- (A)rousal ranges from mental awareness (+A) to sleepiness (-A)
- (D)ominance describes the agent‘s feelings of control over the situation
Introduction
Related work
Emotion system
Conclusion
1. Dynamic component
• Simulating the alleviating and fortifying
effects of emotions (x) on moods (y) (as
indicated by the vertical arrows)
• Physical simulation of two spiral springs with
spring constants dx and dy
• Added the concept of boredom
• If no significant positive or negative valence
present  boredom value increases linearly
• Otherwise  boredom value reset to zero
Provides triple (x, y, z) at every timestep t with 25Hz!
Introduction
Related work
Emotion system
Conclusion
Mapping to PAD space
Emotion (x), mood (y) and boredom (z) are mapped to (p)leasure and
(a)rousal according to the following equations:
d
y
p
z
x
a
The dominance value is not deducible from the dynamic component!
Introduction
Related work
Emotion system
Conclusion
2) Categorization (PAD) component
• Nine emotion categories
represented by their PAD triples:
( p, a , d )
1. angry
(-0.8, 0.8, 1.0)
2. annoyed
(-0.5, 0.0, 1.0)
3. bored
( 0.0, -0.8, 1.0)
4. concentrated ( 0.0, 0.0, +/- 1.0)
5. depressed
( 0.0, -0.8, -1.0)
6. fearfull
(-0.8, 0.8, -1.0)
7. friendly
( 0.5, 0.0, +/- 1.0)
8. sad
(-0.5, 0.0, 1.0)
9. surprised
( 0.8, 0.8, +/- 1.0)
d
• Two thresholds are defined:
• Activation threshold .
• Saturation threshold .
Introduction
Related work
Emotion system
Conclusion
Visualisation of the emotional state
Output of the emotion system:
•
•
•
Valence of mood and degree of
boredom
Corresponding PAD triple
Activated emotion category with weight
w or state of „confusion“ otherwise
1) „Non-cognitive“ effects:
•
•
•
•
2) „Cognitive“ effects:
Weighted facial expressions according
to the actual emotion category
Modulation of breath and eye-blink
frequency by arousal value
Modulation of pitch and rate of speech
according to the emotion category
Triggering of secondary actions by
boredom value
•
Emotion categories incessantly asserted as belief of the agent
•
Agent‘s plan selection influenced by current emotion category
Introduction
Related work
Emotion system
Conclusion
Example
HNF scenario since January 2004:
• Emotion system provided with emotional impulses by perception, interpretation
and dialog manager
• It influences dialog manager (cognitive) and behavior generation (non-cognitive)
emotion
system
interpretation
dialog
manager
behavior
planning
behavior
generation
perception
Video
Introduction
Related work
Emotion system
Conclusion
Conclusion
+ Combination of moods, emotions and boredom to achieve a coherent
long-time behavior as well as convincing spontaneous reactions
+ Involuntary facets of Max‘s behavior, his facial expressions and his
deliberation process are continuously modulated by the emotion system
- Higher-level emotion categories not representable
Open question:
How can the dominance value be specified automatically?
Introduction
Related work
Emotion system
Conclusion
Thank you for your attention!
Introduction
Related work
Emotion system
Conclusion