Cognitive Modeling – Affective Models
Affective Models
Cognitive Modeling
Dominic Heger,
Felix Putze,
Tanja Schultz
21.6.2012
1/52
Outline
• Affective Computing
• Emotion Models
Cognitive Modeling – Affective Models
• Descriptive models
• Appraisal theories
• Big Five Personality Theory
• Affect Recognition
• Facial expressions, Voice, Physiology
• Examples for Affective Models in Systems
• ALMA
• Kismeth
• Affect Expression in Emotional Robots
2/52
Cognitive Modeling – Affective Models
1. Affective Computing
3/52
Affective Computing
Cognitive Modeling – Affective Models
• Affective Computing is a very active research topics
• Affective Computing (Rosalind Picard, MIT Press 1997)
• Computing that relates to, arises from, or deliberately influences
emotions (p. 3)
• How and why computers might be designed to recognize, express, and
“have” emotions
• “Affect refers to the experience of feeling or emotion”1
• → Emotion and Affect are often used equivalently
• Affective States in Human Computer Interaction
• Emotion
• Mood
• Activation, Workload, …
1APA
4/52
Dictionary of Psychology (2006)
Media Equation
Cognitive Modeling – Affective Models
• Reeves & Nass: Media equation
• Media = real life
• Humans treat computers, televisions, and other media in the same
way they treat people
• Apply social rules from human interaction to machines
 Users react to social signals sent by the system
 Users expect systems to react to social signals they send
 Machine, which do not follow those rules are perceived as
unintelligent and impolite
• Numerous experiments:
 People are polite to computers
 Treat computers with female voice
differently than computes with
male voice (gender differences)
 Etc.
5/52
Fields for affective computers
• Human-Computer Interaction is unnatural
Cognitive Modeling – Affective Models
• Adaptation to the user
• Humanoids and Embodied Conversational Agents
• Text based communication and virtual reality do not carry
emotional information
• Emotion carrying communication and telepresence
• Emotion recognition → communication channel → emotion expression
• Monitoring of humans
• Call centers, elderly care, surveillance, etc.
• Educational systems
• Emotions are known to influence learning
• E.g. Yerkes-Dodson law, flow, etc.
6/52
Emotion Theory
• Numerous theories have been proposed
• Theories have different focus and different schools
Cognitive Modeling – Affective Models
• Conceptualization by six aspects (components1) of emotions
•
•
•
•
•
•
Subjective experiences (feelings)
Psychophysiological changes (bodily sensations)
Appraisals of events (triggered by a stimulus)
Emotion regulation
Motor expressions (face, voice, gestures)
Action tendencies
• Changes in one component can lead to corresponding changes
in others (interrelated)
1Fontaine,
7/52
et al. (2007): “The world of emotions is not two-dimensional”
Why do we have Emotions?
Cognitive Modeling – Affective Models
• Functions of emotions
• Matter of communication
 Essential part of social interaction
• Regulate intensity and duration of actions
 E.g. influences motivation
• Influence on learning
• Decision making and intuitions
 “Selling is a transfer of emotions”
• Control behavior according to needs and situation
 E.g. Signals danger situations (faster responses than rational
decisioning)
• Etc.
• Emotional Intelligence is important in human life
8/52
Frontal Lobe – Emotions – Decision Making
Cognitive Modeling – Affective Models
• Descartes’ Error (Damasio, 1994)
• No dualist separation of mind and body / rationality and emotions
• In complex and uncertain situations with limited time emotions play
critical role in decision making (e.g. gut feelings)
• Somatic marker hypothesis: emotional memory sends decision signals
• Emotions guide behavior and decision making
• Rationality requires emotional input
• Case of Phineas Gage (19th century)
• One of most famous cases in neuro psychology
• Accident damaged left frontal lobe by iron rod
• No fundamental inabilities but strong change in mental
attitude, emotions, and decision making abilities
10/52
Cognitive Modeling – Affective Models
2. Emotion Models
11/52
Discrete/Categorical Models
Cognitive Modeling – Affective Models
• Anger, disgust, fear, joy, sadness, or surprise ?
12/52
Discrete/Categorical Models
• Six or more basic emotions
Cognitive Modeling – Affective Models
• Darvin, Ekman, Plutchik, …
• E.g. Anger, fear, disgust, surprise, joy, sadness, etc.
• Distinctive universal signals (e.g. facial expressions)
• Cross-culturally displayed and recognized
13/52
Cognitive Modeling – Affective Models
Different Basic Emotion Models
14/52
From Ortony and Turner, 1990: “What’s basic about basic emotions”
Dimensional Models
Cognitive Modeling – Affective Models
• Describe a small number of dimensions
• Emotion is point in this multi-dimensional space
• Russell (1980): Bipolar circumplex model (dims: Valence and Arousal)
• Mehrabian (1995): PAD model (Pleasure, Arousal, Dominance)
 Valence (quality): unpleasant to pleasant
 Positive vs. negative affective states
 Arousal (quantity): calm to excited
 Mental and/or physical activity level
 Dominance (control): weak to strong
 Control or lack of control over others or situations
• Mostly generated from data corpora (data-driven approach)
• Emotional adjectives, Facial expressions, Emotional experiences, …
• Statistical methods to find latent dimensions in data corpora
 Factor analysis, Principal Component Analysis, …
15/52
Dimensional Models
• How many dimensions do emotion models need?
Cognitive Modeling – Affective Models
• Depends on application and goals
• E.g. Difficult to discriminate anger, fear, and stress in Valence-ArousalModel
16/52
Cognitive Modeling – Affective Models
Relationship Discrete and Dimensional Models
Adapted from Russel 1980 (by Picard)
17/52
Plutchik’s theory of basic emotions
Cognitive Modeling – Affective Models
• 1960-1980s Emotion – A Psychoevolutionary Synthesis
• 3D circumplex model of emotions
•
•
•
•
Analogous to color wheel
8 Primary bipolar emotions
Different intensities of emotions
Combines discrete and
dimensional models
• Can be mixed
(secondary emotions)
18/52
Appraisal Models
• Appraisals
• Mental representations of a situation by an individual
(including interpretations and explanations)
Cognitive Modeling – Affective Models
• Emotions in Appraisal Models
• Emotions are cognitive appraisals of antecedent situations
• I.e. responses based on evaluations of situations and events
• Appraisal Theory can explain shortcomings of other theories
• Variability and degree of response in emotional reactions
• Similar situations and different reactions
• Similar reaction to different situations
19/52
OCC Emotion Model
Cognitive Modeling – Affective Models
•
•
•
•
Ortony, Clore und Collins (1988)
One of the most famous emotion models for virtual emotional characters
Several more or less simplified implementations for exist
Distinguishes 22 emotion categories by evaluating
• Consequences of events (Happy or unhappy)
• Aspects of objects (Like or dislike)
• Actions of agents (approve or disapprove)
20/52
Cognitive Modeling – Affective Models
OCC Model Structure
21/52
OCC Emotion Model
• Knowledge (goals, standards, attitudes) needed for evaluation
Cognitive Modeling – Affective Models
• Specified by designer of the character
• World model: E.g. Large Table with events, actions, objects, affected
emotion category,…
• Intensity variables for each emotion category
• After classification of the emotion category
• Depend on agent’s goals and history of events
• Example: User gives agent bunch of bananas
• Traversal of the flow diagram from the perspective of the agent
• Appraisal evaluation results in emotion
22/52
OCC Model Structure
Cognitive Modeling – Affective Models
• Consequences of giving bananas for the user → Pity
23/52
OCC Model Structure
Cognitive Modeling – Affective Models
• Consequences of giving bananas for the agent itself (agent is hungry) →
Satisfaction
24/52
OCC Model Structure
Cognitive Modeling – Affective Models
• Action of the user (agent respects selfless action) → Admiration
25/52
OCC Model Structure
Cognitive Modeling – Affective Models
• Aspect of the object (agent has passion for bananas) → Love
26/52
Component Process Model of Emotion
• Scherer (2001): “Appraisal Considered as a Process of Multilevel
Sequential Checking”
Cognitive Modeling – Affective Models
• Cognitive appraisal modeled by Stimulus Evaluation Checks (SEC)
• “Minimal set […] to account for the differentiation of the major families of
emotional states”
• Fixed sequence of checks
• Four Appraisal Objectives (types or classes of information
required for an organism to prepare an appropriate reaction)
1.
2.
3.
4.
Relevance Detection
Implication Assessment
Coping Potential Determination
Normative Significance Evaluation
27/52
Cognitive Modeling – Affective Models
CPM Stimulus Evaluation Checks
1.
Relevance Detection
 Novelty Check (e.g. predictability, familiarity)
 Intrinsic Pleasantness Check (likely to result in pleasure or pain?)
 Goal Relevance Check (e.g. relevance for own goals and needs)
2.
Implication Assessment
 Causal Attribution Check (reason for the event)
 Outcome Probability Check (how certain is the expected result?)
 Discrepancy from Expectation Check (is the event as expected?)
 Goal/Need Conduciveness Check (helpful for own goals?)
 Urgency Check (direct action required?)
28/52
Cognitive Modeling – Affective Models
CPM Stimulus Evaluation Checks
3.
Coping Potential Determination
 Control Check (How far is the event controllable?)
 Power Check (How far am I able to control the event?)
 Adjustment Check (How well deal with consequences of an event?)
4.
Normative Significance Evaluation
 Internal Standards Check (Matches my ideals and moral?)
 External Standards Check (e.g. social consequences of an action,
compatibility with standards of a salient reference group?)
29/52
Component Process Model
• Predict specific changes caused by patterns of SEC results
• Response patterns for organismic subsystems
Cognitive Modeling – Affective Models
• Example response patterns for Novelty Check:
30/52
Component Process Model
• Major “modal“ emotions (happiness, disgust, anxiety, …) can
be predicted by SECs
Cognitive Modeling – Affective Models
• Relationship between organismic responses patterns and SECs
• Relationship between SECs and modal emotion
• Specific response profiles by vocal and facial expressions for
modal emotions exist
31/52
Example: Attitude vs Mood vs Emotion
• Attitude (evaluation of a concept in general)
• “I find it uncomfortable to go to the dentist.”
• Mood (longer lasting, general feeling)
Cognitive Modeling – Affective Models
• “I am nervous because I have to go to the dentist tomorrow.”
• Emotion (short-lived, stimulus-triggered feeling)
• “I just heard someone scream in the doctor’s room. I am frightened.”
33/52
Personality Modeling
Numerous Personality theories exist
 One of many definitions of personality:
Cognitive Modeling – Affective Models

„Characteristics of the person that account for
consistent patterns of feeling, thinking, and
behaving“
[Pervin et al., 2001]

Major differences to emotions
Personality is stable across long periods of time
 Not caused by an event or appraisal of an event


Personality fundamentally coins human behavior
34/52
Big Five Theory of Personality Traits


Cognitive Modeling – Affective Models


Five personality traits describing general dimensions of
human personality
Personality can be seen as point in a 5-dim space
Very widespread and widely researched
psychological personality theories
Based on the Lexical Hypothesis:
All aspects relevant to describe personality
are encoded as words in language


Developed using factor analysis
Elaborated personality assessment by questionnaires
→ NEO-FFI
35/52
Big Five Dimensions

Neuroticism: Stability/lability of (negative) emotional experiencing
Worried, concerned, unsure, Calm, balanced,
anxious carefree, collected

Extraversion: Sociability, positive emotions, excitement-seeking,...
Cognitive Modeling – Affective Models
Self-assured, talkative, Preference of being on their
optimistic, cheerful own

Openness to Experience: Interest in new experiences and impressions
Fancyfull, imaginative, Conventional, practical,
prefer diversification straightforward

Agreeableness: Altuism, desire for harmony,...
Benevolent, caring, Antagonistic, egocentric,
helpful, cooperative misrustfull

Conscientiousness: Achievment striving, self-discipline,...
Dutiful, reliable, Careless, incurious,
precise, squeamish less systematic
36/52
Cognitive Modeling – Affective Models
3. Emotion Recognition
37/52
Empirical Affect Modeling
• Empirical Modeling of Affect, i.e. Emotion Recognition
Cognitive Modeling – Affective Models
• Speech, visual signals, biophysiological signals
• Emotional Speech Classification
• Prosody features
 Pitch variables (F0 level, range, contour, jitter, etc.)
 Voice quality (articulation manner, voice timbre, etc.)
 Speaking rate, …
• No set of voice cues to reliably discriminate among emotions (Juslin &
Scherer, 2005)
• Syntax, Semantic, dialog strategy, …
38/52
Visual Signals
Cognitive Modeling – Affective Models
• Facial Action Coding System (Ekman & Friesen)
•
•
•
•
•
Comprehensive description of facial expressions
Action Units (AUs) for all visually distinguishable facial movements
Intensity of AUs (5 point scale)
Multiple AUs can be combined
Mapping between
basic emotions and
combinations of AUs
• MPEG-4 Facial Animation
Parameters
• Facial Expressions
• Visemes
• Other visual signals: Gestures, Pose, ...
39/52
Physiological Biosignals
• Several physiological correlates to emotions
Cognitive Modeling – Affective Models
• Blood pressure, Heart rate, Respiration rate
 Arousal: Increase
 Negative stimuli: Ambiguous results, most studies
report decrease (at least for picture viewing)
• Skin conductance
 Arousal
 Emotional stimuli (positive and negative): increase
• Brain activity
 No specific emotion areal
 Reward system (dopamine pathways): ventral tegmentum, medial
forebrain, nucleus accumbens
 Several correlation to activity in amygdala, hippocampus (limbic
system), asymmetry at prefrontal cortex, etc.
40/52
Affective Data Collection
Cognitive Modeling – Affective Models
• Picard et al., 2001: “Toward machine emotional intelligence”
• Spontaneous vs. posed
 Asked to elicit a certain emotion?
• Lab setting vs. real-world
 Usual environment of the subject?
• Expression vs. feeling
 External or internal, sender or receiver?
• Open recording vs. hidden recording
 Is subject aware of data collection?
• Emotion-purpose vs. other-purpose
 Does subject know that she is part of an emotion experiment?
• Challenges
• Segmentation and annotation (ground truth) is difficult
• No common test data to compare recognition rates
42/52
Emotion and Expression
Cognitive Modeling – Affective Models
• Difference between true emotional state and communicated
state
• Smiling associated with happiness, but:
• happiness is neither necessary not sufficient for smiling
• ∃ evidence that smiles appear limited to social circumstances
• ∃ evidence that smiles occur in humorous rather than pleasant
circumstances
• ∃ evidence that some expressions are uncorrelated with
emotion
• r = 0.78 between cognitive appraisal of unexpectedness and selfreported surprise
• r = 0.46 between self-reported surprise and facial expression
43/52
Cognitive Modeling – Affective Models
4. Affective Models in Systems
44/52
ALMA – A Layered Model of Affect
Cognitive Modeling – Affective Models
• Example for virtual characters: VirtualHuman (DFKI, 2005)
• Goal: Human-like conversational characters, to generate a
Learning group experience for the user
• Layered model of affect
• Short-term affect: Emotions (OCC, PAD)
• Medium-term affect: Mood (PAD)
• Long-term affect: Personality (Big Five)
• Map OCC emotions to PAD space
• Big Five personality traits as
default PAD mood
• Example for a mood change
• Slightly anxious mood
• Several fear emotion events
• Mood changes to fully anxious
45/52
Student Sven and teacher Valerie
ALMA – A Layered Model of Affect
• XML definitions for each character for
• Five personality traits,
• Appraisal rules
Cognitive Modeling – Affective Models
• Modeled affect used to
•
•
•
•
Select wording and phrasing
Select dialog strategies
Trigger idle gestures
Change the characteristics of
conversational gestures
• Control of facial expressions
46/52
Affect Expressive Robots
Cognitive Modeling – Affective Models
• Kismet (Breazeal, MIT, 1990s)
• Anthropomorphic robot head for face-to-face interaction with humans
• Sensors: Cameras, microphones (prosody)
• Affect expressive abilities to communicate likes/dislikes: Tone of voice,
Facial expressions, Postures
• Cognitive architecture to model drives and emotions
47/52
Cognitive Modeling – Affective Models
• Video!!!
48/52
Cognitive Modeling – Affective Models
Kismet’s Cognitive Architecture
Breazeal (2003), “Emotion and sociable humanoid robots”
49/52
Cognitive Modeling – Affective Models
Kismet’s Cognitive Architecture
• Visual and auditory stimuli sensed by the robot
• Filtered by feature extractors (e.g. color, motion, pitch, etc.)
• High-level perceptual system (binds features by releaser
processes representing current state of the robot)
• Affective appraisal phase: Active releasers are tagged with
affective information
• 4 types of appraisal
 Intensity
 Relevance
 Intrinsic Pleasantness
 Goal Directedness
→ Arousal (energy), valence (favorable), and stance (approachable)
50/52
Cognitive Modeling – Affective Models
Kismet’s Cognitive Architecture
• Appraisals are filtered through the emotion elicitors for each
emotion process
• Emotion arbitration phase: emotion processes compete for
activation in a winner-take-all scheme
• Winner evokes a corresponding facial expression, body
posture, and vocal quality (expressive motor system)
• May also evoke a corresponding behavioral response by the
corresponding behavior in the behavior system
51/52
Affect Expressive Robots
• Albert (UCSD, 2009)
• Video
http://www.youtube.com/watch?v=pkpWCu1k0ZI
• Hiroshi Ishiguro (Osaka, ATR)
Cognitive Modeling – Affective Models
• Telepresence humanoid robots
52/52