Affective Game Engines:
Motivation &
Requirements
Eva Hudlicka
Psychometrix Associates
Blacksburg, VA
[email protected]
psychometrixassociates.com
DigiPen Institute of Technology
February 20, 2009
1
Outline
• Why we need Affective Game Engines?
• How can Affective Computing help?
• What are some of the functionalities
needed?
• What might an Affective Engine look like
• Conclusions
2
Where We Are Now
• Tremendous advances in gaming technologies
• Focused primarily on:
– Physical realism of game characters & game environments
– Complexity & performance of simulations & networking
• Today’s games still limited in:
– Affective realism game characters
– Social complexity & realism of their interactions
– Ability to adapt to player’s state
3
In Terms of the Full Potential of Gaming..
We Are About Here…
4
To Achieve the “next big leap”
• ..in engagement & effectiveness
• Need to enhance social & affective complexity &
realism of:
–
–
–
–
Game characters
Their interaction with each other
… and the players
Game narrative as a whole
• Especially true for serious games:
– Education
– Training of affective & social skills
– Rehabilitation & therapy
5
So How Do We Get Here?
We Are Here…
6
How Do We Get There?
• Add a lot more affect
– Affect-focused game design
– Affective gaming
• Add it easily - develop the right tools
– Affective game engines
• Draw on research & technology base in affective
computing
– Recognition
– Modeling
– Expression
7
Why Emotions?
8
Emotion is a Key Factor…
in both “Play” & “Work”
• Mediates motivation
• Influences memory & learning
• Key role in decision-making & problem-solving
• Central factor in engagement
• Key factor in serious games
9
Future Games Need To…
• Recognize & adapt to players’ emotions
• Understand players’ affective profiles
WE NEED DEVELOPMENT
TOOLS TO MAKE ALL THIS
Increase visual affective realism of game
EASY
characters & player avatars
… and maybe even fun?)
• Increase affective complexity & autonomy of game
characters
•
• Increase affective complexity of the entire game
experience
10
Need for Affective Game
Engines
• Game development greatly enhanced by game
engines
• Existing engines support construction of:
– Visually-realistic characters
– Visually realistic physical environments
– 3-D objects
• But so far, no engines support affective gaming
11
Affective Game Engines:
Requirements
• What functionalities should affective game
engines support?
• Emotion sensing & recognition
• Affective modeling of players
• Game characters w/ more affective & social
realism
– Affective models guide character behavior
– Affective expression in game characters
12
Relevant Disciplines
• AI
–
–
–
–
AI is more than RBS, A*, FSA
Symbolic agent architectures
Probabilistic reasoning (Bayesian belief nets)
Learning
• Affective Computing
– Emotion recognition
– Affective modeling
– Emotion expression
13
Outline
• Why we need Affective Game Engines?
• How can Affective Computing help?
• What are some of the functionalities
needed?
• What might an Affective Engine look like
• Conclusions
14
Overview of Affective Computing
MAX (Becker, Prendinger et al.)
Breazeal
De Rosis
Hudlicka
15
Intelligent Affective Game
Characters
Affective Virtual Character - Max
Becker et al., 2005
16
Outline
• Why we need Affective Game Engines?
• How can Affective Computing help?
• What are some of the functionalities
needed?
• What might an Affective Engine look like
• Conclusions
17
Affective Game Engines
Should Help Design Games
That….
• Recognize player emotion
• Adapt to player’s emotions
• Have more realistic game characters
–
–
–
–
Affective & Social realism
…React with appropriate emotion
…Express the emotion in a believable manner
…Whose behavior is influenced by the emotion
18
Emotion Recognition &
Expression in Games
• Much recent progress in basic emotion
recognition (fear, anger, joy, sadness)
• Multi-modal approaches approach human
recognition rates
– Audio-visual; Audio-visual-physiological
• Recognition of spontaneous emotions
• Recognition of complex emotions (e.g.,
embarrassment)
19
Challenges in Game Contexts
• Different emotions in different game categories
– FPS vs. social games (Sims) vs. serious games for
training / therapy
• Different platforms
– VR vs. Wii vs. Nintendo vs. iPhone
• Non-intrusive sensors
• Noisy data
– Player movement
– Lighting conditions
20
Cohn 2006
21
Emotion Signatures:
Modalities & Time
• Emotions occur across multiple modalities
– Expressive / Behavioral
– Cognitive
– Physiological
– Experiential
• Different emotions have distinct multimodal
signatures
22
Emotion Effects on Behavior
Facial
expression
Gestures
Emotion
Posture
Behavior
Blah blah blah
23
Anger
•
Trigger:
– Progress toward a goal hindered
… esp. by other agent
•
Cognitive:
–
–
–
–
•
Focus attention (very strong effect)
Assign blame to the perceived causal agent (typically another agent)
Overestimate chances of own success
Try alternate strategies
Physiological – mobilize and sustain high energy levels:
–
–
–
–
–
•
www.firstpeople.us
Higher diastolic blood pressure (than fear)
Greater peripheral resistance (than fear)
Larger increase in heart rate (than disgust)
Larger heart rate acceleration (than happiness)
Larger increase in finger temperature (than fear)
Behavioral:
– Eagerness to act
– Fight & aggression
– Social: prevent (or facilitate) aggression
24
Emotion Signatures:
Modalities & Time
• Temporal & spatial congruence makes
emotion recognition easier
– Multimodal approaches more successful
• Visual (face) + audio (speech)
• Physiological (arousal) + visual…
25
Emotion Signatures:
Modalities & Time
• Requirements for temporal &
spatial congruence make emotion
expression more difficult
– Facial expression, tone, words, gestures…
– All must agree to be believable
– …across temporal intervals
26
Semantic Primitives for
Recognition & Expression
• Identify primitives for each channel
• … to facilitate recognition & expression
• Associate each emotion with a specific
configuration of ‘primitives’
• Some channels have established primitives
–
–
–
–
Facial expressions (established & mature)
Speech (generic signal properties)
Posture (emerging)
Movement (being adopted from choreography - Laban)
27
Facial Action Coding Units
• Basic emotions reflected in the face via characteristic
configurations of facial muscles
–
–
–
–
Shape of lips
Shape of eyebrows
Narrowing of eyes
Raising cheeks
• Facial action coding system (FACS) provides codes
for these configurations (Ekman & Friesen, 1978)
– Action Units (AU’s) correspond to individual muscle positions
& movements
– All possible expressions can be analyzed in terms of AU’s
28
FACS-Coded
Facial Expression
Movellan et al.
http://mplab.ucsd.edu/
29
Modality, Channel & Sensor
Selection Criteria
• Which emotions need to be recognized / expressed?
• What are their signatures… along each channel
• What semantic primitives are available
– Facial action units
– Speech? Posture?
– Physiological data (arousal, EEG)
• What sensors are available
– Non-intrusive
– Generate adequately ‘clean’ raw data
– Appropriate for context & platform
30
Sensors for Gaming
Emotiv Systems
EEG & facial movements
Emsense:
EEG, heart rate,
respiration, head
motion, temperature
Neurosky
31
Non-Intrusive GSR Sensing
Picard et al., 200832
Computational Affective
Modeling
• Agent architectures for game characters
–
–
–
–
Control NPC behavior
…More affective & social realism
…More believable characters & their interactions
…More engaging & effective games
• Affective User Models
– Models of player affective profiles
– Support recognition of player’s affective state
– Support adaptation to player’s state
33
Affective Agent Architectures
• Enable game characters to:
– React to evolving situations in game
– React to other characters in game
– React to player’s state and behavior
• … by dynamically generating appropriate
emotions
• … which influence decision-making & behavior
• … and by supporting their realistic display
34
Affective Agent Architectures
Control Game Character Behavior
Effects of Emotions
(on cognition & behavior)
Generation of Emotions
(via cognitive appraisal)
Stimuli
Agent
Architecture
Emotions
35
MAMID Architecture: Semantics & Data Flow
Cues: State of the world
Cues
“Growling dog approaching”
Attention
Situations: Perceived state
“Aggressive dog”
Situation
Assessment
Expectations: Expected state
Expectation
Generator
“Dog will bite me”
Affect
Appraiser
Affective state & emotions:
Valence:
Negative
Happiness: Low
Fear:
High
Goals: Desired state
“Avoid being bitten”
Goal
Manager
Action
Selection
Actions
Actions: to accomplish goals
“Climb a tree”
36
How Difficult Is This?
• Depends on game complexity… game type..
• Which emotions are necessary?
• What features of the game context are available to
trigger an emotion?
• Simple games may not need much
• Sophisticated ‘social’ games & serious games need:
–
–
–
–
–
More emotions
Real-time generation of appropriate emotion
Realistic influence of emotion on perception + cognition
Real-time expression of appropriate emotion
More realistic affective dynamics
37
How Do We Do It?
• Black-box models
– Stimulus ---> Emotion
– Simple but ‘clunky’ - does not generalize
• Process models
– Explicit models of some underlying processes
– Emotion generation
– Emotion effects on
•
•
•
•
Perception
Decision-making
Behavior
Expression
38
Black Box Models
Character
ridiculed
?????
Directly map
stimuli-->
onto
emotions:
Character gains points ---> Happy
Ooops!
Character
No rule
for that oneloses points ---> Sad
outsmarted ---> Angry
NowCharacter
what?
39
Process Models
Model underlying mechanisms of emotion
generation
• Cognitive appraisal
• Stimuli ---> Appraisal Dimensions ---> Emotion
• More general - more extensible
Goals
Stimuli
Situations
Cognitive
Appraisal
Emotions
Expectations
40
Appraisal Variables
STIMULI
FEAR
high
Novelty
Valence
low
Goal
relevance
high
Agency
other
Outcome
probability
high
Goal
congruence
low
Urgency
v. high
Coping
potential
Norms
low
41
Affective User Models
• Represent player’s affective profile
• What makes them:
– Happy? Frustrated? Bored? Engaged? Angry?
• How is this manifested in THIS player?
• How do we find this out?
– Learning algorithms
• Baseline data
• Training period
• How do we represent the info
– Augmented state transition diagrams
–…
42
Outline
• Why we need Affective Game Engines?
• How can Affective Computing help?
• What are some of the functionalities
needed?
• What might an Affective Engine look like
• Conclusions
43
So What Does All This
Suggest for the Affective
Game Engine Requirements?
44
Emotion Sensing
& Recognition
Affective User
Modeling
Central Shared
“Emotion”
Knowledge-Base
Modelling of
Game Character
Emotions
Emotion Expression
In Game
Characters
45
Shared
Emotion Knowledge-Base
• Generic knowledge about emotions
– Triggers, influences, manifestations
• Affective model of player
– Idiosyncratic triggers, influences, manifestations
• Affective models of game characters
– As above
• Start with basic emotions
• Include states relevant for gaming / training
– Surprise, boredom, engagement, ‘flow’
• Progress to social & more complex emotions
46
Possible KB Structure
-
Generic Schema
-
Organized in an
inheritance hierarchy
Different modules
access different slots
Not all slots needed
by each modules
Instantiated
Schema
47
Outline
• Why we need Affective Game Engines?
• How can Affective Computing help?
• What are some of the functionalities
needed?
• What might an Affective Engine look like
• Conclusions
48
Summary
• Games need more focus on emotion
• Affective game engines would help
develop affective games
• Recognition & generation of arbitrary
emotions still difficult..
– Esp. in noisy, realistic contexts
• But… we’re getting there
49
Conclusions
• Is it too early for affective game engines?
• Different features grouped in different tools
– … analogous to today’s engines
• Concrete requirements support systematic design
of affective games
– … and the tools for their development
• Facilitate development of better games
• Provide platforms for research in AI, Aff Comp,
Psychology,…
50
Thank you
Eva Hudlicka
Psychometrix Associates
Blacksburg, VA
[email protected]
psychometrixassociates.com
51