Advanced Decision Architectures Collaborative Technology Alliance
A Computational Model of Naturalistic
Decision Making and the Science of
Simulation
Walter Warwick
Amy Santamaria
& many, many others
Overview
Advanced Decision Architectures Collaborative Technology Alliance
• The M&S big picture
• The work
– Birth
• You can’t model this; that’s not a model
– Life
• Where’s the data
– Quiet reflection
• The science of simulation
The Big Picture
Advanced Decision Architectures Collaborative Technology Alliance
• An effort to improve human behavior
representations for M&S but incorporating
a better model of decision making
– Better than: a “tactical” or probabilistic
decision
– Allows new kinds of behavior to play inside of
task network models
• Not an exercise in theory validation
– Though we’d like our work to illuminate theory
The Birth of the RPD Widget
Advanced Decision Architectures Collaborative Technology Alliance
• From a descriptive model to a theoretical model:
– A clash of traditions
– A lot of thrashing
– The emergence of a cottage industry and an M&S land grab
• A new decision type (“RPD”) in the Micro Saint
Sharp family of task network modeling tools
• Widget intended to capture:
– Experience-based decision making via a multiple trace model of
memory and simple reinforcement routine
– Recognitional decision making via similarity-based recall
mechanism that draws on *every* past experience
– Expectancy generation and feedback—several different versions
implemented, rarely used and no clear indication that we can do
anything interesting with it
Using the Widget
Advanced Decision Architectures Collaborative Technology Alliance
• To specify an RPD decision type, the
modeler supplies:
• Cues that prompt recognition (map MSS variables into
“subjective,” discrete cues)
• Alternative courses of action (usually given by the structure
of the task network)
• Reinforcement (seat of the pants)
• Set run-time properties and parameters (seat of the pants)
• This defines the structure of each “trace”—
a individual decision making experience
comprising the cue values at decision
time, the action that was taken and the
outcome (good or bad)
What You Get
Advanced Decision Architectures Collaborative Technology Alliance
• Four applications (validation studies); two
flavors:
– Categorization: Brunswik Faces and Weather
Prediction
– Dynamic behavior: Prisoners’ Dilemma and
Dynamic Stocks and Flows
Brunswik Faces
Advanced Decision Architectures Collaborative Technology Alliance
The Results
Advanced Decision Architectures Collaborative Technology Alliance
1.00
Probability of Responding Category A
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Human Data
RPD Model
ACT-R Model
No Preference
Regression Model
Weather Prediction
Advanced Decision Architectures Collaborative Technology Alliance
Pattern
1
2
3
4
5
6
7
8
9
10
11
12
13
14
1
1
1
1
1
1
1
Cues
(cards present)
2
3
2
4
2
3
4
3
4
2
2
2
2
3
3
4
4
3
3
4
4
Total
Frequency
19
9
26
9
12
6
19
19
6
12
9
26
9
19
200
Probability of
fine weather
0.895
0.778
0.923
0.222
0.833
0.500
0.895
0.105
0.500
0.167
0.556
0.077
0.444
0.105
0.500
The Results
Advanced Decision Architectures Collaborative Technology Alliance
Prisoners’ Dilemma
Advanced Decision Architectures Collaborative Technology Alliance
Player A
Cooperate
Defect
Cooperate
(3,3)
(4,0)
Defect
(0,4)
(1,1)
Player B
The Results
Advanced Decision Architectures Collaborative Technology Alliance
Dynamic Stocks and Flows
Advanced Decision Architectures Collaborative Technology Alliance
The Results
Advanced Decision Architectures Collaborative Technology Alliance
Some Interesting Comparisons
Advanced Decision Architectures Collaborative Technology Alliance
• Categorization
– Isomorphic internal representation for different
tasks
• Dynamic Models
– Very different internal representations for
similar tasks
• In general, fits are satisfying, but not very
illuminating
– Model vs modeler vs task vs ???
Developing a Science for
Simulation
Advanced Decision Architectures Collaborative Technology Alliance
• Model comparison has roots in two traditions
• The AI tradition
– Long tradition in AI of “tests” for general intelligence
– Similarly, competition has emerged a means for establishing
benchmarks of performance
– In both cases, the proof is in the pudding
• Success is the metric of performance
• The Hypothetico-Deductive tradition
– Theories generate predictions; if the predictions are confirmed
by observation, the theory is confirmed
– In this case, build a model and see if it predicts (retrodicts) actual
human performance
– Experimental science 101
Conventional Wisdom
Advanced Decision Architectures Collaborative Technology Alliance
• AI competition + HD method = Model
Comparison
– Pick a task
– Develop a bunch of models
– See which ones make the best predictions
(given some measure of goodness-of-fit)
– Declare a winner!
Familiar Concerns
Advanced Decision Architectures Collaborative Technology Alliance
• Concerns about fitting the data (does a
good fit really confirm anything?)
• Concerns about simulating the task
environment (have we made too many
simplifying assumptions?)
• Concerns about models interacting with
the task environment (is the model really
performing the task?)
• Lots of valuable and important discussions
here
A Deeper Concern
Advanced Decision Architectures Collaborative Technology Alliance
• The real focus in a model comparison
shouldn’t be on the “winner” but on
understanding how the various
approaches are implemented
– Good predictions are a minimum requirement
• The relationship between theory and
model is not easily assessed
– Often the most difficult part of the comparison
– But the most important part
• Is there anything better than a qualitative
assessment of reasonableness?
Toward a Science of Model
Comparison
Advanced Decision Architectures Collaborative Technology Alliance
• A general problem here is that the history
of computer simulation as experiment is
not yet well understood
– Cognitive models are just one application
– Working at a strange intersection of theory
and engineering (cf. “Computer Science as
Empirical Inquiry”)
• Absent a theory of the simulation as
experiment, the best we can do is look at
current and, we hope, best practices