Decision Making – an Overview
göttingen
Decision Making – an Overview
Hecke Schrobsdorff
[email protected]
CNS Seminar
Nikolaus 2007
Decision Making – an Overview
What we will learn today
göttingen
Decision Making – an Overview
Decision Making Sessions
today, 6.12.
Overview on decision making (Hecke)
Emotions in decision making (Wei)
next week, 13.12.
POMDPs (Hecke)
Neural Correlates of decision making (Dima)
göttingen
Decision Making – an Overview
Outline
1
Overviewing the Overview
2
Learning vs. Decision Making
3
Neuroeconomics
4
Utility Theory
5
Social Decision Making
6
Mentalizing and Theory of Mind
7
Reinforcement Learning
8
Actor-Critic Model
9
Sutton-Barto-model
göttingen
Decision Making – an Overview
Overviewing the Overview
Overviewing the Overview
göttingen
Decision Making before Neuroscience
the basis of human decisions is interesting for a wide field
of research
very important for money making
mathematical formalization in game theory
phenomenological models of rationality
Decision Making in the Neurosciences
learning theories from AI research
finding neural correlates of everything
structurally realistic models of decision making pathways in
the brain
Decision Making – an Overview
Learning vs. Decision Making
Learning vs. Decision Making
göttingen
Asking Psychologists or Pedagogues
Learning is a relatively permanent change in behaviour.
Behaviour...
... now is how an individual decides to react on its environment.
Therefore we have a great overlap between decision
making and learning.
at least concerning computational models.
Decision Making – an Overview
Learning vs. Decision Making
What is Decision Making
göttingen
Decision Making – an Overview
Neuroeconomics
Neuroeconomics
Ferdinand Porsche
"Wir wollen Autos bauen, die keiner braucht
aber jeder haben will."
See my old talk at www.nld.ds.mpg.de/˜hecke/archiv/eco.pdf
göttingen
Decision Making – an Overview
Neuroeconomics
Rational Choice Theory
dominant theoretical paradigm in microeconomics
central to modern political science
and other disciplines such as sociology
individuals choose the best action according to stable
preference functions and constraints facing them
göttingen
Decision Making – an Overview
Neuroeconomics
Neuroeconomics
göttingen
homo economicus
is widely assumed to be a rational and self-interested decision
maker.
in reality
routines to approximate optimal decision making:
emotionally driven
experience based
taking into account decisions of others
estimate outcome of alternative actions
Review: D. Lee: Neural basis of quasi-rational decision making. Current Opinion in
Neurobiology 2006, 16: 191–198
Decision Making – an Overview
Neuroeconomics
;-)
göttingen
Decision Making – an Overview
Utility Theory
Utility Theory
Background
choice between different alternatives
choose a preferred alternative over a nonpreferred
preference might be characterized in terms of several
factors
Mathematical Formalization by Fishburn (1970)
based on binary relations
preference as a weak order
utility as an order preserving function from alternatives to
real numbers
göttingen
Decision Making – an Overview
Utility Theory
Utility Theory
göttingen
decision under certainty
assign a number (utility) to each alternative such that
an alternative is preferred over another if and only if it’s
utility is hgher
the utility of an alternative is the sum of the utilities of its
factors
question of strength of preference
decision under uncertainty
preference based on utilities of consequences
and probabilities of theses consequences
probilities are highly subjective, confidence a person has in
the truth of a proposition
Decision Making – an Overview
Utility Theory
What the World Thinks of Dec Mak
göttingen
Decision Making – an Overview
Social Decision Making
Social Decision Making
göttingen
game theory
characterizes the problem of social interaction
by a payoff matrix for multiple players and individual actions
Classical game theory (rational decisions!) predicts the
reaching of a Nash equilibrium.
Nash equilibrium
a set of strategies for all players from which no
player can increase their payoffs by altering their
strategies individually.
Decision Making – an Overview
Social Decision Making
Social Decision Making
Ultimatum Game
göttingen
Trust Game
Decision Making – an Overview
Social Decision Making
göttingen
Decision Making – an Overview
Mentalizing and Theory of Mind
Mentalizing and Theory of Mind
göttingen
Mentalizing
repeated playing with the same other players
accumulation of information about the behaviors
theory of mind is
the ability to represent
mental states of others:
beliefs
desires
knowledge
theory of mind
might play a key role in
optimizing decision making
strategies during social
interactions
mostly unique to humans
Decision Making – an Overview
Mentalizing and Theory of Mind
Prisonners Dilemma Game
göttingen
Two players choose to
Cooperate or to Defect.
They learn about the other’s
choice afterwards.
They get reward according to
the matrix on the left.
Decision Making – an Overview
Mentalizing and Theory of Mind
göttingen
Decision Making – an Overview
Reinforcement Learning
Reinforcement Learning
göttingen
Conditioning
Classical Conditioning
reward or punishment are delivered independently of any
actions
Instrumental Conditioning
reward or punishment are based on the animals behaviour
Reinforcement Learning
learning about stimuli or actions solely on the basis of
reward or punishment
minimally supervised
Decision Making – an Overview
Reinforcement Learning
Reinforcement Learning
göttingen
Predict reward: value function
V (s) = E [r (t) + γr (t + 1) + γ2r (t + 2)|s(t) = s]
Q(s, a) = E [r (t) + γr (t + 1) + γ2r (t + 2)|s(t) = s, a(t) = a]
Select action:
greedy: a = argmaxQ(s, a)
Boltzmann: P(a|s) ∝ exp[βQ(s, a)]
Update prediction: TD error
δ(t) = r (t) + γV (s(t + 1)) − V (s(t))
∆V (s(t)) = αδ(t)
∆Q(s(t), a(t)) = αδ(t)
Decision Making – an Overview
Reinforcement Learning
RL in Basal Ganglia
göttingen
Decision Making – an Overview
Actor-Critic Model
Actor-Critic Model
Critic reward prediction
Actor action selection
Learning external reward r
internal reward δ
göttingen
Decision Making – an Overview
Sutton-Barto-model
Temporal Difference Model
a model of classical conditioning
”reinforcement in classical conditioning is the time
derivative of a composite association combining innate
(US) and acquired (CS) associations”
learning the rewards to base decisions on
∆V = (level of US processing) × (level of CS processing)
∆Vi = β Ẏ × αi X̄i
göttingen
Decision Making – an Overview
Sutton-Barto-model
Sutton-Barto-model
göttingen
Decision Making – an Overview
Sutton-Barto-model
Summarizing the Overview
göttingen
Decision Making before Neuroscience
the basis of human decisions is interesting for a wide field
of research
very important for money making
mathematical formalization in game theory
phenomenological models of rationality
Decision Making in the Neurosciences
learning theories from AI research
finding neural correlates of everything
structurally realistic models of decision making pathways in
the brain
who noticed the 10 differences to the introduction?
Decision Making – an Overview
Sutton-Barto-model
göttingen