Artificial intelligence
Statement
Organisation
Methods in AI Research:
Markov models for multi-agent learning
Part I: introduction and motivation
Gerard Vreeswijk
Intelligent Systems Group, Computer Science Department
Faculty of Sciences, Utrecht University, The Netherlands
October, 2016
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Outline
1
Artificial intelligence
State spaces
The state space of a robot
2
Statement
3
Organisation
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Outline
1
Artificial intelligence
State spaces
The state space of a robot
2
Statement
3
Organisation
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Automated reasoning.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Automated reasoning.
Logic.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Automated reasoning.
Logic.
Artificial languages.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Automated reasoning.
Logic.
Artificial languages.
Scripted dialogue.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Automated reasoning.
Logic.
Artificial languages.
Scripted dialogue.
Artificial life.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Automated reasoning.
Logic.
Artificial languages.
Scripted dialogue.
Artificial life.
Neural networks.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Automated reasoning.
Logic.
Artificial languages.
Scripted dialogue.
Artificial life.
Neural networks.
Machine learning.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Automated reasoning.
Logic.
Artificial languages.
Scripted dialogue.
Artificial life.
Neural networks.
Machine learning.
Embodied
intelligence.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Artificial intelligence
Playing chess, go,
backgammon, poker.
Automated reasoning.
Logic.
Artificial languages.
Scripted dialogue.
Artificial life.
Neural networks.
Machine learning.
Embodied
intelligence.
Robotics.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Outline
1
Artificial intelligence
State spaces
The state space of a robot
2
Statement
3
Organisation
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Combinatorial puzzle
Puzzle:
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Combinatorial puzzle
Puzzle:
A wolf, a goat
and a coal.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Combinatorial puzzle
Puzzle:
A wolf, a goat
and a coal.
All three
must be
brought to
the other
side.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Combinatorial puzzle
Puzzle:
A wolf, a goat
and a coal.
All three
must be
brought to
the other
side.
Boat fits one
object and
you.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Combinatorial puzzle
Puzzle:
A wolf, a goat
and a coal.
All three
must be
brought to
the other
side.
Boat fits one
object and
you.
Wolf > Goat.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Combinatorial puzzle
Puzzle:
A wolf, a goat
and a coal.
All three
must be
brought to
the other
side.
Boat fits one
object and
you.
Wolf > Goat.
Goat > Coal.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
State graph for a combinatorial puzzle
Image from http://yongouyang.
blogspot.nl/2013/04/
solving-farmer-wolf-goat-cabbage-ri
html.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
State graph for a game tree
Image from http://www.seas.gwu.edu/˜simhaweb/champalg/chess/chess.html.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
More complicated games: Hold’em
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Game tree for Rhode Island Hold’em
Image from: Finding equilibria in large sequential games of imperfect information, Gilpin & Sandholm, 2005, CMU-CS-05-158.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
Outline
1
Artificial intelligence
State spaces
The state space of a robot
2
Statement
3
Organisation
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
The Italian dust bot
“Robot garbage cart set to hit Italian streets,” Posted May 29, 2009 - 10:58 by Emma Woollacott.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
The Italian dust bot
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
The tin can robot
From: Barto and Sutton, Reinforcement Learning: An Introduction, MIT Press, 1998.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
State spaces
The state space of a robot
State graph for a tin can robot
From: Barto and Sutton, Reinforcement Learning: An Introduction, MIT
Press, 1998.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Outline
1
Artificial intelligence
State spaces
The state space of a robot
2
Statement
3
Organisation
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
AI decision makers must deal with an uncertain world.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
AI decision makers must deal with an uncertain world.
A dominant approach in AI is to represent the world in terms
of states, and transitions between states.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
AI decision makers must deal with an uncertain world.
A dominant approach in AI is to represent the world in terms
of states, and transitions between states.
To this end, text books on artificial intelligence discuss these
and much more.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
AI decision makers must deal with an uncertain world.
A dominant approach in AI is to represent the world in terms
of states, and transitions between states.
To this end, text books on artificial intelligence discuss these
and much more.
Markov decision processes.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
AI decision makers must deal with an uncertain world.
A dominant approach in AI is to represent the world in terms
of states, and transitions between states.
To this end, text books on artificial intelligence discuss these
and much more.
Markov decision processes.
Learning the actual state space (≈ reinforcement learning).
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
AI decision makers must deal with an uncertain world.
A dominant approach in AI is to represent the world in terms
of states, and transitions between states.
To this end, text books on artificial intelligence discuss these
and much more.
Markov decision processes.
Learning the actual state space (≈ reinforcement learning).
Hidden Markov models (HMM).
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
AI decision makers must deal with an uncertain world.
A dominant approach in AI is to represent the world in terms
of states, and transitions between states.
To this end, text books on artificial intelligence discuss these
and much more.
Markov decision processes.
Learning the actual state space (≈ reinforcement learning).
Hidden Markov models (HMM).
These books cover a lot!
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
AI decision makers must deal with an uncertain world.
A dominant approach in AI is to represent the world in terms
of states, and transitions between states.
To this end, text books on artificial intelligence discuss these
and much more.
Markov decision processes.
Learning the actual state space (≈ reinforcement learning).
Hidden Markov models (HMM).
These books cover a lot! They therefore have little space to
lay a firm mathematical basis.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Statement
AI decision makers must deal with an uncertain world.
A dominant approach in AI is to represent the world in terms
of states, and transitions between states.
To this end, text books on artificial intelligence discuss these
and much more.
Markov decision processes.
Learning the actual state space (≈ reinforcement learning).
Hidden Markov models (HMM).
These books cover a lot! They therefore have little space to
lay a firm mathematical basis.
“Markov models for multi-agent learning” attempts to
provide some of this mathematical basis.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Outline
1
Artificial intelligence
State spaces
The state space of a robot
2
Statement
3
Organisation
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Organisation
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Organisation
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Organisation
This week
Basic concepts.
Conditional expectation.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Organisation
This week
Second week
Basic concepts.
Conditional expectation.
Markov processes.
Markov reward processes.
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Organisation
This week
Basic concepts.
Conditional expectation.
Second week
Markov processes.
Markov reward processes.
November 9
Exam.
(And WSOP 2016 Final
Table without Will Kassouf.)
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning
Artificial intelligence
Statement
Organisation
Organisation
This week
Basic concepts.
Conditional expectation.
Second week
Markov processes.
Markov reward processes.
November 9
Exam.
(And WSOP 2016 Final
Table without Will Kassouf.)
All information on this page, and only on this page:
http://www.cs.uu.nl/docs/vakken/mmair/
Gerard Vreeswijk
MAIR: Markov models for multi-agent learning