Aim of the work Methodology Traffic control Conclusion
Bayesian Approach to Multi-Agent Systems
Václav Šmídl, Jan Přikryl
Institute of Information Theory and Automation, Czech Academy of Sciences, Prague, Czech
Republic
SOAS 2005, University of Paisley, 2005
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Outline
1
Aim of the work
2
Methodology: Bayesian Decision Making
Example: temperature control
Bayesian Decision-Making
Bayesian Agents
Negotiation
3
Traffic control
Introduction to Traffic Control in Cities
Agent-based control
4
Conclusion
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Aim of the work
Bayesian decision makers:
decisions = control,
communication, negotiation,
theory of adaptive control
decisions under uncertainty =
probability calculus
Consistent methodology:
Bayesian theory
Drawback: only one
decision-maker
Goal: distribution of the DM
process
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Aim of the work
Multi-agent Systems:
Bayesian decision makers:
decisions = control,
communication, negotiation,
theory of adaptive control
decisions under uncertainty =
probability calculus
Consistent methodology:
Bayesian theory
Drawback: only one
decision-maker
Goal: distribution of the DM
process
advanced theory of agent
communication and
cooperation
self-organization
utility functions for decision
making
uncertainty is often neglected
or underrated
no generally accepted
methodology of design
(consistency?)
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Example: temperature control
Fictious room:
Actions have long term consequences.
Temperature control on long horizon.
Václav Šmídl, Jan Přikryl
utia-logo
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Example: temperature control
Fictious room:
Bayesian methodology:
1
2
3
System parameterization,
Model of consequences,
Description of aims
Actions have long term consequences.
Temperature control on long horizon.
Václav Šmídl, Jan Přikryl
utia-logo
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Example: temperature control
Fictious room:
Bayesian methodology:
1
System parameterization,
θ2,t
u1,t
θ1,t
yt
Observed variables (sensors):
temperature
yt ,
u2,t
u3,t
Internal variables (state): temperature,
doors...
Θt = [θ1,t , θ2,t , θ3,t ] ,
Actions have long term consequences. Controlled: switching devices on/off
Temperature control on long horizon.
utia-logo
ut = [u1 , u2 , u3 ] .
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Formal description
Enviroment
model
Enviroment
Θt
Strategy
yt
ut
Observation
model
decision-maker
Dt
Param.
Observations
yt
Internals
Θt
Actions
u1,t , u2,t , u3,t
Experience
Dt−1
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Formal description
Enviroment
model
Enviroment
Θt
Strategy
yt
ut
Observation
model
decision-maker
Dt
Param.
Models
Observations
yt
f (yt |Θt , Dt−1 )
Internals
Θt
f (Θt |Θt−1 , ut )
Actions
u1,t , u2,t , u3,t
f (ut |Dt−1 )
Experience
Dt−1
f (Θt |Dt−1 )
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Formal description
Enviroment
model
Enviroment
Θt
Strategy
yt
ut
Observation
model
decision-maker
Dt
Param.
Models
Ideals
Observations
yt
f (yt |Θt , Dt−1 )
bI
f (yt |Θt , Dt−1 )
Internals
Θt
f (Θt |Θt−1 , ut )
bI
f (Θt |Θt−1 , ut )
Actions
u1,t , u2,t , u3,t
f (ut |Dt−1 )
bI
f (ut |Dt−1 )
Experience
Dt−1
f (Θt |Dt−1 )
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Bayesian Decision Making
Inputs of the methodology:
1
2
3
System parameterization,
Model of consequences,
Description of aims.
Outputs of the methodology:
Learning: of changes of the enviroment
R
f (θt |Dt−1 ) = {Obs.model + Internal.model + Experience}
Strategy: how to achieve our aims1
R
R
f (ut |Dt−1 ) = · · · {Obs.model + Internal.model + Ideals}
utia-logo
1 (Loss/Utility
function is statistical divergence)
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Bayesian Decision Making
Inputs of the methodology:
1
2
3
System parameterization,
Model of consequences,
Description of aims.
Outputs of the methodology:
Learning: of changes of the enviroment
R
f (θt |Dt−1 ) = {Obs.model + Internal.model + Experience}
Strategy: how to achieve our aims1
R
R
f (ut |Dt−1 ) = · · · {Obs.model + Internal.model + Ideals}
Decision making rules are output of a methodology!
If it does not work, it is the three inputs what is wrong.
utia-logo
1 (Loss/Utility
function is statistical divergence)
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Bayesian Decision Making II
Advantages:
Bayesian theory is (the only) consistent theory of decision
making under uncertainty,
fully probabilistic design: formalization of aims as distributions
and loss/utility function as statistical divergence between the
observation based distributions and the ideal ones.
closed form functional solution,
the result is special case of dynamic programing,
naturally embraces multi-criteria decision-making
many current decision making techniques can be interpreted as
special cases: (i) LQ control, (ii) dynamic Bayesian networks, (iii)
reinforcement learning.
Disadvantages:
computational issues, tractability,
need for approximations,
the assumption of only one decision maker
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
utia-logo
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Example: temperature control
Fictious room:
Parameterization:
A1
A2
yt
yt
Θ[1],t
Θ[2],t
u1,t
u2,t , u3,t
Observed:
Θ[1],t
u1,t
Unobserved:
Controlled:
yt
Θ[2],t
u2,t
Can we achive the same
performance?
u3,t
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Getting close to global optimum...
Supernatural machine:
1
2
3
4
reads local models,
builds global models,
optimizes the constrained
strategy,
sends the results back.
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Getting close to global optimum...
Supernatural machine:
1
2
3
4
Communicating agents:
reads local models,
builds global models,
optimizes the constrained
strategy,
sends the results back.
1
2
report marginal distributions
on the common variables,
merge distributions,
f[1] , f[2] → f̃[1] .
1
re-design strategies locally
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Getting close to global optimum...
Supernatural machine:
1
2
3
4
Communicating agents:
reads local models,
builds global models,
optimizes the constrained
strategy,
sends the results back.
1
2
report marginal distributions
on the common variables,
merge distributions,
f[1] , f[2] → f̃[1] .
1
re-design strategies locally
These approaches are equivalent IF the global loss function is
chosen as convex combination of KL divergences.
THEN: the optimal merging function is:
f̃[1] = αf[2] + (1 − α) f[1] .
Where α is the weighting coefficient of the loss function. How to
choose α?
Interpretable as belief (trust) in the neighbour.
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
utia-logo
Aim of the work Methodology Traffic control Conclusion
Example Bayes DM Bayes Agents Negotiation
Negotiation as decision-making process
Consider α as another variable to be chosen (decided). Following the
Baysian methodology:
1
System parameterization:
1
2
2
3
α has the role of ut , however it is not observed,
increase of uncertainty Θt = [θ1,t , . . . , θp,t , θα, ],
Model of consequences: f (Θt |Θt−1 , ut , αt ),
Description of aims: bIf (Θt |Θt−1 , ut , αt ), bIf (αt )
The optimal selection of weights is output of the methodology.
learning: f (θα,t |Dt ),
strategy: f (αt |Dt ),
Not implemented yet.
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Optimal Traffic Control
Motivation
Traffic flow in town centers is increasing
Construction of new roads is forbidden in historical areas
Can we get an improvement by better control?
Approach
Current solution: signal plans (off-line optimization), manual
switching
Feedback control: on-line optimization of green lights based on
observations
Challenges
Large uncertainty (where the cars go?, measurement accuracy)
Long-term consequences of decisions
Large dimensionality of the problem
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Model Parameterization
Two junctions connected by one arm:
u[1]
y2
ξ[2],O[2]
ξ[1],O[1]
y1
u[2]
Parameterization:
Observations (measured quantities):
Intensity yt [vehicles]. Amount of vehicles passing a traffic detector
over some period.
Occupancy O [%]. Ratio of detector occupied/not occupied times
over some period.
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Model Parameterization
Two junctions connected by one arm:
u[1]
y2
ξ[2],O[2]
ξ[1],O[1]
y1
u[2]
Parameterization:
Observations (measured quantities):
Internals (unobserved quantities):
Queue length ξ [vehicles]. How many cars is waiting.
Turning rate r [%] Where the cars go?
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Model Parameterization
Two junctions connected by one arm:
u[1]
y2
ξ[2],O[2]
ξ[1],O[1]
y1
u[2]
Parameterization:
Observations (measured quantities):
Internals (unobserved quantities):
Actions (signal plan settings):
Green time tg [sec]. The length of the green signal.
Cycle length TC [sec]. The total length of a single signal plan cycle.
Relative green u [%]. Proportion of tg to the cycle length TC .
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
utia-logo
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Model of consequences
Principles
State-space model
Physically-based (hydrodynamic flow analogy)
Basic assumption: time of travel ∼ queue length
Queues are not directly observable
Facing saturated conditions (troubles with observability and
controlability)
saturation of traffic flow [no.of cars/min]
saturation in space, congestions
Exceptional events: accidents, football matches, etc.
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Model of consequences
First attempt
Internal model (without uncertainty):
ξt
ξt−1
=A
+ But−1 + F
Ot
Ot−1
Observation model (without uncertainty):
ηt
ξt
=C
+G
Ot
Ot
Adding uncertainty:
Internal model: f (Θt |Θt−1 , ut−1 ) = N (AΘt−1 + But−1 + F, Q)
Observation model: f (yt |Θt ) = N (CΘt + G, R)
N (µ, σ) is a Gaussian pdf
A, B, C, F, G are matrices from deterministic system description
matrices Q, R describe allowed variance in the model description utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Multi-Agent Scenario
Agents interaction over a single arm:
u[1]
y2
ξ[2],O[2]
ξ[1],O[1]
y1
u[2]
Parameterization:
Observed:
Unobserved:
Controlled:
A1
yt
ξ[1],t , O[1],t
u[1],t
A2
yt
ξ[2],t , O[2],t
u[2],t
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Ideal distributions
Setup
Every agent wants to:
minimise its queue lengths:
bI
f (ξt ) = tN(0, Vξ , h0, ξmax i)
keep the maximum capacity of the intersection:
bI
f (yt |ξt ) = tN(µ(ξt ), Vy , h0, ymax i)
maximise the output flow:
bI
f (ηt |ξt ) = tN(ηmax , Vη , h0, ηmax i)
Since ξt is internal for each agent, we can exchange only marginal
distributions:
bI
f (yt |ξt ) → bI f (yt )
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Output of the methodology
Estimates of queues: f (ξt |Dt−1 )
Strategy for signal plan of green lights: f (ut |Dt−1 )
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Introduction Agent-based control
Output of the methodology
Estimates of queues: f (ξt |Dt−1 )
Strategy for signal plan of green lights: f (ut |Dt−1 )
In practical situations, we have to combine it with engineering
common sense.
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Conclusion
Bayesian Agent is:
Bayesian decision maker with ability to communicate,
An ‘agent’ working in terms of probability density functions.
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Conclusion
Bayesian Agent is:
Bayesian decision maker with ability to communicate,
An ‘agent’ working in terms of probability density functions.
Possible advantages:
solid theory of decision making under uncertainty,
communication of aim in the same terms as knowledge,
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems
Aim of the work Methodology Traffic control Conclusion
Conclusion
Bayesian Agent is:
Bayesian decision maker with ability to communicate,
An ‘agent’ working in terms of probability density functions.
Possible advantages:
solid theory of decision making under uncertainty,
communication of aim in the same terms as knowledge,
Disadvantages
operations with pdf are more computationally demanding
the need for approximation breaks optimality of results
Traffic Application
Test of different models
Communication strategy
utia-logo
Václav Šmídl, Jan Přikryl
Bayesian Approach to Multi-Agent Systems