Master 2 MOSIG
Knowledge Representation and Reasoning
Markov Chain
Elise Arnaud
Université Joseph Fourier / INRIA Rhône-Alpes
[email protected]
Markov chain
a good reference
Markov Chains, R. Norris, Cambridge series in statistical and probabilistic
mathematics
Markov chain
a little bit of history ...
Markov chains have been introduced by the Russian mathematician Andrei
Andreyevich Markov in the early 1900s.
Markov was born in 1856 in Ryazan (Russia). He became a professor at the
University of Saint-Petersburg in 1886.
Until 1900, he mostly worked in analysis and number theory. After 1900, he
worked in probability theory.
He died in 1922 at the age of 66.
Markov chains
– we observe a system at discrete times 0, 1, 2, ..., t.
– The system can be in one state of a collection of possible states
– The observation of the system is considered as an experience whose (random)
result is the state’s system → stochastic process
examples :
– state of an engine (working, not working)
– weather (rain, cloud, snow, sun)
– robot’s position on a grid
The system is evolving in time
Markov chains
Markov property : the state of a system at time t only depends on the state at
time t − 1
Knowing the present, we can forget the past to predict the future
Chaı̂nes de Markov
Markov property : the state of a system at time t only depends on the state at
time t − 1
Knowing the present, we can forget the past to predict the future
Let x be a Markov chain :
x = {x(0) , x(1) , x(2) , . . . , x(n) . . .} = {x(t) ; t > 0}
x(i) takes its value in a finite set of possible values : the state space X
(discrete / continuous state space - discrete / continuous time)
P (x(t+1) = y|x(0) = x0 , x(1) = x1 , . . . x(t) = xt ) = P (x(t+1) = y|x(t) = xt )
∀y, x0 , . . . xt ∈ X
Markov chains
Example of a Markovian process : What is the weather today ?
Let x(t) be the weather the day t. x = {x(t) , t > 0} is a Markovian process.
x is defined by :
Markov chains
Example of a Markovian process : what is the weather today ?
Let x(t) be the weather the day t. x = {x(t) , t > 0} is a Markovian process.
x is defined by :
– the state space : cloud, sun, rain, snow
Markov chains
Example of a Markovian process : What is the weather today ?
Let x(t) be the weather the day t. x = {x(t) , t > 0} is a Markovian process.
x is defined by :
– the state space : cloud, sun, rain, snow
– the initial distribution
P (x(0) = sun) = 0.2; P (x(0) = cloud) = 0.6;
P (x(0) = snow) = 0.05; P (x(0) = rain) = 0.15;
Markov chains
Example of a Markovian process : What is the weather today ?
Let x(t) be the weather the day t. x = {x(t) , t > 0} is a Markovian process.
x is defined by :
– the state space : cloud, sun, rain, snow
– the initial distribution
P (x(0) = sun) = 0.2; P (x(0) = cloud) = 0.6;
P (x(0) = snow) = 0.05; P (x(0) = rain) = 0.15;
– the transition probabilities :
P (x(t+1) = cloud|x(t) = cloud) ; P (x(t+1) = cloud|x(t) = sun)
P (x(t+1) = cloud|x(t) = rain) ; P (x(t+1) = cloud|x(t) = snow)
P (x(t+1) = sun|x(t) = cloud) ; P (x(t+1) = sun|x(t) = sun) . . .
→ represented by a transition matrix Q
Markov chains
Transition probabilities / transition matrix
The transition matrix Q is defined from the transition probabilities :
Qij = P (x(t+1) = j|x(t) = i) = pij (∀t)
Markov chains
Transition probabilities / transition matrix
The transition matrix Q is defined from the transition probabilities :
Qij = P (x(t+1) = j|x(t) = i) = pij (∀t)
Example of a Markovian process : what is the weather today ?
cloud sun rain snow
cloud
Q = sun
0.6
0.2
0.2
0
0.7
0.3
0
0
0
0.4
0.05
0
0.5
0.5
rain 0.55
snow
0
Markov chains
Transition matrix / representation using a grapical model
We usually represent Q using an oriented graph : the nodes corresponds to the m
possible states and the directed edges to the ordered pairs (i, j) such that pij > 0.
Markov chains
Transition matrix / representation using a grapical model
We usually reprensent Q using an oriented graph : the nodes corresponds to the m
possible states and the directed edges to the ordered pairs (i, j) such that pij > 0.
Example of a Markovian process : what is the weather today ?
0.4
0.5
0.05
rain
snow
0.5
0.55
0.2
0.2
cloud
sun
0.7
0.6
0.3
Markov chains
A Markov chain is a random walk on an oriented graph where the weights of the
arrows are the transition probabilities (therefore the following condition : the sum
of outgoing arrows from a node is equal to 1).
To define a Markov chain, x = {x(t) ; t > 0}, one needs :
– the state space X (the m possible values is the state space is discrete)
(0)
(0) (0)
– the initial distribution (law of x(0) : µ(x(0) ) = ( p(0)
nu pso ppl pne ))
– the transition matrix Q
To do inference calculation, we will use the joint law :
Y
(0:t)
(0)
p(x
) = p(x )
p(x(n) |x(n−1) )
n=1:t
Markov chains - properties
– the chain is said to be invariant if Q(x(t) |x(t−1) ) , Q
∀t
Markov chains - properties
– the chain is said to be invariant if Q(x(t) |x(t−1) ) , Q
∀t
– the chain is said to be irreducible if it is possible to go from any state to
anyone else (but not necesserally in one step).
∀i, j ∈ X,
∃ n t.q. P (x(n) = j|x(0) = i) > 0
(i.e. there exists a path from i to j in the graph)
Markov chains - properties
– the chain is said to be invariant if Q(x(t) |x(t−1) ) , Q
∀t
– the chain is said to be irreducible if it is possible to go from any state to
anyone else (but not necesserally in one step).
∀i, j ∈ X,
∃ n t.q. P (x(n) = j|x(0) = i) > 0
(i.e. there exists a path from i to j in the graph)
– the chain is said to be aperiodic if it does not get stuck in a cycle
Markov chains - properties, invariant distribution
Example of a Markovian process : what is the weather today ?
If we observe the weather during an infinite number of days, will it be possible
to determine P (x(∞) = sun) ; P (x(∞) = cloud) ; P (x(∞) = rain) ;
P (x(∞) = snow) ?
Can we give, in expectation, the probability that the weather is rain, snow, sun,
cloud ?
→ Does it exist a stationnary distribution towards wich will converge the chain,
whatever is the weather the initial day ?
Markov chains - properties, invariant distribution
If a Markov chain is aperiodic and irreducible, then it exists a unique stationnary
distribution, i.e., from any initial state, the chain will converge towards an
invariant distribution π(x)
π(x) = lim µ(x(n) ) = lim µ(x(0) ) Qn
n→∞
n→∞
π=Qπ
Markov chains - properties, invariant distribution
Example of a Markovian process : what is the weather today ?
irreducible ? aperiodic ?
0.4
0.5
0.05
rain
snow
0.5
0.55
0.2
0.2
cloud
sun
0.7
0.6
0.3
Markov chains - properties, invariant distribution
Example of a Markovian process : what is the weather today ?
irreducible OK ; aperiodic OK ⇒ ∃ an invariant distribution such as
πQ = π = πI ⇒ π(I − Q) = 0
20
1 0
13
1 0 0 0
0.6 0.2 0.2 0
6B
C B
C7
6B 0 1 0 0 C B 0.7 0.3 0
7
0 C
6B
C B
C7
0 = π 6B
C−B
C7
6B 0 0 1 0 C B 0.55 0 0.4 0.05 C7
4@
A @
A5
0 0 0 1
0
0 0.5 0.5
0
1
0.4 −0.2 −0.2
0
B
C
” B −0.7 0.7
“
0
0 C
B
C
0 = pnu pso ppl pne B
C
B −0.55 0
0.6 −0.05 C
@
A
0
0 −0.5 0.5
Markov chains - properties, invariant distribution
Example of a Markovian process : what is the weather today ?
irreducible OK ; aperiodic OK ⇒ ∃ an invariant distribution such as
πQ = π = πI ⇒ π(I − Q) = 0
0
0=
“
pnu pso ppl pne
0.4
−0.2 −0.2
0
1
B
C
” B −0.7 0.7
0
0 C
B
C
B
C
B −0.55 0
0.6 −0.05 C
@
A
0
0 −0.5 0.5
We also have the condition pnu + pso + ppl + pne = 1
⇒ linear system, with 4 unknows, 5 equations.
Markov chains - properties, invariant distribution
invariant distribution π(x) independant from initial condition
Markov chains
c Wikipedia)
Bob’s example (
Bob, the lazy hamster knows only 3 places in his cage : the wood where he sleeps, the
place where he eats and the wheel where he practices sports. Every minutes, Bob can
change his activity or continue the current one. Bob is a process without memory ...
– When he sleeps, there is 9 chances over 10 that he will not wake up the next minute.
– When he wakes up, there is 1 chance over 2 that he is going to eat, and 1 chance over
2 that he will go in his wheel a little.
– The meal lasts only 1 minute.
– After eating, there is 3 chances over 10 that he will run on his weel, and 3 chances
over 10 that he will go back to sleet.
– Running is tiring ... so there is 8 chances over 10 that Bob goes back to sleep after
one minute. Otherwise he continues running.
Markov chains
Bob’s example
Markov chains
Bob’s example
food
Markov chains
Bob’s example
The transition matrix of the system is the following (lines and columns correspond to the
ordered states : to sleep, to eat, to run) :
1
0
0.9 0.05 0.05
C
B
B
Q = @ 0.7 0
0.3 C
A
0.8 0
0.2
“
”
(0)
If Bob sleeps the first minute of the study :µ(x ) = 1 0 0
After one minute, we can predict : µ(x(1) ) = µ(x(0) ) Q =
After two minutes, we can predict : µ(x(2) ) =
more generally, after t minutes : µ(x(t) ) = µ(x(t−1) ) Q = µ(x(0) ) Qt
at the limit : π =
Markov chains
Bob’s example
The transition matrix of the system is the following (lines and columns correspond to the
ordered states : to sleep, to eat, to run) :
1
0
0.9 0.05 0.05
C
B
B
Q = @ 0.7 0
0.3 C
A
0.8 0
0.2
“
”
(0)
If Bob sleeps the first minute of the study :µ(x ) = 1 0 0
“
”
(1)
(0)
After one minute, we can predict : µ(x ) = µ(x ) Q = 0.9 0.05 0.05
“
”
(2)
(1)
(0)
2
After two minutes, we can predict : µ(x ) = µ(x ) Q = µ(x ) Q = 0.885 0.045 0.07
more generally, after t minutes : µ(x(t) ) = µ(x(t−1) ) Q = µ(x(0) ) Qt
at the limit : π =
“
0.884 0.00442 0.0718
i.e. Bob is sleeping 88.4% of his time !
”