Chapter 7 : Markov Chains
ENSIIE - Operations Research Module
Dimitri Watel ([email protected])
2016-2017
Dimitri Watel
MRO Chap 07 Markov chains
Random variables and process
Definition
A random variable X is a S-valued function where S is a set of
states.
For instance, a dice roll has a value X ∈ S = {1, 2, 3, 4, 5, 6}.
Definition
A stochastic process is a family (Xt )t∈T of random variables where
T ⊂ R.
T is called the time, Xt is the state of X at time t.
For instance, we do a dice roll every second, Xt is the face of the
dice after t seconds.
Dimitri Watel
MRO Chap 07 Markov chains
Markov process and time-homogeneous process
Definition
We say a process (Xt )t∈T is a markov process if and only if the
futur depends only on the present :
∀t1 < t2 < · · · < tn < tn+1 ∈ T , ∀A ⊂ S
P(Xtn+1 ∈ A| Xt1 Xt2 , . . . , Xtn ) = P(Xtn+1 ∈ A| Xtn )
Definition
We say a process (Xt )t∈T is a time-homogeneous process if and
only if the conditional probabilities does not change when the time
grows :
∀t, t 0 ∈ T , s > 0\t + s, t 0 + s ∈ T , A ⊂ S
P(Xt+s ∈ A|Xt ) = P(Xt 0 +s ∈ A|Xt 0 )
Dimitri Watel
MRO Chap 07 Markov chains
Markov chain
Definition
A Markov chain is a time-homogeneous process and a Markov
process (Xt )t ∈T where T is a countable set.
In addition, we assume S to be finite and discrete.
T =N
Process : X1 , X2 , . . . , Xt , . . .
States : 1, 2, . . . , |S|
Dimitri Watel
MRO Chap 07 Markov chains
Transition probability
Definition
pij is the probability for the system to move from the state i to the
state j in one step. This probabilty does not depend on the moment
t ∈ T when the state of the system is i.
∀i, j ∈ S, ∃pij
\ ∀t ∈ N P(Xt+1 = j|Xt = i) = pij
Transition matrix :
1
2
p
p
11
12
p22
 p21
 .
..
 .
.
.
P= 

pi2
 pi1
 .
..
 ..
.
p|S|1 p|S|2
···
···
..
.
···
..
.
···
Dimitri Watel
j
p1j
p2j
..
.
pij
..
.
p|S|j
···
···
..
.
···
..
.
···
|S|
p1|S|
p2|S|
..
.
pi|S|
..
.









p|S||S|
MRO Chap 07 Markov chains
1
2
i
|S|
Property
The sum of the elements of a line of P is always 1.
|S|
X
pij = 1
j=1
Property : p-transitions
(P 2 )ij = P(Xt+2 = j| Xt = i)
(P 3 )ij = P(Xt+3 = j| Xt = i)
(P p )ij = P(Xt+p = j| Xt = i)
Dimitri Watel
MRO Chap 07 Markov chains
The states probability vector
Definition
We call Q(t) = (q1 (t), q2 (t), . . . , q|S| (t)) the states probability
vector. qi (t) is the probability P(Xt = i).
|S|
X
qi (t) = 1
i=1
Q(t) = Q(t − 1) · P
Q(t) = Q(t − 2) · P 2
Q(t) = Q(0) · P t
Dimitri Watel
MRO Chap 07 Markov chains
Graph of a Markov chain
Definition
The graph G = (V , A) of a Markov chain is a complete directed
graph where every node is a state S (thus V = S), and every arc
linking i to j is weighted with pij .
pi
i
Remark : the graph may contain loop arcs.
pi|S|
i
S
pi1
1
Dimitri Watel
pi
2
2
MRO Chap 07 Markov chains
Remark
(P p )ij > 0 if there exists a path with p arcs of weight non zero
between i and j.
Dimitri Watel
MRO Chap 07 Markov chains
State classification
Accessible states
A state j is accessible from i if there is a path from i to j in G .
∃p\(P p )ij > 0
Communicating states
Two states i and j are said communicating if i is accessible from j
and conversely.
Communicating class
A communicating class is a strongly connected component of G . In
other words, it is a maximal set of pairwise communicating states.
Irreducible chain
A chain with one communicating class is irreducible.
Dimitri Watel
MRO Chap 07 Markov chains
Transient and recurrent states
Definition
A state i is transient if there is a state j such that j is accessible
from i but i is not accessible from j.
It is possible to leave that state and never be able to come back.
Definition
A state i is recurrent if for every accessible state j from i, i is
accessible from j.
It is always possible to visit again a permanent state.
Definition
A state i is absorbing if pii = 1.
Dimitri Watel
MRO Chap 07 Markov chains
Property
Theorem
In a communicating class, every state is reccurent or every state is
transient.
Dimitri Watel
MRO Chap 07 Markov chains
Periodic states
(P p )ii > 0 if there exists a circuit with p arcs of weight non zero
going through i.
Let di = GCD(n \ (P n )ii > 0, n > 0).
Definition
A state i is said to be periodic if di > 1. Otherwise, the state is
aperiodic. di is the period of i.
Property
An absorbing state is aperiodic.
Theorem
The period of every states in a communicating class is the same.
Dimitri Watel
MRO Chap 07 Markov chains
Convergence to a stationary distribution
Definition
A chain is said to be regular if the stationary distribution
Q ∗ = lim Q(t) exists and is the same whatever Q(0) is.
t→+∞
Theorem
A chain is regular if and only if every recurrent state is in the same
class and if all those states are aperiodic.
Theorem
In a regular chain, P ∗ = lim P p exists and every line of P ∗ is Q ∗ .
p→+∞
(proof on board)
Dimitri Watel
MRO Chap 07 Markov chains
Compute the vector Q ∗ .
Either, we use the following system :
Q(t + 1) = Q(t) · P ⇒ Q ∗ = Q ∗ · P
AND
|S|
X
qi∗ = 1
i=1
Or we compute P ∗ .
Dimitri Watel
MRO Chap 07 Markov chains