Markov Chains
Chia-Ping Chen
Professor
Department of Computer Science and Engineering
National Sun Yat-sen University
Probability
Dependence and Conditional Independence
In general, random variables in a process are dependent.
A Markov process, or Markov chain, allows
dependence of random variables.
In a Markov chain, instead of complete independence,
conditional independence property is assumed.
Prof. C. Chen
Markov Chains
Discrete-time Markov Chains
Prof. C. Chen
Markov Chains
State
The value of a Markov process is often referred to as a state.
The set of possible states of a random process is called
state space
A sample path of a random process is a sequence of states
Prof. C. Chen
Markov Chains
Markov Assumption
For a discrete-time Markov chain
S0 , S1 , . . .
it is assumed that the conditional independence relationships
{Sn+1 , Sn+2 , . . . } ⊥
⊥ {Sn−1 , . . . , S0 } | Sn ,
hold.
The future is independent of the past given now.
Prof. C. Chen
Markov Chains
∀n∈N
Representation
A Markov chain can be represented by a state transition
graph or a transition probability matrix.
state transition graph
node: state
edge: state transition
annotation: transition probability
transition probability matrix

p11

 p21
T=
 ..
 .
p12
p22
..
.
...
...
..
.
pm1 pm2 . . .

p1m
p2m 

.. 
,
. 
pij = P(Sn+1 = j|Sn = i)
pmm
Prof. C. Chen
Markov Chains
Example 7.1 Catch-up
Alice is taking a probability course.
In each week, she is either up-to-date or fallen-behind.
If she is up-to-date in a given week, the probability that
she will be up-to-date in the next week is 0.8.
If she is fallen-behind in a given week, the probability that
she will be up-to-date in the next week is 0.6.
Construct a Markov chain.
Prof. C. Chen
Markov Chains
Example 7.2 Flies and Spiders
A fly moves along a straight line in unit increments.
At each time period, it moves one unit to the left with
probability 0.3, one unit to the right with probability 0.3,
and stays in place with probability 0.4.
Two spiders are lurking at positions 1 and m: if the fly
lands there, it is captured by a spider.
Construct a Markov chain.
Prof. C. Chen
Markov Chains
Example 7.3 Machines
A machine can be either working or broken down on a given day.
If it is working, it will break down the next day with
probability b, and will continue working with probability
1 − b.
If it breaks down on a given day, it will be repaired and be
working on the next day with probability r, and will
continue to be broken down with probability 1 − r.
Construct a Markov chain.
Suppose whenever the machine remains broken down for l days,
it is replaced by a new working machine. What is the new
Markov chain?
Prof. C. Chen
Markov Chains
Sequence of States
The probability of a sequence of states
S0 = s0 , . . . , Sn = sn
is
P(S0 = s0 , . . . , Sn = sn )
= P(S0 = s0 )P(S1 = s1 |S0 = s0 ) . . . P(Sn = sn |Sn−1 = sn−1 , . . . )
= P(S0 = s0 )P(S1 = s1 |S0 = s0 ) . . . P(Sn = sn |Sn−1 = sn−1 )
= π s0
" n
Y
#
psk−1 sk
k=1
where
πi = P(S0 = i)
is the probability of an initial state.
Prof. C. Chen
Markov Chains
n-Step State Transition Probability
The n-step state transition probability is the probability of
starting at one state and ending at another state in n steps
rij (n) = P(Sn = j|S0 = i)
The n-step state transition probabilities constitute a matrix
R(n) = {rij (n)}
called the n-step state transition probability matrix.
Prof. C. Chen
Markov Chains
Chapman-Kolmogorov Equation
For n ≥ 2
rij (n) =
m
X
rik (n − 1)pkj
k=1
rij (n) = P(Sn = j|S0 = i)
=
=
=
=
m
X
k=1
m
X
k=1
m
X
k=1
m
X
P(Sn = j, Sn−1 = k|S0 = i)
P(Sn−1 = k|S0 = i)P(Sn = j|Sn−1 = k, S0 = i)
P(Sn−1 = k|S0 = i)P(Sn = j|Sn−1 = k)
rik (n − 1)pkj
k=1
Prof. C. Chen
Markov Chains
1-Step and n-Step Transition Probability Matrix
R(n) = Tn
The Chapman-Kolmogorov equation can be written as
R(n) = R(n − 1)T
By definition
R(1) = T
By induction
R(n) = Tn
Prof. C. Chen
Markov Chains
Visualization
State sequences of a Markov chain can be visualized through
state transition graph or trellis.
state transition graph
a sequence of states is a path on the state transition graph
each path on the state transition graph has a probability
an n-step transition probability is the sum of probabilities
of the paths with designated start state and end state
trellis
an expansion of the state transition graph over time
Prof. C. Chen
Markov Chains
Classification of States
Prof. C. Chen
Markov Chains
Accessibility between States
State j is said to be accessible to state i if
∃n such that rij (n) > 0
This is denoted by
i → j
Alternative interpretations
on state transition graph, there is a path from i to j with
the edge directions along the path
starting from i, the probability of reaching j is non-zero
starting from i over and over again, j will be reached
eventually
Prof. C. Chen
Markov Chains
Recurrent State and Transient State
A state is recurrent if it is accessible to every state that is
accessible to it. That is, i is recurrent if
i → j ⇒ j → i
for every j.
A non-recurrent state is transient. That is, i is transient if
there exists j such that
i → j
but
j 9 i
Prof. C. Chen
Markov Chains
Revisiting Property
A recurrent state, once visited, is certain to be revisited.
Imagine a token traveling in the state transition graph of a
discrete-time Markov chain. The difference between a recurrent
state and a transient state is not whether the token will leave,
but whether the token will return.
Prof. C. Chen
Markov Chains
Recurrent Classes
The recurrent states of a Markov chain can be grouped into one
or more classes, such that
recurrent states in the same class are accessible
recurrent states in different classes are not accessible
Prof. C. Chen
Markov Chains
Periodic Class
A class is periodic if it can be partitioned into multiple subsets
C = S1 ∩ · · · ∩ Sd
so that all valid state transitions are from one subset to the
next subset cyclicly
S1 → S2 → · · · → Sd → S1
A class is aperiodic if it is not periodic. We focus on aperiodic
classes unless stated otherwise.
Prof. C. Chen
Markov Chains
Long-term Behavior
Prof. C. Chen
Markov Chains
Steady-State Probability
For a Markov chain with a single aperiodic recurrent class,
the steady-state probability of a state is defined by
πj = lim rij (n)
n→∞
Note that it is independent of the initial state.
Prof. C. Chen
Markov Chains
Probability of a State
πj is approximately the probability of Xn = j for large n.
Conditioned on the initial state, we have
P(Xn = j) =
=
m
X
i=1
m
X
P(Xn = j|X0 = i)P(X0 = i)
rij (n)P(X0 = i)
i=1
For a large n
P(Xn = j) ≈ πj
m
X
P(X0 = i) = πj
i=1
Prof. C. Chen
Markov Chains
Balance Equations
m
X
πj =
πk pkj
k=1
By the Chapman-Kolmogorov equations
rij (n) =
m
X
rik (n − 1)pkj
k=1
Let n → ∞
m
X
πj =
πk pkj
k=1
Balance equations can be represented by vectors. Let π be the
column vector of steady-state probabilities
π0 = π0P
Prof. C. Chen
Markov Chains
Example 7.5 Two-state Markov Chain
Consider 2-state Markov chain with transition probabilities
p11 = 0.8,
p12 = 0.2
p21 = 0.6,
p22 = 0.4
Find the steady-state probabilities.
π must satisfy the balance equations
π1 = π1 p11 + π2 p21 = 0.8π1 + 0.6π2
π2 = π1 p12 + π2 p22 = 0.2π1 + 0.4π2
and sum to 1
1 = π1 + π2
Prof. C. Chen
Markov Chains
Example 7.6 Don’t Forget Umbrellas
An absent-minded professor has 2 umbrellas that he uses when
commuting from home to office and back. If it rains and an
umbrella is available in his location, he takes it. If it is not
raining, he always forgets to take an umbrella. Suppose that it
rains with probability p each time he commutes. What is the
steady-state probability that he gets wet during a commute?
Let state i represent that i umbrellas are with the professor.
π0 = π2 (1 − p)
π1 = π1 (1 − p) + π2 p
π2 = π0 · 1 + π1 p
1 = π0 + π1 + π2
The probability that he gets wet during a commute is pπ0 .
Prof. C. Chen
Markov Chains
Example 7.7 Superstition
A superstitious professor works in a circular building with m
doors, where m is odd, and never uses the same door twice in a
row. Instead, he uses with probability p (or probability 1 − p)
the door that is adjacent in the clockwise (or counter-clockwise)
direction to the door he used last. What is the probability that
a given door will be used on some particular day far into the
future?
Let the doors be numbered from 0 to m − 1 along clockwise
direction. Let state i represents that door i is used by the
professor in a given day. By symmetry
πi =
Prof. C. Chen
1
m
Markov Chains
Long-term Frequency of a State
For a Markov chain with single aperiodic class, the steady-state
probability πj is the long-term frequency of visits to state j.
Starting from state i, let vij (n) be the number of visits to
state j within the first n transitions. Then
vij (n) a. s.
−−−−→ πj
n
Prof. C. Chen
Markov Chains
Long-term Frequency of a State Transition
For a Markov chain with single aperiodic class, πj pjk is the
long-term frequency of state transitions from j to k.
Starting from state i, let qjk (n) be the number of transitions
from state j to state k within the first n transitions. Then
qjk (n)
vij (n) qjk (n) a. s.
=
−−−−→ πj pjk
n
n vij (n)
Prof. C. Chen
Markov Chains
Frequency Interpretation for Balance Equations
Consider a balance equation
πj =
X
πk pkj
k
left side: the long-term frequency of visits to state j
right side: the sum of long-term frequencies of state
transitions to state j
equality: a transition to j is a visit to j
Prof. C. Chen
Markov Chains
Birth-Death Process
A birth-death process is a Markov chain with a linear
transition graph, in which only self-transitions or transitions
between neighboring states are allowed.
A transition to a higher-indexed state is called a birth.
The probability of birth at state i is denoted by bi
P(Sn+1 = i + 1|Sn = i) = bi
A transition to a lower-indexed state is called a death.
The probability of death at state i is denoted by di
P(Sn+1 = i − 1|Sn = i) = di
Prof. C. Chen
Markov Chains
Local Balance Equations
For a birth-death process, the steady-state probabilities satisfy
local balance equations
πi−1 bi−1 = πi di ,
i = 2, . . . , m
For any n, the number of transitions from i to i − 1 and the
number of transitions from i − 1 to i differ by at most 1
qi,i−1 (n) = qi−1,i (n) ± 1
⇒
qi−1,i (n) 1
qi,i−1 (n)
=
±
n
n
n
⇒ πi−1 pi−1,i = πi pi,i−1
⇒ πi−1 bi−1 = πi di
Prof. C. Chen
Markov Chains
Simplification
Every local balance equation involves only two adjacent states,
so the computation of steady-state probabilities is simplified.
π1 b1 = π2 d2
..
.
πm−1 bm−1 = πm dm
⇒ π2 = π1
b1
,
d2
...,
πm = π1
b1 . . . bm−1
d2 . . . dm
Normalize total probability to 1 to find π1 , then find π2 , . . . , πm .
Prof. C. Chen
Markov Chains
Example 7.8 Random Walk
English walks along a straight line and, at each time period,
takes a step to the right with probability b, and a step to the
left with probability 1 − b. She starts in one of the positions
1, . . . , m, but if she reaches the position 0 (or position m + 1),
her step is instantly reflected back to 1 (or m, respectively). We
introduce a Markov chain model whose states are the positions
1, . . . , m. What are the steady-state probabilities?
Prof. C. Chen
Markov Chains
Example 7.9 Buffer Size
Packets arrive at a node of a communication network, where
they are stored in a buffer and then transmitted. Let the size of
the buffer be m. If m packets are already present, any newly
arriving packets are discarded. We discretize time in very small
periods, and we assume that in each period, at most one event
can happen that can change the number of packets stored in the
node. We assume that at each period, exactly one of the
following occurs:
one new packet arrives, with probability b > 0
one existing packet completes transmission, with
probability d > 0 if there is at least one packet in the node,
and with probability 0 otherwise
no new packet arrives and no existing packet completes
transmission, with probability 1 − b − d if there is at least
one packet in the node, and with probability 1 − b otherwise
Prof. C. Chen
Markov Chains
The buffer can be modeled as a birth-death process, with
b0 = · · · = bm−1 = b,
d0 = 0,
bm = 0
d1 = · · · = dm = d
⇒ πi−1 b = πi d,
i = 1, . . . , m
⇒ πi = ρπi−1 = · · · = ρi π0 ,
⇒ π0 =
ρ=
b
d
1−ρ
1
=
1 + ρ + · · · + ρm
1 − ρm+1
⇒ πi = π0 ρi =
Prof. C. Chen
1−ρ i
ρ
1 − ρm+1
Markov Chains
Short-term Behavior
Prof. C. Chen
Markov Chains
Absorbing State
An absorbing state is a state making only self transitions.
(token absorbed)
That is, s is an absorbing state if
pss = 1
a transient state cannot be absorbing
an absorbing state is in a recurrent class by itself
multiple absorbing states in a Markov chain are possible
Prof. C. Chen
Markov Chains
Absorption Probability
Starting at state i, the absorption probability for a particular
absorbing state s is
ai = P
lim Sn = s S0 = i
n→∞
For a recurrent state i
ai = δis
Prof. C. Chen
Markov Chains
Equation for Absorption Probabilities
For a transient state i
ai =
ai = P
=
=
=
=
m
X
j=1
m
X
j=1
m
X
j=1
m
X
X
lim Sn = s|S0 = i
pij aj
n→∞
P
lim Sn = s, S1 = j|S0 = i
n→∞
P (S1 = j|S0 = i) P
P (S1 = j|S0 = i) P
lim Sn = s|S1 = j
n→∞
lim Sn−1 = s|S0 = j
n→∞
pij aj
j=1
Prof. C. Chen
Markov Chains
Example 7.11 Gambler’s Ruin
A gambler wins $1 at each round with probability p, and loses
$1 with probability 1 − p. He plays continuously until he either
accumulates a target amount of $m, or loses all his money.
What is the probability that he wins?
Let state i represent that gambler has i dollars. We want to
find the absorption probabilities to absorbing state m.
am = 1, a0 = 0
For i = 1, . . . , m − 1
ai =
m
X
pij aj = (1 − p)ai−1 + pai+1
j=0
Prof. C. Chen
Markov Chains
⇒ (1 − p)(ai − ai−1 ) = p(ai+1 − ai )
⇒ (ai+1 − ai ) =
⇒ δi = ρδi−1 ,
1−p
(ai − ai−1 )
p
δi = ai+1 − ai , ρ =
1−p
p
⇒ δi = ρδi−1 = · · · = ρi δ0
am − a0 = (am − am−1 ) + · · · + (a1 − a0 )
⇒ 1 − 0 = δm−1 + · · · + δ0 = δ0 (ρm−1 + · · · + 1)
⇒ δ0 =
ρm−1
1
+ ··· + 1
⇒ ai = a0 + δ0 + · · · + δi−1 =
Prof. C. Chen
ρi−1 + · · · + ρ0
ρm−1 + · · · + 1
Markov Chains
Expected Time to Absorption
The expected time to absorption for a state is defined by
µi = E[A|S0 = i]
where
A = arg min {Sn is recurrent}
n>0
is the first time a recurrent state is visited.
The name is a little confusing.
Prof. C. Chen
Markov Chains
Recursion
If i is recurrent
µi = 0
If i is transient
µi = E[A|S0 = i] = E[E[A|S1 ]|S0 = i]
=
=
=
m
X
j=1
m
X
j=1
m
X
P(S1 = j|S0 = i)E[A|S1 = j, S0 = i]
pij E[A|S1 = j]
pij (µj + 1)
j=1
=1+
m
X
pij µj
j=1
Prof. C. Chen
Markov Chains
Example 7.12 Spider-and-Fly
Consider the spider-and-fly model of Example 7.2. This
corresponds to a Markov chain. Assume m = 4. What is the
expected number of steps until the fly is captured?
States 1 and 4 are recurrent states. For the expected times to
absorption
µ1 = 0
µ2 = 1 + 0.3µ1 + 0.4µ2 + 0.3µ3
µ3 = 1 + 0.3µ2 + 0.4µ3 + 0.3µ4
µ4 = 0
⇒ µ2 = µ3 =
Prof. C. Chen
10
3
Markov Chains
Continuous-time Markov Chains
Prof. C. Chen
Markov Chains
Representation of a Continuous-time Markov Chain
A continuous-time Markov chain (CTMC) can be
represented by
T
T
T
T
Tn+1
Tn+2
n
S0 −−1→ S1 −−2→ S2 −−3→ . . . −−−
→ Sn −−−−→ Sn+1 −−−−→ . . .
where Tn+1 is the waiting time for state Sn to transit to state
Sn+1 .
DTMC: state transitions happen only at discrete times
CTMC: state transitions can happen at any time
Prof. C. Chen
Markov Chains
Markov Property
The Markov property, that the future is independent of the
past given the current state, holds for CTMC by assumption.
At time t, the random variables are
current state Sn
waiting time for state transition to occur Tn+1
next state when state transition occurs Sn+1
Given Sn , both Tn+1 and Sn+1 are independent of the past
P(Tn+1 , Sn+1 | Sn , {X(t0 ), t0 < t}) = P(Tn+1 , Sn+1 | Sn )
Prof. C. Chen
Markov Chains
Conditional Description
Conditioned on the current state Sn = i
Tn+1 , the time to leave i, is an exponential random
variable with some parameter νi
Sn+1 , the next state, is j (wth j 6= i) with probability pij
P(Sn+1 = j, Tn+1 ≥ t|Sn = i)
= P(Tn+1 ≥ t|Sn = i)P(Sn+1 = j|Sn = i)
= e−νi t pij
A CTMC is completely specified by a set of state transition
probabilities pij ’s and state transition rates νi ’s.
Prof. C. Chen
Markov Chains
Rate of Transitions Out of a State
Given Sn = i, the expected time to the next state transition is
E[Tn+1 |Sn = i] =
Z ∞
tνi e−νi t dt =
0
1
νi
on average, it takes νi−1 to leave state i
on average, νi transitions occur per unit time when in i
the rate of transitions out of state i is νi
Prof. C. Chen
Markov Chains
Rate of Transitions between States
When in state i
a fraction pij of the state transitions out of i go to state j
on average, the number of state transitions from i to j per
unit time is
νi pij
the rate of transitions from state i to state j is
qij = νi pij
Prof. C. Chen
Markov Chains
Example 7.14 Machines
A machine, once in production mode, operates continuously
until an alarm signal is generated. The time up to the alarm
signal is an exponential random variable with parameter 1.
Subsequent to the alarm signal, the machine is in test mode for
an exponentially distributed amount of time with parameter 5.
The test results are positive, with probability 1/2, in which case
the machine returns to production mode, or negative, with
probability 1/2, in which case the machine is taken for repair.
The duration of the repair is exponentially distributed with
parameter 3. Construct a continuous-time Markov chain.
Prof. C. Chen
Markov Chains
The states are production, test, and repair. The rates of
transitions out of a state are
ν1 = 1,
ν2 = 5,
ν3 = 3
The state transition probabilities are


0 1 0

1
P =  2 0 21 
1 0 0
The rates of transitions between states are


0 1 0

5
Q =  2 0 52 
3 0 0
Prof. C. Chen
Markov Chains
Starting with State Transition Rates
The rates of transitions out of a state and state transition
probabilities can be determined by the rates of transitions
between states.
Suppose the rates of transitions between states are
∀ i 6= j
qij ,
The rate of transitions out of a state are
νi = νi
X
pij =
j
X
νi pij =
j
X
j
The state transition probabilities are
pij =
qij
qij
=P
νi
qij 0
j0
Prof. C. Chen
Markov Chains
qij
Auxiliary Discrete-time Markov Chain
Suppose X(t) is a continuous-time Markov chain. For δ > 0,
the discrete-time random process
Z = Z0 , Z1 , . . .
where
Zn = X(nδ)
is an auxiliary discrete-time Markov chain of X(t).
Prof. C. Chen
Markov Chains
State Transition Probabilities
The state transition probabilities of Z, denoted by {pij (δ)}, are
related to the rates of transitions between states of {X(t)}.
For j 6= i
pij (δ) = P(Zn+1 = j|Zn = i)
transition out of i
transition to j
}|
z}|{
z
{
= (νi δ + o(δ)) ×
pij
= νi pij δ + o(δ)
= qij δ + o(δ)
For j = i
pii (δ) = 1 −
X
pij (δ)
j6=i
=1−δ
X
qij + o(δ)
j6=i
Prof. C. Chen
Markov Chains
Example 7.14 (continued)
Neglecting o(δ) terms, the state transition probability matrix
for an auxiliary DTMC is

1−δ


 5
 2δ


δ
0
1 − 5δ
3δ
Prof. C. Chen
0




5
δ

2


1 − 3δ
Markov Chains
Alternative Characterization of CTMC
Consider a continuous-time random process. If, conditioned on
the current state i, δ time later the state is in another state j
with probability
qij δ + o(δ), j 6= i
for qij ≥ 0, then the process is a CTMC.
Prof. C. Chen
Markov Chains
Example 7.15 Communication Network
Packets arrive at a node of communication network according
to a Poisson process with rate λ. The packets are stored at a
buffer with room for up to m packets, and are then transmitted
one at a time. However, if a packet finds a full buffer upon
arrival, it is discarded. The time required to transmit a packet
is exponentially distributed with parameter µ. Show that this
can be modeled as a continuous-time Markov chain.
Let state i represent i packets in the buffer.
P(X(t + δ) = i − 1|X(t) = i) = µδ + o(δ),
i = 1, . . . , m
P(X(t + δ) = i + 1|X(t) = i) = λδ + o(δ),
i = 0, . . . , m − 1
Thus {X(t)} is a continuous-time Markov chain.
Prof. C. Chen
Markov Chains
Steady-State Probabilities
For a CTMC X(t) with single recurrent class, each state has a
steady-state probability
lim P(X(t) = j) = πj
t→∞
The steady-state probabilities of a CTMC agree with those of
an auxiliary DTMC
Zn = X(nδ)
Prof. C. Chen
Markov Chains
Balance Equations
The steady-state probabilities of a CTMC satisfy balance
equations
X
X
πj
qjk =
πk qkj , ∀ j
k6=j
k6=j
For auxiliary DTMC Zn = X(nδ)
πj =
X
πk pkj = πj pjj +
k
X
πk pkj
k6=j


⇒ πj = πj 1 − δ
X
qjk + o(δ) +
k6=j
⇒ πj
X
X
k6=j
qjk =
k6=j
Prof. C. Chen
X
πk qkj
k6=j
Markov Chains
πk (qkj δ + o(δ))
Example 7.14 Machines
What are the steady-state probabilities?
5
π1 = π2 + 3π3
2
5π2 = π1
5
3π3 = π2
2
π1 + π2 + π3 = 1
⇒ π1 =
30
6
5
, π2 = , π3 =
41
41
41
Prof. C. Chen
Markov Chains
Continuous-time Birth-Death Processes
The states in a continuous-time birth-death process are linearly
arranged and only transitions to a neighboring state are allowed.
That is
qij = 0,
for |i − j| > 1
Prof. C. Chen
Markov Chains
Local Balance Equations
The rates of transition from i to j and from j to i must be the
same
πi qij = πj qji
for j = i ± 1.
They are called local balance equations.
Prof. C. Chen
Markov Chains
Example 7.15 Communication Network
What are the steady-state probabilities?
Use local balance equations
λ
µ
πi λ = πi+1 µ ⇒ πi+1 = πi
= πi ρ,
⇒ πi = π0 ρi
π0 + π1 + · · · + πm = 1
⇒ π0 (1 + ρ + · · · + ρm ) = 1
⇒ π0 = (1 + ρ + · · · + ρm )−1
⇒ πi =
ρi
1 + ρ + · · · + ρm
Prof. C. Chen
Markov Chains
ρ=
λ
µ