Finite State Markov models
Checkout counter model
LECTURE 20
• Xn : state after n transitions
• Discrete time n = 0 , 1 , . . .
Markov Processes - I
• Customer arrivals : Bernoulli(p )
- geometric interarrival times
• Readings : Sections 6 .1 - 6 .2
• Customer service times : geometric (q )
Lecture outline
• “State ” Xn : number of customers at
time n
• Checkout counter example
• Markov process definition
• n -step transition probabilities
0
1
2
3
9
Time n - 1
i
0.5
0.8
Generic question :
• Does rij converge to something ?
0.5
Time n
1
p 1j
2
0.5
0.2
k
r ik (n - 1 )
pij = P (X n + 1 = j | Xn = i)
= P (X n + 1 = j | Xn = i, Xn − 1 , . . . , X 0 )
Example
rij (n ) = P (Xn = j | X 0 = i)
1
• Markov property /assumption :
(given current state , the past does not
matter)
• Modeling steps :
- identify the possible states
- mark the possible transitions
- record the transition probabilities
n -step transition probabilities
• State occupancy probabilities ,
given initial state i:
r i 1 (n - 1 )
- X 0 is either given or random
10
• Classification of states
Time 0
- belongs to a finite set, e .g ., { 1 , . . . , m }
p
kj
1
j
1
r im (n - 1 )
p mj
n=0
n=1
n=2
n=2563n=2564
r 1 1 (n )
m
r ik (n − 1 )pkj
k=1
- With random initial state :
m
P (Xn = j ) =
P (X 0 = i)rij (n )
i=1
rij (n ) =
3
1
n odd : r 2 2 (n ) = n even : r 2 2 (n ) =
m
- Key recursion :
0.5
2
r 1 2 (n )
• Does the limit depend on initial state ?
0.4
1
2
0.3
r 1 1 (n ) =
r 3 1 (n ) =
3
0.3
4
Recurrent and transient states
• State i is recurrent if:
starting from i,
and from wherever you can go ,
there is a way of returning to i
• If not recurrent , called transient
Periodic states
• A recurrent state is periodic if:
there is an integer d > 1
such that pii (k ) = 0
when k is not an integer multiple of d
r 2 1 (n ) =
9
3
4
5
6
4
7
5
6
1
8
2
8
- i transient :
P (Xn = i) → 0 ,
i visited finite number of times
• Recurrent class :
collection of recurrent states that
“communicate ” to each other
and to no other state
3
2
1
7
• Then , pii (n ) cannot converge .