IENG 362
Markov Chains
Motivation; Inventory
Inventory Level
4
Inventory
•Consider (S,s)
inventory system:
3
2
1
0
0
2
4
6
time
8
10
Motivation; Inventory
•Consider (S,s)
inventory system:
•
4
Inventory
•order at s=0
•Dt = demand in per. t
•Xt = inv. on hand at t
Inventory Level
3
2
1
0
0
2
4
6
time
8
10
Motivation; Inventory
•Consider (S,s)
inventory system:
4
Inventory
•order at s=0
•Dt = demand in per. t
•Xt = inv. on hand at t
Inventory Level
3
2
1
0
0
2
4
6
8
time
•
Inventory = 3, Dt =2,
Inventory End = 1
10
Motivation; Inventory
•Consider (S,s)
inventory system:
4
Inventory
•order at s=0
•Dt = demand in per. t
•Xt = inv. on hand at t
Inventory Level
3
2
1
0
0
2
4
6
8
time
•
Inventory = 2, Dt =3,
Inventory End = 0
10
Motivation; Inventory
•Consider (S,s)
inventory system:
4
Inventory
•order at s=0
•Dt = demand in per. t
•Xt = inv. on hand at t
Inventory Level
3
2
1
0
0
2
4
6
8
10
time
•
Inventory = 0, Dt =2,
Inventory End = 3 - 2
Motivation; Inventory
•Consider (S,s)
inventory system:
Inventory Level
•order at s=0
•Dt = demand in per. t
•Xt = inv. on hand at t
•
Inventory
4
3
2
1
0
0
2
4
6
time
max (Xt - Dt , 0)
, Xt > 1
max (3 - Dt , 0)
, Xt = 0
Xt+1 =
8
10
Inventory Example
• Recall,
•
Xt = 0, 1, 2, 3
• Let
•
Pij
= P{End Inv. = j | Start Inv. = i)
•
= P{Xt+1= j | Xt =i)
Events and Probabilities
Current
State
Events
Probability
Next
State
S=3
Demand = 0
Demand = 1
Demand = 2
Demand = 3
Demand > 3
P33
P32
P31
P30
P30
S=3
S=2
S=1
S=0
S=0
S=2
Demand = 0
Demand = 1
Demand = 2
Demand > 2
P22
P21
P20
P20
S=2
S=1
S=0
S=0
Inventory Example
P =transition matrix showing probabilities from
state i to state j.
p00 p01 p02 p03
P=
p10 p11 p12 p13
p20 p21 p22 p23
p30 p31 p32 p33
Pij = P{Xt+1=j | Xt = i)
Computing Transition Prob.
p10 = P{ X t +1 = 0 | X t = 1}
= P{Dt +1  1}
Computing Transition Prob.
p10 = P{ X t +1 = 0 | X t = 1}
= P{Dt +1  1}
Let’s suppose that demand follows a Poisson
distribution with l =1.
Computing Transition Prob.
p01 = P{ X t +1 = 0 | X t = 1}
= P{Dt +1  1}
Let’s suppose that demand follows a Poisson
distribution with l =1.
P{Dt = x} =
lx e - l
-1
e
0.36788
=
=
x!
x!
x!
Computing Transition Prob.
p10 = P{ X t +1 = 0 | X t = 1}
= P{Dt +1  1}
-1
e
=
P{Dt x} =
-1
-1
x!
e
e
+
+ ...
p10 = P{Dt+1 1} =
1! 2 !

1
-1
=e 
x =1 x !
Computing Transition Prob.
p10 = P{ X t +1 = 0 | X t = 1}
= P{Dt +1  1}
p10 = 1 - P{Dt+1= 0}
-1
e
= 1= 0.632
0!
-1
e
=
P{Dt x} =
x!
Computing Transition Prob.
p20 = P{ X t +1 = 0 | X t = 2}
= P{Dt +1  2}
= 1 - P{Dt +1  1}
Computing Transition Prob.
p20 = P{ X t +1 = 0 | X t = 2}
= P{Dt +1  2}
= 1 - P{Dt +1  1}
-1
-1
e
e
= 1- [
+
]
0! 1!
= 0.264
Computing Transition Prob.
• Class Exercise:
•
Compute the following transition probabilities
•
a.
p21
d.
p00
•
b.
p22
e.
p01
•
c.
p23
Inventory Example
• Transition Matrix
0.080 0.184 0.368 0.368
P=
0.632 0.368
0.0
0.264 0.368 0.368
0.0
0.0
0.080 0.184 0.368 0.368
Example; Weather
• Let State
•
0
=
•
1
=
dry
rain
• p00 = P{dry today | dry yesterday} = 0.7
• p01 = P{rain today | dry yesterday} = 0.3
• p10 = P{dry today | rain yesterday} = 0.5
• p11 = P{rain today | rain yesterday} = 0.5
Example; Weather
• Let State
•
0
=
•
1
=
P=
dry
rain
0.7 0.3
0.5 0.5
Example; Weather II
• Let State
•
0
=
•
1
=
•
2
=
•
3
=
dry today and yesterday
dry today and rain yesterday
rain today and dry yesterday
rain today and yesterday
• p00 = P{dry today & yesterday | dry yesterday & day before}
•
= 0.9
Example; Weather
• Let State
•
0
•
1
•
2
•
3
=
=
=
=
dry today and yesterday
dry today and rain yesterday
rain today and dry yesterday
rain today and yesterday
• p01 = P{dry today & rain yesterday | dry yesterday & day before}
•
= not possible
Example; Weather
• State
•
0
•
1
•
2
•
3
=
=
=
=
dry today and yesterday
dry today and rain yesterday
rain today and dry yesterday
rain today and yesterday
0.9 0.0 01
. 0.0
P=
0.6 0.0 0.4 0.0
0.0 0.5 0.0 0.5
0.0 0.3 0.0 0.7
Example; Gambling
•Gambler bets $1 with each play. He wins $1
with probability p and loses $1 with probability
1-p. Game ends when he wins $3 or goes broke.
Example; Gambling
•Gambler bets $1 with each play. He wins $1
with probability p and loses $1 with probability
1-p. Game ends when he wins $3 or goes broke.
•Let,
• Xt = money on hand = 0, 1, 2, 3
Example; Gambling
•Gambler bets $1 with each play. He wins $1
with probability p and loses $1 with probability
1-p. Game ends when he wins $3 or goes broke.
•Let,
• Xt = money on hand = 0, 1, 2, 3
1
0
0
1- p 0
p
P=
0 1- p 0
0
0
0
0
0
p
1
Classification of States
• Communicate
•
If Pij(n) > 0 for some n, i,j communicate
• For gambler’s ruin,
1
0
0
1- p 0
p
P=
0 1- p 0
0
0
0
0
0
p
1
Classification of States
• Communicate
•
If Pij(n) > 0 for some n, i,j communicate
• Properties,
•
•
•
1.
2.
3.
with k.
Any state communicates with itself.
If i communicates with j, j communicates with i
If i comm. with j, j comm. with k, then i comm.
Classification of States
• Irreducible
•
all states communicate.
• Inventory example,
0080
.
0632
.
=
P
0264
.
0080
.
0184
.
0368
.
0368
.
0184
.
0368
.
00
.
0368
.
0368
.
0368
.
00
.
00
.
0368
.
0
1
2
3
Classification of States
• Irreducible
• all states communicate.
• Gambler’s ruin,
1
0
0
1- p 0
p
P=
0 1- p 0
0
0
0
0
0
p
1
0
1
2
3
Classification of States
• Absorbing
• If the one-step transition probability equals
1.0
• pii = 1
1
0
0
1- p 0
p
P=
0 1- p 0
0
0
0
0
0
p
1
0
1
2
3
Period
• Gambler’s ruin
1 0
1- p 0
P=
0 1- p
0 0
0 0
p 0
0 p
0 1
0
1
2
3
• It is possible to enter state 1 at times t = 2, 4, 6, . .
.
•
period = 2
Period
•The period of state I is defined to be the integer
t (t > 1) such that Pii(n) = 0 for all values of n
other than
•t, 2t, 3t, . . .
•State with period = 1 is aperiodic