Chapter 16
Markov Analysis
Prepared by Lee Revere and John Large
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-1
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Learning Objectives
Students will be able to:
• Determine future states or conditions
using Markov analysis.
• Compute long-term or steady-state
conditions using only the matrix of
transition.
• Understand the use of absorbing
state analysis in predicting future
conditions.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-2
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Chapter Outline
16.1 Introduction
16.2 States and State Probabilities
16.3 Matrix of Transition Probabilities
16.4 Predicting Future Market Share
16.5 Markov Analysis of Machine
Operations
16.6 Equilibrium Conditions
16.7 Absorbing States and the
Fundamental Matrix: Accounts
Receivable Applications
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-3
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Introduction
Markov Analysis
• A technique dealing with probabilities of
future occurrences with currently known
probabilities
• Numerous applications in
• Business (e.g., market share analysis),
• Bad debt prediction
• University enrollment predictions
• Machine breakdown prediction
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-4
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Markov Analysis
Matrix of Transition Probabilities
 Shows the likelihood that the system will
change from one time period to the next
 This is the Markov Process.
• It enables the prediction of future states or
conditions.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-5
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
States and State
Probabilities
States are
• used to identify all possible conditions of a
process or system
• A system can exist in only one state at a
time. Examples include:
• Working and broken states of a machine
• Three shops in town, with a customer able to
patronize one at a time
• Courses in a student schedule with the
student able to occupy only one class at a
time
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-6
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Assumptions of
Markov Analysis
1. A finite number of possible states
2. Probability of changing states remains
the same over time
3. Future state predictable from previous
state and the transition matrix
4. Size and states of the system remain
the same during analysis
5. States are collectively exhaustive
• All possible states have been identified
6. States are mutually exclusive
• Only one state at a time is possible
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-7
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
States and State
Probabilities continued
1. Identify all states.
2. Determine the probability that the
system is in this state.
•
This information is placed into a vector of
state probabilities.
p(i) = vector of state probabilities for period i
= (p1, p2, p3,…,pn)
where
n = number of states
p1, p2,…,pn = P (being in state 1, 2, …, state n)
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-8
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
States and State
Probabilities continued
For example:
•
If dealing with only 1 machine, it may be
known that it is currently functioning
correctly.
•
The vector of states can then be shown.
p(1) = (1,0)
where
p(1) = vector of states for the machine in period 1
p1 = 1 = P (being in state 1) = P (machine working)
p2 = 0 = P (being in state 2) = P (machine broken)
•
Most of the time, problems deal with more
than one item!
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-9
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
States and State
Probabilities continued
Three Grocery Store example:
•
100,000 customers monthly for the 3 grocery
stores
•
State 1 = store 1 = 40,000/100,000 = 40%
•
State 2 = store 2 = 30,000/100,000 = 30%
•
State 3 = store 3 = 30,000/100,000 = 30%
vector of state probabilities:
p(1) = (0.4,0.3,0.3)
where
p(1) = vector of state probabilities in period 1
p1 = 0.4 = P (of a person being in store 1)
p2 = 0.3 = P (of a person being in store 2)
p3 = 0.3 = P (of a person being in store 3)
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-10
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
States and State
Probabilities continued
Three Grocery Store example, continued:
•
Probabilities in the vector of states for the stores
represent market share for the first period.
•
In period 1, the market shares are
•
Store 1: 40%
•
Store 2: 30%
•
Store 3: 30%
•
But, every month, customers who frequent one
store have a likelihood of visiting another store.
•
Customers from each store have different
probabilities for visiting other stores.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-11
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
States and State
Probabilities continued
Three Grocery Store example, continued:
Store-specific customer probabilities for visiting a
store in the next month:
Store 1:
Store 2:
Return to Store 1 = 80%
Visit Store 1 = 10%
Visit Store 2 = 10%
Return to Store 2 = 70%
Visit Store 3 = 10%
Visit Store 3 = 20%
Store 3:
Visit Store 1 = 20%
Visit Store 2 = 20%
Return to Store 3 = 60%
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-12
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
States and State
Probabilities continued
Three Grocery Store example, continued:
Combining the starting market share with the
customer probabilities for visiting a store next period
yields the market shares in the next period:
Initial
Share P(1)
P(2)
P(3)
40%
80%
10%
10%
32%
4%
4%
10%
70%
20%
3%
21%
6%
20%
20%
60%
Next Period:
6%
6%
18%
New Shares:
41%
Store 1:
Next Period:
Store 2:
30%
Next Period:
Store 3:
30%
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-13
31% 28% = 100%
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Matrix of Transition
Probabilities
To calculate periodic changes, it is much more
convenient to use
• a matrix of transition probabilities.
• a matrix of conditional probabilities of being in
a future state given a current state.
Let Pij = conditional probability of being in state j
in the future given the current state of i, P (state
j at time = 1 | state i at time = 0)
For example, P12 is the probability of being in
state 2 in the future given the event was in state
1 in the prior period
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-14
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Matrix of Transition
Probabilities continued
Let P = matrix of transition probabilities
Row
Sum
P=
P11 P12 P13 ***** P1n
1
P21 P22 P23 ***** P2n
1
*
*
*
*
Pm1
*
*
*
*
******
Pmn
1
Important:
Each row must sum to 1.
But, the columns do NOT necessarily sum to 1.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-15
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Matrix of Transition
Probabilities continued
Three Grocery Stores, revisited
The previously identified transitional
probabilities for each of the stores can
now be put into a matrix:
P=
0.8
0.1
0.1
0.1
0.7
0.2
0.2
0.2
0.6
Row 1 interpretation:
0.8 = P11 = P (in state 1 after being in state 1)
0.1 = P12 = P (in state 2 after being in state 1)
0.1 = P13 = P (in state 3 after being in state 1)
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-16
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Predicting Future
Market Shares
Grocery Store example
A purpose of Markov analysis is to predict the future
•
Given the
1.
vector of state probabilities and
2.
matrix of transitional probabilities.
It is easy to find the state probabilities in the future.
•
This type of analysis allows the computation of the
probability that a person will be at one of the grocery
stores in the future.
•
Since this probability is equal to market share, it is
possible to determine the future market shares of the
grocery store.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-17
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Predicting Future Market
Shares continued
Grocery Store example
When the current period is 0, finding the state
probabilities for the next period (1) can be found
using:
p(1) = p(0)P
Generally, in any period n, the state probabilities for
period n+1 can be computed as:
p(n+1) = p(n)P
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-18
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Predicting Future
States continued
π = state probabilities
p (0) = [.4 .3 .3
]
.8 .1 .1


P =  . 1 .7 .2 
.2 .2 .6
p (1) = p (0)P
p (1) = [.4 .3 .3
]
.8 .1 .1


.
1
.
7
.
2


.2 .2 .6 
p ( 1) = [(.4*.8 + .3*.1 + .3*.2) +
(.4*.1 + .3*.7 + .3*.2) +
(.4*.1 + .3*.2 + .3*.6)]
p (1) = [ 0.41 0.31 0.28]
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-19
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Predicting Future Market
Shares continued
In general,
p(n) = p(0)Pn
Therefore, the state probabilities n periods in
the future can be obtained from the
•
current state probabilities and the
•
matrix of transition probabilities.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-20
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Markov Analysis of
Machine Operations
P=
0.8 0.2
0.1 0.9
where
P11 = 0.8 = probability of machine working
this period if working last period
P12 = 0.2 = probability of machine not
working this period if working last
P21 = 0.1 = probability of machine working
this period if not working last
P22 = 0.9 = probability machine not working
this period if not working last
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-21
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Markov Analysis of
Machine Operations
continued
What is the probability the machine will be
working next month?

p(1) = p(0)P
0.8 0.2
= (1,0)
0.1 0.9
= [(1)(0.8)+(0)(0.1), (1)(0.2)+(0)(0.9)]
= (0.8, 0.2)
If the machine works this month, then there
is
• an 80% chance that it will be working
next month and
• a 20% chance it will be broken.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-22
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Markov Analysis of
Machine Operations
continued
What is the probability the machine will be
working in two months?

p(2) = p(1)P
0.8 0.2
= (0.8, 0.2)
0.1 0.9
= [(0.8)(0.8)+(0.2)(0.1), (0.8)(0.2)+(0.2)(0.9)]
= (0.66, 0.34)
If the machine works next month, then in
two months there is
• a 66% chance that it will be working
and
• a 34% chance it will be broken.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-23
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Equilibrium
Conditions
• Equilibrium state probabilities are
the long-run average probabilities
for being in each state.
• Equilibrium conditions exist if state
probabilities do not change after a
large number of periods.
• At equilibrium, state probabilities
for the next period equal the state
probabilities for current period.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-24
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Equilibrium
Conditions continued
• One way to compute the equilibrium
share of the market is to use Markov
analysis for a large number of periods
and see if the future amounts approach
stable values.
• On the next slide, the Markov analysis is
repeated for 15 periods for the machine
example.
• By the 15th period, the share of time the
machine spends working and broken is
around 66% and 34%, respectively.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-25
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Machine Example:
Periods to Reach
Equilibrium
Period
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
State 1
1.0
.8
.66
.562
.4934
.44538
.411766
.388236
.371765
.360235
.352165
.346515
.342560
.339792
.337854
16-26
State 2
0.0
.2
.34
.438
.5066
.55462
.588234
.611763
.628234
.639754
.647834
.653484
.657439
.660207
.662145
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
The Markov Process
Equilibrium Conditions
p(n) 
Current
State
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
P =
Matrix of
Transition
16-27
p(n+1)
New
State
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Equilibrium Equations
p (i + 1) = p (i)P
Assume: p (i) = [p 1
 p11
p 2 ], P = 
 p21
p12 

p22 
Then:
[p1
p 2 ]= [p 1 P11 + p 2 P21 p 1 P12 + p 2 P22 ]
or :
p 1 = p 1 P11 + p 2 P21,
Therefore:
p 2 p21
p1 =
1 - p11
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
p 2 = p 1 P12 + p 2 P22
and p 2 =
16-28
p 1 p12
1 - p22
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Equilibrium Equations
continued
It is always true that
p (next period) = p (this period) P
or
p (n+1) = p (n) P
At Equilibrium:
* p (n+1) = p (n) = p (n) P
* p (n) = p (n) P
Dropping the n term:
p=pP
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-29
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Equilibrium Equations
continued
Machine Breakdown example
At Equilibrium:
p=pP
(p1, p2) = (p1, p2)
0.8 0.2
0.1 0.9
Applying matrix multiplication:
(p1, p2) = [(p1)(0.8) + (p2)(0.1), (p1)(0.2) + (p2)(0.9)
Multiplying through yields:
p1 = 0.8 p1 + 0.1 p2
p2 = 0.2 p1 + 0.9 p2
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-30
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Equilibrium Equations
continued
Machine Breakdown example
The state probabilities must sum to 1,
therefore:
S p’s = 1
In this example, then:
p1 + p2 = 1
In a Markov analysis, there are always n state
equilibrium equations and 1 equation of state
probabilities summing to 1.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-31
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Equilibrium Equations
continued
Machine Breakdown Example
Summarizing the equilibrium equations:
p1 = 0.8 p1 + 0.1 p2
p2 = 0.2 p1 + 0.9 p2
p1 + p2 = 1
Solving by simultaneous equations:
p1 = 0.333333
p2 = 0.666667
Therefore, in the long-run, the machine
will be functioning 33.33% of the time
and broken down 66.67% of the time.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-32
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Absorbing States
• Any state that does not have a
probability of moving to another state is
called an absorbing state.
• If an entity is in an absorbing state now,
the probability of being in an absorbing
state in the future is 100%.
• An example of such a process is
accounts receivable.
• Bills are either paid, delinquent, or written
off as bad debt.
• Once paid or written off, the debt stays
paid or written off.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-33
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Absorbing States
Accounts Receivable example
• The possible states are
•
•
•
•
Paid
Bad debt
Less than 1 month old
1 to 3 months old
• A transition matrix for this would look
similar to:
Paid
Bad
<1
1-3
Paid
1
0
0.6
0.4
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
Bad
0
1
0
0.1
16-34
<1
0
0
0.2
0.3
1-3
0
0
0.2
0.2
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Markov Process
Fundamental Matrix
0
I
P=
1
0
0
1
0.6 0
0
0
0.2
0
0
0.2
0.4 0
0.3
0.2
B
A
Partition the probability matrix into 4
quadrants to make 4 new sub-matrices:
I, 0, A, and B
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-35
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Markov Process
Fundamental Matrix
continued
I
Let P = 
A
0

B
Where I = Identity matrix,
and 0 = Null matrix
Then
F = [I - B]
-1
Once F is found, multiply by the A matrix: FA
The FA indicates the probability that an amount
in one of the non-absorbing states will end up
in one of the absorbing states.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-36
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458
Markov Process
Fundamental Matrix
continued
Once the FA matrix is found, multiply by
the M vector, which is the starting values
for the non-absorbing states, MFA,
where
M = (M1, M2, M3, … Mn)
The resulting vector will indicate how
many observations end up in the first nonabsorbing state and the second nonabsorbing state, respectively.
To accompany Quantitative Analysis
for Management, 9e
by Render/Stair/Hanna
16-37
© 2006 by Prentice Hall, Inc.
Upper Saddle River, NJ 07458