Chapter 8
Markov Processes
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
1 of 60
Outline
8.1 The Transition Matrix
8.2 Regular Stochastic Matrices
8.3 Absorbing Stochastic Matrices
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
2 of 60
8.1 The Transition Matrix
1.
2.
3.
4.
5.
6.
7.
Markov Process
States
Transition Matrix
Stochastic Matrix
Distribution Matrix
Distribution Matrix for n
Interpretation of the Entries of An
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
3 of 60
Markov Process
Suppose that we perform, one after the other, a
sequence of experiments that have the same set
of outcomes. If the probabilities of the various
outcomes of the current experiment depend (at
most) on the outcome of the preceding
experiment, then we call the sequence a Markov
process.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
4 of 60
Example Markov Process
A particular utility stock is very stable and, in the
short run, the probability that it increases or
decreases in price depends only on the result of
the preceding day's trading. The price of the
stock is observed at 4 P.M. each day and is
recorded as "increased," "decreased," or
"unchanged." The sequence of observations
forms a Markov process.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
5 of 60
States
The experiments of a Markov process are
performed at regular time intervals and have the
same set of outcomes. These outcomes are called
states, and the outcome of the current experiment
is referred to as the current state of the process.
The states are represented as column matrices.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
6 of 60
Transition Matrix
The transition matrix records all data about
transitions from one state to the other. The form of
a general transition matrix is
.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
7 of 60
Stochastic Matrix
A stochastic matrix is any square matrix that
satisfies the following two properties:
1. All entries are greater than or equal to 0;
2. The sum of the entries in each column is 1.
All transition matrices are stochastic matrices.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
8 of 60
Example Transition Matrix
For the utility stock of the previous example, if
the stock increases one day, the probability that
on the next day it increases is .3, remains
unchanged .2 and decreases .5. If the stock is
unchanged one day, the probability that on the
next day it increases is .6, remains unchanged .1,
and decreases .3. If the stock decreases one day,
the probability that it increases the next day is .3,
is unchanged .4, decreases .3. Find the transition
matrix.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
9 of 60
Example Transition Matrix (2)
The Markov process has three states:
"increases," "unchanged," and "decreases."
The transitions from the first state ("increases")
to the other states are
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
10 of 60
Example Transition Matrix (3)
The transitions from the other two states are
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
11 of 60
Example Transition Matrix (4)
Putting this information into a single matrix so
that each column of the matrix records the
information about transitions from one particular
state is the transition matrix.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
12 of 60
Distribution Matrix
The matrix that represents a particular state is
called a distribution matrix.
Whenever a Markov process applies to a group
with members in r possible states, a distribution
matrix for n is a column matrix whose entries give
the percentages of members in each of the r states
after n time periods.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
13 of 60
Distribution Matrix for n
Let A be the transition matrix for a Markov
process with initial distribution matrix   ,
 0
then the distribution matrix after n time periods
is given by
 
n  
A     .
 0  n
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
14 of 60
Example Distribution Matrix for n
Census studies from the 1960s reveal that in the
US 80% of the daughters of working women also
work and that 30% of daughters of nonworking
women work. Assume that this trend remains
unchanged from one generation to the next. If
40% of women worked in 1960, determine the
percentage of working women in each of the next
two generations.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
15 of 60
Example Distribution Matrix for n (2)
There are two states, "work" and "don't work."
The first column of the transition matrix
corresponds to transitions from "work".
The probability that a daughter from this state
"works" is .8 and "doesn't work" is 1 - .8 = .2.
Similarly, the daughter from the "don't work"
state "works" with probability .3 and "doesn't
work" with probability .7.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
16 of 60
Example Distribution Matrix for n (3)
The transition matrix is
.
The initial distribution is
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
.4 
.6  .
 0
17 of 60
Example Distribution Matrix for n (4)
.8 .3 .4  .5
  .
In one generation, 



.2 .7  .6  0 .51
So 50% women work and 50% don't work.
For the second generation,
2
.8 .3 .4  .70 .45 .4 .55
.2 .7  .6   .30 .55 .6  .45 .

  0 
  0  2
So 55% women work and 45% don't work.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
18 of 60
Interpretation of the Entries of An
The entry in the ith row and jth column of the
matrix An is the probability of the transition from
state j to state i after n periods.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
19 of 60
Example Interpretation of the Entries
2
Interpret
.8 .3 .70 .45
.2 .7   .30 .55

 

from the last example.
If a woman works, the probability that her
granddaughter will work is .7 and not work is .3.
If a woman does not work, the probability that
her granddaughter will work is .45 and not work
is .55.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
20 of 60
Summary Section 8.1 - Part 1
 A Markov process is a sequence of
experiments performed at regular time intervals
involving states. As a result of each experiment,
transitions between states occur with
probabilities given by a matrix called the
transition matrix. The ijth entry in the transition
matrix is the conditional probability
Pr(moving to state i|in state j).
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
21 of 60
Summary Section 8.1 - Part 2
 A stochastic matrix is a square matrix for
which every entry is greater than or equal to 0
and the sum of the entries in each column is 1.
Every transition matrix is a stochastic matrix.
The nth distribution matrix gives the percentage
of members in each state after n time periods.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
22 of 60
Summary Section 8.1 - Part 3
 An is obtained by multiplying together n
copies of A. Its ijth entry is the conditional
probability Pr(moving to state i after n time
periods | in state j). Also, An times the initial
distribution matrix gives the nth distribution
matrix.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
23 of 60
8.2 Regular Stochastic Matrices
1. Regular Stochastic Matrix
2. Stable Matrix and Distribution
3. Properties of Regular Stochastic Matrix
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
24 of 60
Regular Stochastic Matrix
A stochastic matrix is said to be regular if some
power has all positive entries.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
25 of 60
Example Regular Stochastic Matrix
Which of the following stochastic matrices are
regular?
.6 .2
0 .5
0 1 
a) 
b) 
c) 



.4 .8
1 .5
1 0 
a) All entries are positive so the matrix is
regular.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
26 of 60
Example Regular Stochastic Matrix (2)
2
0 .5 .5 .25
b) 



1
.5
.5
.75

 

All entries of the square are positive so the
matrix is regular.
2
3
 0 1  1 0 
0 1  0 1 
c) 

and 





1 0   0 1 
1 0  1 0 
All powers will be one of the above two
matrices so the original matrix is not regular.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
27 of 60
Stable Matrix and Distribution
If a stochastic matrix A has the properties that
1. as n gets large, An approaches a fixed matrix,
and
2. any initial distribution approaches a fixed
distribution for large n, then
the fixed matrix is called the stable matrix of A
and the fixed distribution is called the stable
distribution of A.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
28 of 60
Example Stable Matrix and Distribution
In Jordan, 25% of the women currently work.
The effect of maternal influence of mothers on
their daughters is given by the matrix
.6 .2 
A
.

.4 .8
Find the stable matrix and the stable distribution
of A.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
29 of 60
Example Stable Matrix and Distribution (2)
It appears that the powers are approaching
1
 3
2
 3
1 
3
which is the stable matrix.
2 
3
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
30 of 60
Example Stable Matrix and Distribution (3)
1
For the initial distribution,  3
2
 3
1 
1 

3  .25  3 

.


2  .75  2 
3
 3
However, for any initial distribution,
1
 3
2
 3
1 
1 

3   p   3  which is the stable



2  1  p   2 
3
 3
distribution.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
31 of 60
Properties of Regular Stochastic Matrix
Let A be a regular stochastic matrix.
1. The powers An approach a certain matrix as n
gets large. This limiting matrix is called the
stable matrix of A.
2. For any initial distribution [ ]0, An[ ]0
approaches a certain distribution. This limiting
distribution is called the stable distribution of A.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
32 of 60
Properties of Regular Stochastic Matrix (2)
3. All columns of the stable matrix are the same;
they equal the stable distribution.
4. The stable distribution X = [ ] can be
determined by solving the system of linear
equations
 sum of the entries of X  1

AX  X .

Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
33 of 60
Example Properties Regular Matrices
Use the properties of a regular stochastic matrix
to find the stable matrix and stable distribution of
.6 .2 
A
.

.4 .8
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
34 of 60
Example Properties Regular Matrices (2)
Solve
x  y 1
.6 .2   x   x 
.4 .8  y    y  .

   
This gives
x
y 1
.4 x  .2 y  0
.4 x  .2 y  0.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
35 of 60
Example Properties Regular Matrices (3)
The last equation is (-1) times the second
equation so we can solve just the first two
equations.
1
1
1
1
1
1
1

  2.41 
 .6 2

 .4 .2 0   0 .6 .4  




1 1 1  12 1 0 13 






2
0 1 2 
0
1


3
3

Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
36 of 60
Example Properties Regular Matrices (4)
Therefore, the stable distribution and stable
matrix, respectively, are
1 
1
 3  and  3
2 
2
 3
 3
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
1 
3
.
2 
3
37 of 60
Summary Section 8.2 - Part 1
 A stochastic matrix is called regular if some
power of the matrix has only positive entries.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
38 of 60
Summary Section 8.2 - Part 2
 If A is a regular stochastic matrix, as n gets
large the powers of the matrix, An, approach a
certain matrix called the stable matrix of A and
the distribution matrices approach a certain
column matrix called the stable distribution.
Each column of the stable matrix holds the stable
distribution. The stable distribution can be found
by solving AX = X, where the sum of the entries
in X is equal to 1.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
39 of 60
8.3 Absorbing Stochastic Matrices
1.
2.
3.
4.
5.
6.
Absorbing State
Absorbing Stochastic Matrix
Arranging States in an Absorbing Matrix
Properties of Absorbing Matrix
Stable Matrix of Absorbing Matrix
Fundamental Matrix
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
40 of 60
Absorbing State
A state for which all objects that start in that state,
stay in that state is called an absorbing state. That
is, an absorbing state is a state that always leads
back to itself.
A state is absorbing if
1. the corresponding column has a single 1 and
the remaining entries are 0, and
2. the 1 must be located on the main diagonal of
the matrix.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
41 of 60
Absorbing Stochastic Matrix
An absorbing stochastic matrix is a stochastic
matrix in which
1. there is at least one absorbing state, and
2. from any state it is possible to get to at least
one absorbing state, either directly or through one
or more intermediate states.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
42 of 60
Example Absorbing Stochastic Matrix
For the given stochastic matrix determine the
absorbing states, if any, and whether the matrix
is an absorbing stochastic matrix.
1
0

0

0
0 .3 0 
1 .1 1 
0 .5 0 

0 .1 0 
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
43 of 60
Example Absorbing Stochastic Matrix (2)
States 1 and 2 are absorbing because the two
columns have a single 1 and that 1 appears on
the diagonal. State 4 is not absorbing because,
although it has a single 1, it is not on the
diagonal. State 3 does not have a single 1.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
44 of 60
Example Absorbing Stochastic Matrix (3)
Objects in state 3 can lead to both state 1 with
probability .3 and state 2 with probability .1.
Objects in state 4 will lead to state 2 with
probability 1.
Therefore, the matrix is an absorbing stochastic
matrix.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
45 of 60
Arranging States in an Absorbing Matrix
When considering an absorbing stochastic
matrix, we will always arrange the states so that
the absorbing states come first, then the
nonabsorbing states.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
46 of 60
Example Arranging States
Identify R and S for the given absorbing
stochastic matrix.
1
0

0

0
0 .3 0 
1 .1 1 
0 .5 0 

0 .1 0 
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
47 of 60
Example Arranging States
The absorbing states are already written first.
Identity matrix
1
0

0

0
0 .3 0 

1 .1 1 
0 .5 0 

0 .1 0 
Zero matrix
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
.3 0 
S

.1 1 
.5 0 
R

.1 0 
48 of 60
Stable Matrix of Absorbing Matrix
Let an absorbing stochastic matrix be partitioned
as
I S 


A
,

0 R 
I S I  R
A
R
0
1



then the stable
matrix of A is


I S I  R
A
R
0
1



 I S  I  R 1 

,
R
 0

where I is the same dimension as R in (I - R)-1.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
49 of 60
Example Stable Matrix
Find the stable matrix of
1
0

0

0
0 .3 0 

1 .1 1 
.
0 .5 0 

0 .1 0 
.3 0 
S

.1 1 
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
.5 0 
R

.1 0 
50 of 60
Example Stable Matrix (2)
1 0 .5 0  .5 0
I R





0 1  .1 0  .1 1 
 I  R
1
S  I  R
1  1 0   2 0
 



.5 .1 .5 .2 1 
1
.3 0  2 0 .6 0






.1 1  .2 1  .4 1 
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
51 of 60
Example Stable Matrix (3)
The stable matrix is
1
0

0

0
0 .6 0 
1 .4 1 
.
0 0 0

0 0 0
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
52 of 60
Example Stable Matrix (4)
There is no stable distribution matrix as the long
term trend depends upon the initial matrix.
For example:
1
0

0

0
0 .6 0  .25 .4
1






1 .4 1  .25 .6
0


and
0
0 0 0  .25  0 
   

0 0 0  .25  0 
0
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
0 .6 0  .5 .8





1 .4 1   0  .2

0 0 0  .5  0 
   
0 0 0  0   0 
53 of 60
Fundamental Matrix
The matrix (I - R)-1 used to compute the stable
matrix is called the fundamental matrix and is
denoted by the letter F.
The ijth entry of F is the expected number of
times the process will be in nonabsorbing state i
if it starts in nonabsorbing state j.
The sum of the entries of the jth column of F is
the expected number of steps before absorption
when the process begins in nonabsorbing state j.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
54 of 60
Example Fundamental Matrix
Interpret the fundamental matrix from the
previous example.
 2 0
1
F   I  R  

.2
1


Note: States 3 and 4 were the nonabsorbing
states.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
55 of 60
Example Fundamental Matrix
 2 0
F   I  R  

.2 1 
If an object starts in state 3, it is expected to
be in state 3 for 2 steps and in state 4 for .2
steps and reach an absorbing state (1 or 2) in
2 + .2 = 2.2 steps.
If the object starts in state 4, it will never
reach state 3 and expects to be in state 4 for 1
step and reach an absorbing state in 1 step.
1
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
56 of 60
Summary Section 8.3 - Part 1
 If the probability of moving from a state to
itself is 1, we call that state an absorbing state.
An absorbing stochastic matrix is a stochastic
matrix with at least one absorbing state and in
which from any state it is possible to eventually
get to an absorbing state.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
57 of 60
Summary Section 8.3 - Part 2
 In an absorbing process, the transition matrix should
be arranged so that absorbing states are listed before
nonabsorbing states. The transition matrix will have the
form
I S 
A

0
R


where I is an identity matrix, 0 denotes a matrix of zeros,
and S and R represent the transitions from nonabsorbing
to absorbing states and from nonabsorbing to
nonabsorbing states, respectively.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
58 of 60
Summary Section 8.3 - Part 3
 The stable matrix of the absorbing matrix in
the proceeding transition matrix is
 I S  I  R 1 

.
0
 0

Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
59 of 60
Summary Section 8.3 - Part 4
 The fundamental matrix of the absorbing
matrix in the proceeding transition matrix is the
matrix (I- R)-1. When its columns and rows are
labeled with the nonabsorbing states, its ijth entry
is the expected number of times the process will
be in nonabsorbing state i given that it started in
nonabsorbing state j. The sum of the entries in
the jth column is the expected number of steps
before absorption when the process begins in
state j.
Goldstein/Schnieder/Lay: Finite Math & Its Applications, 9e
60 of 60