Absorbing Markov
Chains — What are
the chances of getting
stuck somewhere?
COPYRIGHT © 2006 by LAVON B. PAGE
Definition
An absorbing state is a state from which
it is impossible to leave.
COPYRIGHT © 2006 by LAVON B. PAGE
An absorbing state is a state from which
it is impossible to leave.
Example:
! 1
0
0 $
#
&
#1 / 2 0 1 / 2&
#
&
#"1 / 3 1 / 2 1 / 6&%
In this example, state #1 is an
absorbing state.
COPYRIGHT © 2006 by LAVON B. PAGE
F = Freshmen
So = Sophomore
Jr = Junior
Sr = Senior
G = Graduated
D = Dropped out
Example: Tracking students
through a university.
COPYRIGHT © 2006 by LAVON B. PAGE
F = Freshmen
So = Sophomore
Jr = Junior
Sr = Senior
G = Graduated
D = Dropped out
Example: Tracking students
through a university.
.1
.8
F
.1
1
G
1
.9
.1
D
.05
Sr
So
.05 .85
.05
J
.9
.05
.05
COPYRIGHT © 2006 by LAVON B. PAGE
Here’s the transition matrix:
G D F
G !# 1 0 0
#
D #0 1 0
F ## 0 .1 .1
So## 0 .05 0
J # 0 .05 0
#
Sr " .9 .05 0
So J Sr
0 0
0 $&
&
0 0
0&
.8 0
0 &&
.1 .85 0 &
&
0 .05 .9 &
&
0 0 .05%
COPYRIGHT © 2006 by LAVON B. PAGE
G D F
G !# 1 0 0
#
D #0 1 0
F ## 0 .1 .1
So## 0 .05 0
J # 0 .05 0
#
Sr " .9 .05 0
So J Sr
0 0
0 $&
&
0 0
0&
.8 0
0 &&
.1 .85 0 &
&
0 .05 .9 &
&
0 0 .05%
Notice the transition matrix splits
into these blocks.
COPYRIGHT © 2006 by LAVON B. PAGE
I
G D F
G !# 1 0 0
#
D #0 1 0
F ## 0 .1 .1
So## 0 .05 0
J # 0 .05 0
#
Sr " .9 .05 0
R
So J Sr
0 0
0 $&
&
0 0
0&
.8 0
0 &&
.1 .85 0 &
&
0 .05 .9 &
&
0 0 .05%
0
Q
COPYRIGHT © 2006 by LAVON B. PAGE
The matrix below is T
G
D
F
3
(approximately).
So
J
Sr
$&
0
0
0
0
0
G !# 1
&
D ## 0
&
1
0
0
0
0
&
F# 0
.193
.001
.024
.170
.612
&
#
So #.689 .142
0
.001 .0149
.153 &&
#
J #.891 .102
0
0
.000125 .00675 &
&
Sr #".947 .0526 0
0
0
.000125 %
COPYRIGHT © 2006 by LAVON B. PAGE
The matrix below is T
G
D
F
3
(approximately).
So
J
Sr
!
$&
0
0
0
0
0
G ## 1
&
D# 0
&
1
0
0
0
0
#
&
F# 0
.193 .001 .024
.170
.612 &
So ##.689 .142
0
.001 .0149
.153 &&
J #.891 .102
0
0
.000125 .00675 &
#"
&
Sr .947 .0526 0
0
0
.000125 %
Probability that a freshman will be
a senior after 3 years in school.
COPYRIGHT © 2006 by LAVON B. PAGE
The matrix below is T
G
D
F
3
(approximately).
So
J
Sr
!
$&
0
0
0
0
0
G ## 1
&
D# 0
&
1
0
0
0
0
#
&
F# 0
.193 .001 .024
.170
.612 &
So ##.689 .142
0
.001 .0149
.153 &&
J #.891 .102
0
0
.000125 .00675 &
#"
&
Sr .947 .0526 0
0
0
.000125 %
Of special interest is this part of the matrix,
because it indicates the probabilities of being
in each of the absorbing states after 3 years.
COPYRIGHT © 2006 by LAVON B. PAGE
Question
Given the initial transition matrix, how can we
find the long term probabilities for entering each
of the absorbing states?
COPYRIGHT © 2006 by LAVON B. PAGE
Question
Given the initial transition matrix, how can we
find the long term probabilities for entering each
of the absorbing states?
Fortunately there’s a recipe for doing
this.
COPYRIGHT © 2006 by LAVON B. PAGE
I
R
G D F
G #! 1 0 0
#
D #0 1 0
F ## 0 .1 .1
So# 0 .05 0
#
J # 0 .05 0
#
Sr " .9 .05 0
So J Sr
0 0
0 &$
&
0 0
0&
.8 0
0 &&
.1 .85 0 &
&
0 .05 .9 &
&
0 0 .05%
0
Q
COPYRIGHT © 2006 by LAVON B. PAGE
I
R
G D F
G #! 1 0 0
#
D #0 1 0
F ## 0 .1 .1
So# 0 .05 0
#
J # 0 .05 0
#
Sr " .9 .05 0
So J Sr
0 0
0 &$
&
0 0
0&
.8 0
0 &&
.1 .85 0 &
&
0 .05 .9 &
&
0 0 .05%
0
Q
1) Find I–Q (where I is same size as Q)
COPYRIGHT © 2006 by LAVON B. PAGE
I
R
G D F
G #! 1 0 0
#
D #0 1 0
F ## 0 .1 .1
So# 0 .05 0
#
J # 0 .05 0
#
Sr " .9 .05 0
So J Sr
0 0
0 &$
&
0 0
0&
.8 0
0 &&
.1 .85 0 &
&
0 .05 .9 &
&
0 0 .05%
0
Q
1) Find I–Q (where I is same size as Q)
2) Find the inverse of this matrix: N = (I–Q)–1
COPYRIGHT © 2006 by LAVON B. PAGE
I
R
G D F
G #! 1 0 0
#
D #0 1 0
F ## 0 .1 .1
So# 0 .05 0
#
J # 0 .05 0
#
Sr " .9 .05 0
So J Sr
0 0
0 &$
&
0 0
0&
.8 0
0 &&
.1 .85 0 &
&
0 .05 .9 &
&
0 0 .05%
0
Q
1) Find I–Q (where I is same size as Q)
2) Find the inverse of this matrix: N = (I–Q)–1
3) Calculate B = NR
COPYRIGHT © 2006 by LAVON B. PAGE
!.9
#
I – Q =# 0
#0
#
#" 0
'.8 0
.9 '.85
0 .95
0
0
0 $
&
0 &
'.9 &
&
.95&%
Carrying out
the recipe
! 0 .1 $
#
&
# 0 .05&
R= #
&
0
.05
#
&
#".9 .05&%
COPYRIGHT © 2006 by LAVON B. PAGE
!1.111 .988 .884 .837 $
#
&
# 0
1.111 .994 .942 &
-1
N = (I – Q) = #
&
0
0
1.053
.997
#
&
#" 0
0
0
1.053&%
!.753
#
#.848
B = NR = #
#.898
#".947
.247$
&
.152&
&
.102&
.053&%
COPYRIGHT © 2006 by LAVON B. PAGE
G
D
F !.753 .247$
#
&
So #.848 .152&
#
&
B = NR =
J #.898 .102&
Sr #".947 .053&%
Label the rows with the non-absorbing states
and the columns with the absorbing states.
COPYRIGHT © 2006 by LAVON B. PAGE
G
D
F !.753 .247$
#
&
So #.848 .152&
#
&
B = NR =
J #.898 .102&
Sr #".947 .053&%
Label the rows with the non-absorbing states
and the columns with the absorbing states.
The numbers in matrix B show the probability
of eventually arriving in each absorbing state
from any of the non-absorbing states.
COPYRIGHT © 2006 by LAVON B. PAGE
G
D
F !.753 .247$
#
&
So #.848 .152&
#
&
B = NR =
J #.898 .102&
Sr #".947 .053&%
For Example: This is the probability that
someone who begins as a freshman will
eventually graduate.
COPYRIGHT © 2006 by LAVON B. PAGE
G
D
F !.753 .247$
#
&
So #.848 .152&
#
&
B = NR =
J #.898 .102&
Sr #".947 .053&%
For Example: This is the probability that
someone who begins as a freshman will
eventually graduate.
And this is the probability that a junior will
drop out before graduating.
COPYRIGHT © 2006 by LAVON B. PAGE
Tennis: Sally and Becky are playing tennis. When deuce
is reached, the player winning the next point has
advantage. On the following point, the player either wins
the game or the game returns to deuce. Suppose that at
deuce, Sally has probability 2/3 of winning the next point
and Becky has 1/3 probability of winning the point. When
Sally has advantage she has probability 3/4 of winning
the next point and when Becky has advantage she has
probability 1/2 of winning the next point. Set this up as a
Markov Chain with states: Sally wins the game, Becky
wins the game, Sally’s advantage, Becky’s advantage,
deuce.
If the game is at deuce, find how long the game is
expected to last and the probability that Becky wins.
If Sally has advantage, what is the probability she
eventually wins the game?
COPYRIGHT © 2006 by LAVON B. PAGE
Tennis
D = deuce
BA = Becky’s advantage
SA = Sally’s advantage
BW = Becky wins
SW = Sally Wins
COPYRIGHT © 2006 by LAVON B. PAGE
Tennis
D = deuce
BA = Becky’s advantage
SA = Sally’s advantage
BW = Becky wins
SW = Sally Wins
1
1/2
BW
BA
1/3
2/3
3/4
D
SA
SW
1/2
1/4
1
COPYRIGHT © 2006 by LAVON B. PAGE
1
1/2
BW
BA
1/3
2/3
D
SA
3/4
1
SW
1/2
1/4
BW SW BA D SA
BW!# 1 0 0 0 0 $&
#
&
SW# 0 1 0 0 0 &
#
&
#
1
1
&
BA #
0 0
0&
2
#2
&
#
1
2 &&
#
D #0 0
0
3
3&
#
&
#
&
SA # 0 3 0 1 0 &
"
%
4
4
COPYRIGHT © 2006 by LAVON B. PAGE
#!
#
#
#
Q =#
#
#
#
#"
1
2
0
1
3
0
1
4
0
!#
#
#
#
#
#
#
#
#
"
&$
0 &
&
2 &&
3 &&
&
0 &%
1
-1
2
-1
3
1
0
-1
4
I-Q
$&
0 &
&
-2 &&
3 &&
&
1 &%
!#
#
#
R= #
#
#
#
#"
!#
#
#
#
#
#
#
#
#
"
5
4
1
2
1
8
1
2
0
0
0
3
4
0
3
4
3
2
3
8
$&
&
&
&
&
&
&
&%
1
2
$&
&
&
&
1 &&
&
5 &&
4 %
-1
(I - Q) = N
COPYRIGHT © 2006 by LAVON B. PAGE
!#
#
#
#
B = NR =#
#
#
#
#
"
!#
#
#
#
= ##
#
#
#"
5
4
1
2
1
8
3
4
3
2
3
8
5
8
1
4
1
16
3 &$
&
8 &
3 &&
4 &&
15 &&
16 %
1
2
$& !#
&#
&#
&#
1 && #
& ##
5 && "#
4 %
1
2
0
0
0
0
3
4
$&
&
&
&
&
&
&
&%
COPYRIGHT © 2006 by LAVON B. PAGE
BA
BA !## 5
#
#
D ##
#
#
SA #
"
4
1
2
1
8
D
3
4
3
2
3
8
SA
1
2
$&
&
&
&
1 &&
&
5 &&
4 %
N = (I - Q)
-1
!#
BA #
#
#
D ##
#
#
SA #"
BW SW
5
8
1
4
1
16
3 $&
&
8 &
3 &&
4 &&
15 &&
16 %
B = NR
COPYRIGHT © 2006 by LAVON B. PAGE
BA
BA !## 5
#
#
D ##
#
#
SA #
"
4
1
2
1
8
D
3
4
3
2
3
8
SA
1
2
$&
&
&
&
1 &&
&
5 &&
4 %
N = (I - Q)
-1
!#
BA #
#
#
D ##
#
#
SA #"
BW SW
5
8
1
4
1
16
3 $&
&
8 &
3 &&
4 &&
15 &&
16 %
B = NR
If the game is at deuce,the probability
that Becky wins the game is 1/4.
COPYRIGHT © 2006 by LAVON B. PAGE
BA
BA !## 5
#
#
D ##
#
#
SA #
"
4
1
2
1
8
D
3
4
3
2
3
8
SA
1
2
$&
&
&
&
1 &&
&
5 &&
4 %
N = (I - Q)
-1
!#
BA #
#
#
D ##
#
#
SA #"
BW SW
5
8
1
4
1
16
3 $&
&
8 &
3 &&
4 &&
15 &&
16 %
B = NR
If the game is at deuce, the probability
that Becky wins the game is 1/4.
If it’s Becky’s advantage,the probability
that Sally wins the game is 3/8.
COPYRIGHT © 2006 by LAVON B. PAGE
BA
BA !## 5
#
#
D ##
#
#
SA #
"
4
1
2
1
8
D
3
4
3
2
3
8
SA
1
2
$&
&
&
&
1 &&
&
5 &&
4 %
N = (I - Q)
-1
!#
BA #
#
#
D ##
#
#
SA #"
BW SW
5
8
1
4
1
16
3 $&
&
8 &
3 &&
4 &&
15 &&
16 %
B = NR
If the game is at advantage Becky, the
expected number of times the game will be
tied at Deuce before the game ends is 3/4.
COPYRIGHT © 2006 by LAVON B. PAGE
BA
BA !## 5
#
#
D ##
#
#
SA #
"
4
1
2
1
8
D
3
4
3
2
3
8
SA
1
2
$&
&
&
&
1 &&
&
5 &&
4 %
N = (I - Q)
-1
!#
BA #
#
#
D ##
#
#
SA #"
BW SW
5
8
1
4
1
16
3 $&
&
8 &
3 &&
4 &&
15 &&
16 %
B = NR
If it’s Sally’s advantage, the expected
number of times it will be Sally’s
advantage before the game ends is 5/4.
COPYRIGHT © 2006 by LAVON B. PAGE
BA
BA !## 5
#
#
D ##
#
#
SA #
"
4
1
2
1
8
D
3
4
3
2
3
8
SA
1
2
$&
&
&
&
1 &&
&
5 &&
4 %
N = (I - Q)
-1
!#
BA #
#
#
D ##
#
#
SA #"
BW SW
5
8
1
4
1
16
3 $&
&
8 &
3 &&
4 &&
15 &&
16 %
B = NR
If the game is tied at deuce, the expected
number of points to be played before the
game ends is the sum of this row: 3.
COPYRIGHT © 2006 by LAVON B. PAGE
Finding the long term probabilities
when tracking students
I
G D F
G !# 1 0 0
#
D #0 1 0
F ## 0 .1 .1
So## 0 .05 0
J # 0 .05 0
#
Sr " .9 .05 0
R
So J Sr
0 0
0 $&
&
0 0
0&
.8 0
0 &&
.1 .85 0 &&
0 .05 .9 &
&
0 0 .05%
Q
0
COPYRIGHT © 2006 by LAVON B. PAGE
Finding the long term probabilities
when tracking students
!1.111 .988 .884 .837 $
#
&
# 0
1.111 .994 .942 &
N = (I – Q)-1 = #
&
0
0
1.053
.997
#
&
#" 0
0
0
1.053&%
COPYRIGHT © 2006 by LAVON B. PAGE
F
So
J
Sr
F !1.111 .988 .884 .837 $
#
&
So # 0
1.111 .994 .942 &
&
J ## 0
0
1.053 .997 &
Sr #" 0
0
0
1.053&%
COPYRIGHT © 2006 by LAVON B. PAGE
F
So
J
Sr
F !1.111 .988 .884 .837 $
#
&
So # 0
1.111 .994 .942 &
&
J ## 0
0
1.053 .997 &
Sr #" 0
0
0
1.053&%
3.819
3.047
2.05
1.053
COPYRIGHT © 2006 by LAVON B. PAGE