Quasi-Stationary Distributions for
Continuous-Time Markov Chains
David Sirl
[email protected]
http://www.maths.uq.edu.au/˜dsirl/
AUSTRALIAN RESEARCH COUNCIL
Centre of Excellence for Mathematics
and Statistics of Complex Systems
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 1
Quasi-Stationary Behaviour
Sample Path of B−D Chemical Process with α=100, k1=1, k−1=1, k2=1
250
200
State
150
100
50
0
0
0.05
0.1
0.15
Time
0.2
0.25
0.3
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 2
The Chemical Reaction
2X
A+X
k1
B
k−1
k2
X
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 3
The Chemical Reaction
2X
A+X
k1
B
k−1
k2
X
Model the number of molecules of X with a CTMC — a
birth-death process on S = {0} ∪ C , where zero is
absorbing and C is an irreducible transient class.
The system can be either closed or open with respect
to A & B. C = {1, 2, . . . , N } or {1, 2, . . .}, respectively.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 3
An Outline:
I will mostly talk about establishing the existence of QSDs.
We will:
Review the theory of quasi-stationarity, looking at the
chemical reaction example.
See that explicit evaluation of QSDs is not usually
possible.
Conjecture an extremely general, somewhat unlikely
existence result.
Briefly introduce some other topics of research in the
area.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 4
Recall . . .
A time-homogeneous CTMC (X(t), t ≥ 0) taking values
in a countable set S (Z+ ) is completely described by its
transition function P (t) = (pij (t), i, j ∈ S, t ≥ 0).
In practice we know only the transition rates
(p0ij (0+) = qij , i, j ∈ S).
If we know P , we can in principle answer any question
about the behaviour of the chain. The challenge is to try
and answer these questions in terms of Q.
We will assume that the process is absorbed with
probability one, and is therefore regular (non-explosive).
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 5
Recall . . .
A Birth-Death Process is a CTMC with state space
S = {0, 1, 2, . . .} such that if the chain is in state i, transitions
can only be made to state i − 1 or i + 1.
The q-matrix has non-zero entries
qi,i+1 = λi
qi,i−1 = µi
(birth rates),
(death rates), and
qii = −(λi + µi ),
(put µ0 = 0) where λi , µi > 0 ∀i ∈ C , and λ0 ≥ 0.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 6
Definitions
A distribution a = (ai , i ∈ C) is a QSD over C if when
the initial
Pdistribution is a, the state probabilities
pj (t) = i∈C ai pij (t) conditioned on non-absorption are
time-invariant (and given by a):
pj (t)
= aj ,
1 − p0 (t)
j ∈ C.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 7
Definitions
A distribution a = (ai , i ∈ C) is a QSD over C if when
the initial
Pdistribution is a, the state probabilities
pj (t) = i∈C ai pij (t) conditioned on non-absorption are
time-invariant (and given by a):
pj (t)
= aj ,
1 − p0 (t)
j ∈ C.
A distribution b = (bi , i ∈ C) is a LCD over C if bj gives
the limiting probability of the process being in state j ,
conditional on non-absorption, independently of the
starting state i:
pij (t)
lim
= bj ,
t→∞ 1 − pi0 (t)
i, j ∈ C.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 7
Definitions
A µ-invariant measure (over C ) for P is a collection of
numbers m = (mi , i ∈ C) which, for some µ > 0, satisfy
X
mi pij (t) = e−µt mj ,
j ∈ C, t ≥ 0.
i∈C
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 8
Definitions
A µ-invariant measure (over C ) for P is a collection of
numbers m = (mi , i ∈ C) which, for some µ > 0, satisfy
X
mi pij (t) = e−µt mj ,
j ∈ C, t ≥ 0.
i∈C
A µ-invariant measure (over C ) for Q is a collection of
numbers m = (mi , i ∈ C) which, for some µ > 0, satisfy
X
mi qij = −µmj ,
j ∈ C.
i∈C
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 8
Finite State Space
Easy because of spectral decomposition (of P ) and
Perron-Frobenius theory.
The LCD and unique QSD is given by the probability
measure m such that
mPC (t) = e−νt m.
This is equivalent to
mQC = −νm,
where −ν is the eigenvalue with maximal real part (it is
real and negative).
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 9
The Decay Parameter
The quantity
− log(pij (t))
λC := lim
t→∞
t
exists and is independent of i, j ∈ C .
Called the decay parameter because
pij (t) ≤ Mij e−λC t ,
0 < Mij < ∞.
Can show that for a µ-invariant measure for P over C to
exist, it is necessary that (0 <)µ ≤ λC .
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 10
Infinite State Space
P is now an infinite matrix, so Perron-Frobenius theory
no longer applies.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 11
Infinite State Space
P is now an infinite matrix, so Perron-Frobenius theory
no longer applies.
QSDs (if they exist) are given by probability measures
m such that
mPC (t) = e−νt m.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 11
Infinite State Space
P is now an infinite matrix, so Perron-Frobenius theory
no longer applies.
QSDs (if they exist) are given by probability measures
m such that
mPC (t) = e−νt m.
The LCD (if it exists) is given by m such that
mPC (t) = e−λC t m.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 11
Infinite State Space
P is now an infinite matrix, so Perron-Frobenius theory
no longer applies.
QSDs (if they exist) are given by probability measures
m such that
mPC (t) = e−νt m.
The LCD (if it exists) is given by m such that
mPC (t) = e−λC t m.
These expressions involve P and λC , which are not
known and impossible (or at best very difficult) to find
analytically — we need conditions in terms of the
q-matrix.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 11
Infinite State Space
Theorem: A µ-invariant measure for Q is also µ-invariant
for P (and is therefore a QSD) iff
X
yi qij = κyj ,
0 ≤ yi ≤ mi , j ∈ C,
i∈C
has only the trivial solution for some (all) κ > −µ.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 12
Infinite State Space
Theorem: A µ-invariant measure for Q is also µ-invariant
for P (and is therefore a QSD) iff
X
yi qij = κyj ,
0 ≤ yi ≤ mi , j ∈ C,
i∈C
has only the trivial solution for some (all) κ > −µ.
This condition does not depend explicitly on P or λC .
Does depend on having a particular µ, m to check.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 12
Infinite State Space
Theorem: If m is a µ-invariant probability measure for Q,
then
X
µ=
mi qi0
i∈C
is neccesary and sufficient for m to be a QSD.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 13
Infinite State Space
Theorem: If m is a µ-invariant probability measure for Q,
then
X
µ=
mi qi0
i∈C
is neccesary and sufficient for m to be a QSD.
This allows us to find all µ-invariant probability
measures for Q.
We can then check which of these are QSDs using the
previous result.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 13
Infinite State Space
In order to find these µ-invariant measures for Q we must
solve the system
X
mi qij = −µmj , j ∈ C.
i∈C
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 14
Infinite State Space
In order to find these µ-invariant measures for Q we must
solve the system
!
X
X
mi qij = −
mi qi0 mj , j ∈ C.
i∈C
i∈C
We can eliminate µ explicitly from the system we need to
solve, however this renders the system non-linear in m.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 14
Infinite State Space
In order to find these µ-invariant measures for Q we must
solve the system
!
X
X
mi qij = −
mi qi0 mj , j ∈ C.
i∈C
i∈C
We can eliminate µ explicitly from the system we need to
solve, however this renders the system non-linear in m.
Finding explicit expressions for QSDs is rarely possible.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 14
Infinite State Space
Once we find a µ-invariant probability measure m for Q, we
need to check if
X
yi qij = κyj ,
0 ≤ yi ≤ mi , j ∈ C,
i∈C
has only the trivial solution for some (all) κ > −µ.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 15
Infinite State Space
Once we find a µ-invariant probability measure m for Q, we
need to check if
X
yi qij = κyj ,
0 ≤ yi ≤ mi , j ∈ C,
i∈C
has only the trivial solution for some (all) κ > −µ.
Suppose QC has no summable left eigenvectors for
some (all) eigenvalues κ > 0.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 15
Infinite State Space
Once we find a µ-invariant probability measure m for Q, we
need to check if
X
yi qij = κyj ,
0 ≤ yi ≤ mi , j ∈ C,
i∈C
has only the trivial solution for some (all) κ > −µ.
Suppose QC has no summable left eigenvectors for
some (all) eigenvalues κ > 0.
Then it cannot possibly have any left eigenvectors
bounded above by m.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 15
Infinite State Space
Theorem: If the equations
X
yi qij = κyj ,
yi ≥ 0, j ∈ C,
i∈C
X
yi < ∞
i∈C
have only the trivial solution for some (all) κ > 0, then all
µ-invariant probability measures for Q are also µ-invariant
for P and are therefore QSDs.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 16
Infinite State Space
Theorem: If the equations
X
yi qij = κyj ,
yi ≥ 0, j ∈ C,
i∈C
X
yi < ∞
i∈C
have only the trivial solution for some (all) κ > 0, then all
µ-invariant probability measures for Q are also µ-invariant
for P and are therefore QSDs.
Call this condition the “Reuter FE Condition”
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 16
Infinite State Space
Theorem: If the equations
X
yi qij = κyj ,
yi ≥ 0, j ∈ C,
i∈C
X
yi < ∞
i∈C
have only the trivial solution for some (all) κ > 0, then all
µ-invariant probability measures for Q are also µ-invariant
for P and are therefore QSDs.
Call this condition the “Reuter FE Condition”
If this condition holds, all we have to do is find a
µ-invariant measure for Q and this is a QSD.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 16
Infinite State Space
Theorem: If the equations
X
yi qij = κyj ,
yi ≥ 0, j ∈ C,
i∈C
X
yi < ∞
i∈C
have only the trivial solution for some (all) κ > 0, then all
µ-invariant probability measures for Q are also µ-invariant
for P and are therefore QSDs.
Call this condition the “Reuter FE Condition”
If this condition holds, all we have to do is find a
µ-invariant measure for Q and this is a QSD.
However we want it to be λC -invariant, so that we have
the LCD; but we don’t know λC .
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 16
Birth-Death Processes
Theorem: For a B-D process with which is absorbed with
probability one, suppose that the initial distribution has
compact support. Then
If D < ∞ then there is a unique QSD which is the LCD.
If D = ∞ then either
λC = 0 and there are no QSDs, or
λC > 0 and there is a one-parameter family of QSDs,
one of which is the LCD.
Here
D=
∞
X
n=1
1
µn πn
∞
X
m=n
πm ,
λ1 · · · λn−1
πn =
.
µ2 · · · µn
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 17
The Chemical Reaction
2X
A+X
k1
B
k−1
k2
X
The birth and death rates are, respectively,
λi = αk1 i,
and
i(i − 1)
µi = k2 i + k−1
.
2
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 18
The Chemical Reaction
One can show that this process is absorbed with
probability one (and is therefore regular).
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 19
The Chemical Reaction
One can show that this process is absorbed with
probability one (and is therefore regular).
We can also show that
D=
∞
X
nΓ(n + r)
k−1
n−1
[nk
+
n(n
−
1)
](αs)
2
n=1
2
∞
X
(αs)m−1
,
mΓ(m + r)
m=n
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 19
The Chemical Reaction
One can show that this process is absorbed with
probability one (and is therefore regular).
We can also show that
D=
∞
X
nΓ(n + r)
k−1
n−1
[nk
+
n(n
−
1)
](αs)
2
n=1
2
∞
X
(αs)m−1
,
mΓ(m + r)
m=n
and that this is finite.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 19
The Chemical Reaction
One can show that this process is absorbed with
probability one (and is therefore regular).
We can also show that
D=
∞
X
nΓ(n + r)
k−1
n−1
[nk
+
n(n
−
1)
](αs)
2
n=1
2
∞
X
(αs)m−1
,
mΓ(m + r)
m=n
and that this is finite.
So there is a unique quasistationary distribution, which
is limiting conditional.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 19
A Connection
For a Birth-Death process, the Reuter FE conditions
hold iff D = ∞.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 20
A Connection
For a Birth-Death process, the Reuter FE conditions
hold iff D = ∞.
So, let’s replace
D diverges (converges)
in van Doorns’ result with
the Reuter FE condition holds (fails).
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 20
Infinite State Space
Conjecture: Suppose a process is absorbed with
probability one, and that the initial distribution has compact
support. Then
If the Reuter FE conditions fail then there is only one
finite µ-invariant measure; it is in fact λC -invariant and is
therefore the LCD.
If the Reuter FE conditions hold, either
λC = 0 and there are no QSDs (and no LCD), or
λC > 0 and there is a one-parameter family of finite
µ-invariant measures (QSDs), 0 < µ ≤ λC , of which
the λC -invariant measure is the LCD.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 21
Approximating QSDs
Closed form expressions for QSDs are very rarely
available — we only have expressions of the form
mQC = −µ(m)m.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 22
Approximating QSDs
Closed form expressions for QSDs are very rarely
available — we only have expressions of the form
mQC = −µ(m)m.
We therefore need approximation methods.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 22
Approximating QSDs
Closed form expressions for QSDs are very rarely
available — we only have expressions of the form
mQC = −µ(m)m.
We therefore need approximation methods.
The most common method is truncation: solve
n
X
(n)
mi qij
= −λ
(n)
(n)
mj ,
j ∈ C (n)
i∈C (n)
for m(n) , where λ(n) is the P-F maximal e-value of QC (n) ,
for succesively larger n. Here {C (n) } is a collection of
increasing, finite subsets of C .
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 22
Approximating QSDs
Closed form expressions for QSDs are very rarely
available — we only have expressions of the form
mQC = −µ(m)m.
We therefore need approximation methods.
The most common method is truncation: solve
n
X
(n)
mi qij
= −λ
(n)
(n)
mj ,
j = 1, . . . , n
i=1
for m(n) , where λ(n) is the P-F maximal e-value of QC (n) ,
for succesively larger n. Here {C (n) } is a collection of
increasing, finite subsets of C . e.g. C (n) = {1, . . . , n}.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 22
Approximating QSDs
X
(n) (n)
mi qij
= −λ
(n)
(n)
mj ,
j ∈ C (n) .
i∈C (n)
What happens to these solutions as n becomes large?
We know that λ(n) ↓ λC as n → ∞.
But under what conditions does m(n) ‘converge’, in
some sense?
If m(n) converges, does it converge to m?
Breyer & Hart (2000) give some sufficient conditions.
We know that northwest corner truncation works for
Birth-Death Processes, and
Subcritical Markov Branching Process.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 23
The Quasi-Stationary Distribution
Sample Path of B-D Chemical Process with α=100, k =1, k =1, k =1
1
-1
2
300
250
State
200
150
100
50
0
0
0.05
0.1
0.15
Time
0.2
0.25
0.3
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 24
Another Chemical Reaction
2X
2X
A+X
k1
B
k2
This is not a Birth-Death process: it has jumps up of
size 1, but jumps down of size 2:
qi,i+1 = αik1 ,
i(i − 1)
.
qi,i−2 = k2
2
Hopefully my conjecture can deal with this!!
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 25
Further Work
The ‘renewal dynamical’ approach of Ferrari, Kesten,
Martínez and Picco:
When the process is absorbed, instantaneously restart
it in C according to some law ν :
ν
qij
= qij + qi0 νj ,
i 6= j ∈ C.
If the original chain is absorbed in finite mean time, this
modified chain P ν is positive recurrent and therefore
has a stationary distribution, say ρ.
Define a function Φ by Φ(ν) = ρ.
If Φ(m) = m then m is a QSD.
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 26
Further Work
The domain of attraction problem:
Our definition of a limiting conditional distribution looked
at the behaviour of
pij (t)
1 − pi0 (t)
We would like to be able to say what happens to
pj (t)
,
1 − pj (t)
pj (t) =
P
i ai pij (t)
with an arbitrary initial distribution a (over C ).
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 27
Further Work
Convergence to quasistationarity (LC-ness) - another
‘decay parameter’
β = sup{γ > 0 : |mij (t) − mj | = O(e−γt ) as t → ∞}
pij (t)
where mij (t) = 1−p
gives the conditional transition
i0 (t)
probabilities and mj = limt→∞ mij (t) is the LCD.
This provides a measure of how quickly the process
settles to a quasistationary regime.
For the chemical process we have looked at,
β ≈ 102 ,
λC ≈ 10−6 .
Quasi-Stationary Distributions for Continuous-Time Markov Chains – p. 28