Lecture IV
Markov Chain Approximation Methods
Nawaf Bou-Rabee
Discrete-Time Schemes
Standard way to simulate SDE:
Noise field
Drift field
dY = µ(Y )dt +
p
2 (Y )dW
where
n
µ:R !R
n
: Rn ! Rn⇥n
is by time discretization, e.g., given time step and current state, update state via:
X̃k+1
p
= X̃k + ht µ(X̃k ) + 2ht (X̃k ) (W (tk+1 ) W (tk ))
|
{z
}
Brownian increments
This method is simple to program, but may be inaccurate in some situations. Key Role of Hydrodynamic Interactions in Colloidal Gellation
Furukawa and Tanaka 2010 PRL gradient dynamics
(
dY = M (Y )DE(Y )dt +
M (x) = (x) (x)T
Initial condition:
1
p
div M (Y )dt + 2
1
(Y )dW
heat bath
Energy function: E(x1 , · · · , xN ) =
N
X1
N
X
U (|xi
i=1 j=i+1
1
r24
U (r)
steep hard core
with attractive tail
D1
r
D2
xj |)
ODE solution
using forward
Euler
SDE solution using
forward Euler with
small time-step
SDE solution using
forward Euler with
large time-step
Cubic
Oscillator
random global attractor
dY =
3
Y dt +
p
2dW
Strong & Weak Divergence in finite-time of Euler’s Method for SDEs with non-globally Lipschitz
continuous coefficients Hutzenhalter, Jentzen, and Kloeden Proc Roy Soc A 2011
Cubic
Oscillator
numerical drift is destabilizing
dY =
Y 3 dt +
p
2dW
Stochastic simulations of DNA in flow: Dynamics & the effects of HI
Jendrejack, de Pablo, and Graham J Chem Phys 2002 linear flow + spring force
(
1
dY = Y dt M (Y )DE(Y )dt +
M (x) = (x) (x)T
div M (Y )dt +
p
2
1
(Y )dW
heat bath
Initial condition
Extensional Flow
WLC
finitely extensible springs
3.0
1.0
2.5
-20
0.5
2.0
0
0.0
y
20
20
0
1.5
x
1.0
- 0.5
0.5
- 1.0
- 1.0
- 0.5
0.0
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
increase flow rate
boundary at zero is
attainable
2ba/σ 2 =0.12
35
30
-log PDF
25
20
15
10
5
0
10 -1
10 0
10 1
x
10 2
10 3
Cox-Ingersoll-Ross
Process 40
Stationary Density Plot, c1 =0.59, c2 =5.01
35
-log(PDF)
30
25
20
15
10
5
10 -4
10 -2
prey
10 0
Lotka-Volterra
Process
Issues with time
§  Exploding trajectories due to stiff coefficients
discretization
§  Nonphysical moves in SDEs with boundaries
§  Not suitable for long time simulations
ODE solution
Same time step and initial condition
SDE solution
Control of the approximation in space would resolve these issues.
Markov Chain Approximation Methods
Due to Harold J. Kushner an American applied mathematician at Brown University. He
introduced these numerical methods to solve stochastic control problems in 1970.
Kolmogorov equations with unbounded coefficients
(
= Lu(x, t) 8t > 0, x 2 ⌦ ⇢ Rn
u(x, 0) = '(x) ' 2 Bb (⌦, R)
@u
@t (x, t)
Drift field
Noise field
Lf (x) = Trace µ(x)Df (x)T + D2 f (x) (x) (x)T
Associated SDE:
dY = µ(Y )dt +
Describes evolution of conditional
expectation of an observable with respect to SDE solution
p
2 (Y )dW
u(x, t) = Ex ('(Y (t)))
semidiscrete Kolmogorov equations
n
u̇h (t) = Quh (t)
8t
0,
uh (0) = '|S
Q must satisfy
(a) Qf
(x)
(b) Qf
(x)
=
X
p
e.g. infinite grid
= Lf (x) + O(h )
q(x, y)(f (y)
f (x)) ,
y2S
(a) implies accuracy
(b) implies Q is realizable
q(x, y)
0
1D Example
0
00
Lf (x) = µ(x)f (x) + M (x)f (x)
Upwind finite difference for drift:
forward difference
+
µ(x)
µ(x)f 0 (x) ⇡
(f (x + x)
x
f (x)) +
backward difference
µ(x)
(f (x
x
x)
f (x))
x)
f (x))
Central finite difference for diffusion:
M (x)
M (x)f (x) ⇡
(f (x + x)
2
x
00
M (x)
f (x)) +
(f (x
2
x
Resulting spatial discretization is (a)  First-order accurate
(b) Realizable stochastic simulation algorithm (SSA)
Popularized by
Dan Gillespie
(1976)
Given current state and time:
X(t0 )
due to “Joe” Doob
(~1945)
=x
(step 1) update time t 1 = t0 + t ,
t ⇠ Exp(
(step 2) update state with probability X
q(x, y))
y2S\{x}
q(x, y)
P(X(t1 ) = y | X(t0 )) = P
y2S\x q(x, y)
Sample Paths
MALA
Qu-SSA
Qc-SSA
10
0
10
-2
10
-3
MFPT Accuracy
10 -3
Qu-SSA
Qc-SSA
10 -6
10 -9
4
-1
10 1 10 2 10 3
step index
2
MFPT
X(t)
6
10
relative error
SSA time
8
10
10-4
x
0
10-5
Cubic oscillator
-2
20
60
80
100
-log PDF
10
!1 error
relative error
10-3
10-4
Cubic oscillator
10-5
10-1
-6
10-2
10-1
spatial step size
-2
10-3
10-4
x
10-5
Qu
Qc
O(δx)
O(δx2 )
10-3
10-2
spatial step size
10-1
x
10
Committor Function Accuracy
10-60
10
10-3
Stationary Density
Accuracy
t
10-1
10-2
40
committor
0
Qu
Qc
O(δx)
O(δx2 )
100
10
Qu
Qc
O(δx)
O(δx2 )
-6
10
-3
-2
10
spatial step size
10
-1
High Order Discretization Schemes for the CIR Process Alfonsi Math of Comp 2010 dY = (a
kY )dt + Y dW ,
Boundary condition at zero is attainable if
Natural Boundary Condition at Zero
2
is large enough.
Regular Boundary Condition at Zero
0.5
0
0.45
100
1.6
10-1
0.4
10-1
SSA time
10
SSA time
1.8
1.4
0.35
10-2
1.2
10-2
0.3
10
-3
1
101
102
step index
103
X(t)
X(t)
Y (0) > 0
10-3
0.8
0.2
0.6
0.15
0.4
0.1
0.2
0.05
0
0
0.1
0.2
0.3
0.4
0.5
t
0.6
0.7
0.8
0.9
1
101
0.25
102
step index
0
0
0.1
0.2
0.3
0.4
Standard integrators may exit the domain of the SDE.
0.5
t
0.6
0.7
0.8
0.9
1
-1
10
Finite-Time Accuracy
Ex exp( Y (1))
natural
regular
O(δx)
O(δx2 )
Truncated Euler needs 50 times more function
evaluations to be this accurate.
-2
10-3
100
natural
regular
O(δx2 )
!δt"
relative error
10
10
-1
10
-2
10
-3
10-4
10-4 -2
10
10-5 -2
10
10-1
δx
10-1
δx
100
100
Long-Time Accuracy
Natural Boundary, 2a/σ 2 =1.56
40
10
Natural Boundary, 2a/σ 2 =1.56
-1
35
10-2
25
coarsest grid
10
-2
10
0
10
10-3
2
!1 error
-log PDF
30
20
10
-4
10
-5
10
-6
15
Qu
Qc
O(δx)
O(δx2 )
10
5
10
-2
10
x
0
10
2
10
-2
-1
10
spatial step size
Long-Time Accuracy
Regular Boundary, 2a/σ 2 =0.02
30
10
25
10-2
coarsest grid
10
-4
10
-2
10
10-3
0
!1 error
20
-log PDF
Regular Boundary, 2a/σ 2 =0.02
-1
15
10-4
10
Qu
Qc
O(δx)
O(δx2 )
10-5
5
0
10
-2
10
x
0
10
2
10
-6
10-2
10-1
spatial step size
Spectrum of OU Operators in Lp spaces with respect to invariant measures
Metafune, Pallara, and Priola 2002 J Funct Anal
(A) = { 1 , 2 }
<( 1 ), <( 2 ) < 0
dY = AY dt + dW
Complex Eigenvalues
5
(L) = {n1
1
+ n2
2
flow-free
-5
-6
λn
-3
-4
-5
-5
-5
-6
5
10
n
15
20
-3
Re
-2
-1
0
Real Eigenvalues
shear
-3
-4
0
-4
-2
-4
-6
-5
-1
-2
-3
0
0
extensional
-1
-2
Im
Real Eigenvalues
0
-1
rotational
: n1 , n2 2 N}
Real Eigenvalues
0
λn
λn
Twenty eigenvalues of largest real part of the
operator L used to benchmark spatial
discretization Q
0
5
10
n
15
20
-6
0
5
10
n
15
20
Accuracy in σ(L)
10
-1
!2 relative error
OU process
10
10
-2
(F0)
(F1)
(F2)
(F3)
O(h)
O(h2 )
-3
10-4
10-1
spatial grid size
10-1
Stationary Density Accuracy
(F0)
(F1)
(F2)
(F3)
O(h2 )
!1 error
Asymmetric
OU process
10-2
10-3 -1
10
100
spatial grid size
Influence of stochastic perturbations on prey-predator systems
Rudnicki and Pichor 2007 Mathematical Biosciences Population dynamics model of predator (y) and prey (x)
-log(PDF)
8
"
#
2
>
k1 x xy
>
1x
>
µ(x,
y)
=
>
<
2
k2 y + xy
2y
"
#
11
12
>
M
M
>
T
>
(
)(x,
y)
=
diag(x,
y)
diag(x, y)
>
:
12
22
M
M
Support of finite-time 2
probability distribution
R+
Stationary Density Plot, c1 =0.74, c2 =1.02
40
Support of stationary distribution
35
R2+
30
25
R ⇥ {0}
20
15
10
5
Q̃
exact
10
-4
10
-2
prey
10
0
Stability by Stochastic Lyapunov Function
Method is geometrically ergodic when underlying SDE is. Assume weak dissipativity condition:
T
µ(x) x  ↵
|x|
2
8x 2 R
n
Markov process is irreducible because for any two grids points, it is
possible to get from one to the other in a finite number of jumps. ✓
|x|
Generator satisfies infinitesimal drift condition
V (x) = exp a
2
QV (x)
?
?
9R
:
V (x)
<0
8|x| > R
By Harris Theorem the process is geometrically ergodic
kPt f
⌫(f )kV  kf
⌫(f )kV e
t
2
◆
Finite-Time Accuracy Define the global error of the numerical method:
✏(t) = u|S (t)
uh (t)
Satisfies a non-homogeneous Cauchy problem on a Banach space
✏(t)
˙
Q✏(t) = (L
Q)u
Since Q is a bounded linear operator and u is continuous, variation
of constants formula gives unique solution
✏(t) =
Z
t
exp((t
0
s)Q)(L
Q)u(s)ds
generator
accuracy
δx =0.32
7.5
Benchmark
Sample
paths
7
Colloidal cluster
collapse
Rg (t)
6.5
δx =0.32
103
10
2
101
SSA time
E(Rg (t))
7
6.5
δx =0.32
7.5
6
6
100
10-1
10-2
10-3
100
5.5
100
102
t
For any
103
step index
104
5.5
10-2
100
t
102
f 2 Bb (⌦, R) there exists positive constant s.t. sup |Ex { f |S (X(t))}
t2[0,T ]
for all
103
102
x 2 S.
Ex {f (Y (t))}|  Cf (x)h
Pass to the limit in global error to obtain
|⌫h ( f |S )
⌫(f )|  Cf h2
2
Recap
ü  In several SDE problems it is desirable to bound the spatial step size of the
approximation. ü  Extended Markov Chain Approximation Methods to steady-state
simulation of SDEs. ü  Spatially discretized the Kolmogorov equation using a discretization that is
locally consistent and realizable. ü  Described how the Markov Chain Approximation Method may be simulated using
the SSA method.
ü  Verified that the scheme is weakly accurate, eliminates nonphysical moves, is
geometrically ergodic when the underlying SDE is, and adjusts its time-lag
according to the stiffness of the SDE coefficients. Continuous-Time Random Walks for the Numerical Solution of Stochastic Differential Equations
Bou-Rabee and Vanden-Eijnden, arXiv:1502.05034