Acta Mathematicae Applicatae Sinica, English Series
Vol. 29, No. 1 (2013) 1–14
DOI: 10.1007/s10255-013-0208-4
http://www.ApplMath.com.cn & www.SpringerLink.com
Acta Mathemacae Applicatae Sinica,
English Series
The Editorial Office of AMAS &
Springer-Verlag Berlin Heidelberg 2013
Construction and Regularity of Transition Functions on
Polish Spaces under Measurability Conditions
Liu-er Ye1 , Xian-ping Guo2†
1 College
2 School
of Economics, Jinan University, Guangzhou, 510632, China (E-mail: [email protected])
of Mathematics and Computational Science, Guangdong Province Key Laboratory of Computational
Science, Sen-Yat Sen University, Guangzhou, 510275, China (E-mail: [email protected])
Abstract This paper concerns the construction and regularity of a transition (probability) function of a nonhomogeneous continuous-time Markov process with given transition rates and a general state space. Motivating
from a lot of restriction in applications of a transition function with continuous (in t ≥ 0) and conservative
transition rates q(t, x, Λ), we consider the case that q(t, x, Λ) are only required to satisfy a mild measurability
(in t ≥ 0) condition, which is a generalization of the continuity condition. Under the measurability condition
we construct a transition function with the given transition rates, provide a necessary and sufficient condition
for it to be regular, and further obtain some interesting additional results.
Keywords
Transition rate, transition function, construction and regularity, measurability condition
2000 MR Subject Classification
1
60J25
Introduction
In this paper we are concerned with the construction and regularity of a transition (probability)
function of a continuous-time Markov process in a Polish space X with the Borel σ-algebra B.
As is well known, a transition function P (s, x; t, Λ) (with x ∈ X, Λ ∈ B, and 0 ≤ s ≤ t)
can uniquely determine a Markov process. But in turn, P (s, x; t, Λ) is usually constructed
from a given function q(t, x, Λ), named as transition rates. (We call q(t, x, Λ) homogeneous (or
nonhomogeneous) if it is independent of (or depends on) t). This subject has been studied by
some authors; see [3,12] for the case of homogeneous transition rates q(t, x, Λ), and [2,5,6,11]
for nonhomogeneous transition rates. In this paper, we focus on the case of nonhomogeneous
transition rates and Polich state spaces and so mainly describe the existing main works by
Breuer[2] , Feller[5] , Gihman, Skorokhod[6] and Hu[11] for the case of Polish state spaces. In
order to construct a transition function from given transition rates q(t, x, Λ) on a Polish space,
to the best of our knowledge, all existing works such as [2,5,6,11] require the so-called continuity
condition that q(t, x, Λ) is assumed to be continuous in time t ≥ 0 (for each fixed x ∈ X and
Λ ∈ B). Basing on the continuity condition on q(t, x, Λ), the associated continuous-time Markov
processes have applications in a wide variety of contexts such as stochastic control problems in
[4,7–9,13,14] and stochastic games in [10], to name a few.
In the real world, however, the above continuity condition on q(t, x, Λ) may fail to hold. In
fact, many discontinuous “policies” in stochastic control problems and stochastic games typically lead to discontinuous transition rates, and moreover, application examples, not satisfying
Manuscript received April 20, 2010.
Supported by the National Natural Science Foundation of China (No.10925107), Guangdong Province Universities and Colleges Pearl River Scholar Funded Scheme, and the Fundamental Research Funds for the Central
Universities (No.11612314).
† Corresponding Author
2
L.E. YE, X.P. GUO
the continuity condition, are easily given. Motivated by the requirement of such applications,
in this paper we weaken the continuity condition to a mild measurability condition that the
transition rates q(t, x, Λ) are required to be measurable in (t, x) (for each fixed Λ ∈ B). Basing
on this measurability condition, we construct a transition function P (s, x; t, Λ) from q(t, x, Λ),
and further establish the regularity of P (s, x; t, Λ). Hence, we extend the main results about the
construction and regularity of transition functions under the continuity condition in [2,5,6,11]
to the case of the measurability condition.
Concerning the construction and regularity of P (s, x; t, Λ) from q(t, x, Λ) under the continuity condition, Feller[5] uses two different ways to obtain the same transition function P (s, x; t, Λ).
The arguments in the two ways require the continuity condition. On the other hand, Gihman,
Skorokhod[6] use a direct and simple approach to obtain P (s, x; t, Λ) from q(t, x, Λ), but this
approach requires the very strict assumption that q(t, x, Λ) is continuous in t ≥ 0 and uniformly
bounded in (t, x, Λ). Also, under the continuity condition, Hu[11] uses one of the two ways in
Feller[5] to obtain a transition function that only satisfies the backward equation. To verify that
such a transition function satisfies the forward equation, Hu[11] further assumes that q(t, x, Λ)
is uniformly bounded in (t, x). In this paper, the transition rates may be unbounded. Then,
under the measurability condition, we construct a transition function which satisfies both the
Kolmogorov’s forward and backward equations (see Theorem 1). Furthermore, we obtain a new
form of necessary and sufficient condition for the regularity of the constructive transition function (see Theorem 2 (ii)), and some interesting and new results such as Lemma 1 (ii), (iii) and
Theorem 2 (iii).
The rest of this paper is organized as follows. In Section 2 we introduce some preliminaries
about a Markov process, as well as the measurability condition. In Section 3 the construction
of a transition function is given, while the regularity of the constructive transition function is
discussed in Section 4. Some conclusions are presented in Section 5.
2
Basic Assumptions and Definitions
Preliminaries. Throughout the following X denotes a given polish space (that is, a complete
and separable metric space) with the Borel σ-algebra B. To begin with, we recall some concepts.
Definition 1.
A real-valued function P (s, x; t, Λ) is called a (standard) transition function
of a Markov process if it satisfies:
(i) for each fixed 0 ≤ s ≤ t, Λ ∈ B, P (s, ·; t, Λ) is a measurable function on X; for each
fixed 0 ≤ s ≤ t and x ∈ X, P (s, x; t, ·) is a measure on B with
0 ≤ P (s, x; t, X) ≤ 1;
(2.1)
(ii) the following continuity condition holds
½
lim P (s, x; t, Λ) = δx (Λ) =
t→s+
1,
0,
where δx (·) denotes the Dirac measure at x; and
(iii) the Chapman-Kolmogorov (C-K) equation
Z
P (s, x; t, Λ) =
P (s, x; u, dy)P (u, y; t, Λ),
if x ∈ Λ,
otherwise,
∀ s ≤ u ≤ t, x ∈ X
(2.2)
(2.3)
X
is true.
As is well known, under some conditions the following quantities exist
lim
∆s→0+
P (s, x; s + ∆s, Λ) − δx (Λ)
=: q(s, x, Λ) a.a. s ≥ 0,
∆s
∀ x ∈ X, Λ ∈ B.
(2.4)
Construction and Regularity of Transition Functions on Polish Spaces under Measurability Conditions
3
As is well known, a Markov process is completely determined by a transition function
P (s, x; t, Λ). From (2.4), the transition function P (s, x; t, Λ) is associated with its transition
rate q(s, x, Λ). Now the main problem is to determine whether or not to any function q(t, x, Λ),
subjected to some assumptions, there corresponds a transition function P (s, x; t, Λ) satisfying
(2.4).
The following assumptions on q(t, x, Λ) will be made throughout the paper.
Assumption A. A function q(t, x, Λ) is defined for all t ≥ 0, x ∈ X and Λ ∈ B, and it
satisfies:
(i) (the measurability condition) q(t, x) := −q(t, x, {x}) is a finite and non-negative measurable function of (t, x) ∈ [0, ∞) × X, and for each fixed x, it is integrable in t on every finite
time interval.
For x ∈
/ Λ and t ≥ 0, q(t, x, Λ) is finite and non-negative, and for each fixed Λ ∈ B, q(·, ·, Λ)
is a measurable function on [0, ∞) × X;
(ii) for fixed t ≥ 0 and x ∈ X, q(t, x, ·) is a signed measure on B satisfying
q(t, x, X) ≤ 0.
(2.5)
Remark 1. It should be mentioned that q(t, x) and q(t, x, Λ) are assumed to be continuous
in t ≥ 0 in [2,5,6,11] (for each fixed x ∈ X and Λ ∈ B) which is the continuity condition.
Assumption A here is a generalization of the continuity condition in [2,5,6,11]. Moreover,
application examples verifying Assumption A but not satisfying the continuity condition in
[2,5,6,11] are easily given and thus the details are omitted.
Let Q(t) be a set of values of q(t, x, Λ), i.e.,
©
Q(t) := q(t, x, Λ) : x ∈ X and Λ ∈ B},
t ≥ 0.
Now we give an exact definition of a transition function induced by Q(t). (The abbreviations
a.e. (almost everywhere) and a.a. (almost all) refer to the Lebesgue measure).
Definition 2. Let q(t, x, Λ) be a function that satisfies Assumption A. If a transition function
P (s, x; t, Λ) satisfies that
lim
∆s→0+
P (s, x; s + ∆s, Λ) − δx (Λ)
= q(s, x, Λ)
∆s
(2.6)
for all x ∈ X, Λ ∈ B, and a.a s ≥ 0. Then P (s, x; t, Λ) is called a (nonhomogeneous) Q(t)transition function.
Further, such a Q(t)-transition function is said to be regular if P (s, x; t, Λ) is the unique
function that satisfies (2.6) and P (s, x; t, X) ≡ 1.
Under Assumption A, we show that q(t, x, Λ) can determine a Q(t)-transition function, and
also give a regularity condition for the Q(t)-transition function.
3
Construction of a Q(t)-transition Function
In Definition 2 we have defined a Q(t)-transition function by a given q(t, x, Λ). Now we encounter a crucial question, that is, how can we ensure the existence of a Q(t)-transition function
with the given function q(t, x, Λ)? The following result shows that this can be done if the function q(t, x, Λ) satisfies Assumption A. Note that the conditions in Definition 2 are not very
strong for the transition function perse.
4
L.E. YE, X.P. GUO
Theorem 1. Given a function q(t, x, Λ) satisfying Assumption A, there exists a Q(t)-transition
function P (s, x; t, Λ). Furthermore, it satisfies the Kolmogorov’s forward and backward equations, that is, for a.a. t > s ≥ 0
∂P (s, x; t, Λ)
=
∂t
Z
P (s, x; t, dy)q(t, y, Λ),
X
∂P (s, x; t, Λ)
=−
∂s
Z
q(s, x, dy)P (s, y; t, Λ).
X
Since the method to prove Theorem 1 is very skillful, we need some important results
given in Lemma 1 below. First, we introduce some notations. Suppose that q(t, x, Λ) satisfies
Assumption A. For all 0 ≤ s ≤ t < ∞, x ∈ X, and Λ ∈ B, let
d(t, x, Λ) := δx (Λ)q(t, x)
(3.1)
and define recursively
Rt
−
q(v,x) dv
(s, x; t, Λ) := δx (Λ) e s
,
(3.2)
Z t Ru
Z
£
¤
(n+1)
(n)
−
q(v,x) dv
P
(s, x; t, Λ) :=
e s
q(u, x, dy) + d(u, x, dy) P (u, y; t, Λ) du
X
s
(3.3)
P
(0)
for every n ≥ 0. Let
P (s, x; t, Λ) :=
∞
X
P
(n)
(s, x; t, Λ).
(3.4)
n=0
Similarly, set
Z
−
q ∗ (s, x; t, Λ) :=
e
Rt
s
q(v,z)dv £
¤
q(s, x, dz) + d(s, x, dz)
(3.5)
Λ
and define recursively
Rt
−
q(v,x)dv
(s, x; t, Λ) := δx (Λ) e s
,
Z tZ
(n+1)
(n)
Q
(s, x; t, Λ) :=
Q (s, x; u, dy)q ∗ (u, y; t, Λ) du
(0)
Q
s
(3.6)
(3.7)
X
for every n ≥ 0. Let
Q(s, x; t, Λ) :=
∞
X
(n)
Q
(s, x; t, Λ).
(3.8)
n=0
Lemma 1. Given a function q(t, x, Λ) satisfying Assumption A. Then, for each x ∈ X, Λ ∈ B,
and 0 ≤ s ≤ t < ∞, we have
(n)
(n)
(i) P (s, x; t, Λ) = Q (s, x; t, Λ) for every n ≥ 0 and so P (s, x; t, Λ) = Q(s, x; t, Λ);
RtR
(ii) P (s, x; t, Λ) = s X P (s, x; v, dy)q(v, y, Λ)dv + δx (Λ);
RtR
(iii) P (s, x; t, Λ) = s X q(v, x, dy)P (v, y; t, Λ)dv + δx (Λ).
Proof. In view of Assumption A, the functions P
defined for each n ≥ 0.
(n)
(n)
(s, x, t, Λ) and Q
(s, x, t, Λ) are well
Construction and Regularity of Transition Functions on Polish Spaces under Measurability Conditions
(i) We use induction on n ≥ 0 to prove (i). Obviously, P
Taking (3.3) and (3.2) into account, we have
(0)
(0)
(s, x; t, Λ) = Q
5
(s, x; t, Λ).
Z t Ru
Z
£
¤ (0)
(1)
−
q(v,x)dv
P (s, x; t, Λ) =
e s
q(u, x, dy) + d(u, x, dy) P (u, y; t, Λ) du
s
X
Z t Ru
Z
Rt
£
¤
−
q(v,x)dv
−
q(v,y)dv
=
e s
q(u, x, dy) + d(u, x, dy) δy (Λ) e u
du
s
X
Z t Ru
Z
£
¤ − R t q(v,y)dv
−
q(v,x)dv
s
=
e
q(u, x, dy) + d(u, x, dy) e u
du
s
Λ
Z t Ru
−
q(v,x)dv ∗
=
e s
q (u, x; t, Λ)du
(by (3.5)).
s
By (3.7) and (3.6),
(1)
Q
Z tZ
(s, x; t, Λ) =
(0)
Q
s
(s, x; u, dy)q ∗ (u, y; t, Λ)du
X
Z tZ
=
−
δx (dy) e
s
Z
=
X
t
−
e
Ru
s
Ru
s
q(v,x)dv ∗
q (u, y; t, Λ)du
q(v,x)dv ∗
q (u, x; t, Λ)du.
s
(1)
(1)
Thus, P (s, x; t, Λ) = Q (s, x; t, Λ). Now assume that (i) holds for some n ≥ 0. In view of
(3.3), (3.7) and the induction hypothesis, it follows that
Z t Ru
Z
£
¤ (n)
(n+1)
−
q(v,x)dv
P
(s, x; t, Λ) =
e s
q(u, x, dy) + d(u, x, dy) P (u, y; t, Λ) du
s
X
Z t Ru
Z
£
¤ (n)
−
q(v,x)dv
=
e s
q(u, x, dy) + d(u, x, dy) Q (u, y; t, Λ)du
s
X
Z t Ru
Z
£
¤
−
q(v,x)dv
=
e s
q(u, x, dy) + d(u, x, dy)
s
X
i
hZ tZ
(n−1)
×
Q
(u, y; r, dz)q ∗ (r, z; t, Λ)dr du
u X
Z tZ
Z
hZ r Ru
i
(n−1)
−
q(v,x)dv
∗
s
=
(u, y; r, dz)du dr
q (r, z; t, Λ) ×
e
[q(u, x, dy) + d(u, x, dy)]P
s
X
s
X
Z tZ
Z tZ
(n)
(n)
=
q ∗ (r, z; t, Λ)P (s, x; r, dz)dr =
Q (s, x; r, dz)q ∗ (r, z; t, Λ)dr
s
X
(n+1)
=Q
s
X
(s, x; t, Λ),
which yields (i).
(ii) To show (ii), we first use induction and (3.1)–(3.3) to prove the following statement:
for each n ≥ 0 and t ≥ s ≥ 0,
Z tZ
P
s
(s, x; v, dy)d(v, y, Λ)dv
X
Z tZ
=
P
s
(n+1)
X
(n)
£
¤
(n+1)
(s, x; v, dy) q(v, y, Λ) + d(v, y, Λ) dv − P
(s, x; t, Λ).
(3.9)
6
L.E. YE, X.P. GUO
Note that by (3.2) and (3.1),
Z tZ
Z tZ
Rv
(0)
−
q(u,x)du
P (s, x; v, dy)d(v, y, Λ)dv =
δx (dy) e s
δy (Λ)q(v, y)dv
s
X
s
X
Z t Rv
(0)
q(u,x)du
−
=δx (Λ)
e s
q(v, x)dv = δx (Λ) − P (s, x; t, Λ).
(3.10)
s
Taking n = 0 in (3.3) and using (3.1) and the Fubini Theorem give
Z tZ
(1)
P (s, x; v, dy)d(v, y, Λ)dv
s
X
Z t Z nZ v Ru
Z
o
£
¤ (0)
−
q(r,x)dr
s
=
e
q(u, x, dz) + d(u, x, dz) P (u, z; v, dy)du d(v, y, Λ)dv
s
X
s
X
Z tZ Z v Ru
Z
Rv
£
¤
−
q(r,x)dr
−
q(r,z)dr
=
e s
q(u, x, dz) + d(u, x, dz) δz (dy) e u
δy (Λ)q(v, y)dudv
s
X s
X
Z tZ v Ru
Z
Rv
£
¤
−
q(r,x)dr
−
q(r,z)dr
=
e s
q(u, x, dz) + d(u, x, dz) δz (Λ) e u
q(v, z)dudv
s
s
X
Z t Ru
Z
hZ t Rv
i
£
¤
−
q(r,z)dr
−
q(r,x)dr
=
q(u, x, dz) + d(u, x, dz) δz (Λ)
e u
q(v, z)dv du
e s
X
u
s
Z t Ru
Z
Rt
h
i
£
¤
−
q(r,x)dr
−
q(r,z)dr
=
e s
q(u, x, dz) + d(u, x, dz) δz (Λ) 1 − e u
du
s
X
Z t Ru
Z
£
¤
−
q(r,x)dr
=
e s
q(u, x, dz) + d(u, x, dz) δz (Λ)du
s
X
Z t Ru
Z
£
¤ (0)
−
q(r,x)dr
−
e s
q(u, x, dz) + d(u, x, dz) P (u, z; t, Λ)du
s
X
Z t Ru
Z
£
¤
(1)
−
q(r,x)dr
=
e s
δx (dz) q(u, z, Λ) + d(u, z, Λ) du − P (s, x; t, Λ)
s
X
Z tZ
£
¤
(0)
(1)
P (s, x; u, dz) q(u, z, Λ) + d(u, z, Λ) du − P (s, x; t, Λ).
=
s
X
Thus, (3.9) holds for n = 0. Now suppose that (3.9) holds for some n ≥ 0. Then, by (3.3) and
the induction hypothesis, we obtain
Z tZ
(n+2)
P
(s, x; v, dy)d(v, y, Λ)dv
s
X
Z t Z nZ v Ru
Z
o
£
¤ (n+1)
−
q(r,x)dr
=
e s
q(u, x, dz) + d(u, x, dz) P
(u, z; v, dy)du d(v, y, Λ)dv
s
X
s
X
Z t Ru
Z
Z Z
i
£
¤h t
(n+1)
−
q(r,x)dr
=
e s
q(u, x, dz) + d(u, x, dz)
P
(u, z; v, dy)d(v, y, Λ)dv du
s
u X
X
Z t Ru
Z
£
¤
−
q(r,x)dr
e s
=
q(u, x, dz) + d(u, x, dz)
s
X
nZ t Z
o
(n)
(n+1)
×
P (u, z; v, dy)[q(v, y, Λ) + d(v, y, Λ)]dv − P
(u, z; t, Λ) du
u
X
(by induction hypothesis)
Z
nZ t Ru
£
¤
−
q(r,x)dr
=
e s
q(u, x, dz) + d(u, x, dz)
s
X
Construction and Regularity of Transition Functions on Polish Spaces under Measurability Conditions
Z tZ
×
u
Z tZ
P
(n)
7
o
(n+2)
(u, z; v, dy)[q(v, y, Λ) + d(v, y, Λ)]dvdu − P
(s, x; t, Λ)
X
£
¤
q(v, y, Λ) + d(v, y, Λ)
=
s
X
Z
nZ v Ru
o
£
¤ (n)
(n+2)
−
q(r,x)dr
s
×
e
q(u, x, dz) + d(u, x, dz) P (u, z; v, dy)du dv − P
(s, x; t, Λ)
s
X
Z tZ
£
¤
(n+1)
(n+2)
P
(s, x; v, dy) q(v, y, Λ) + d(v, y, Λ) dv − P
(s, x; t, Λ),
=
s
X
which means that (3.4) holds for n + 1. Hence, (3.9) is valid for all n ≥ 0.
Now note that (3.9) together with (3.4) gives
∞
X
P
(n+1)
n=0
∞ Z t
X
(s, x; t, Λ) =
n=0
tZ
Z
s
n=0
Z
−
∞ Z tZ
X
P
(n+1)
s
P
(n)
(s, x; v, dy)[q(v, y, Λ) + d(v, y, Λ)]dv
X
(s, x; v, dy)d(v, y, Λ)dv
X
=
P (s, x; v, dy)[q(v, y, Λ) + d(v, y, Λ)]dv
X
Z tZ
Z tZ
(0)
−
P (s, x; v, dy)d(v, y, Λ)dv +
P (s, x; v, dy)d(v, y, Λ)dv
s
X
s
X
Z tZ
(0)
P (s, x; v, dy)q(v, y, Λ)dv + δx (Λ) − P (s, x; t, Λ)
(by (3.10)),
=
s
s
X
which yields (ii).
(iii) The proof of (iii) is quite similar to that of (ii). We will first use induction and (3.1),
(3.5)–(3.7) to prove the following statement: for each n ≥ 0 and t ≥ s ≥ 0,
Z tZ
(n+1)
d(v, x, dy)Q
s
(v, y; t, Λ)dv
X
Z tZ
=
s
£
¤ (n)
(n+1)
(s, x; t, Λ),
q(v, x, dy) + d(v, x, dy) Q (v, y; t, Λ)dv − Q
(3.11)
X
which is analogous to (3.9). Indeed, by (3.6) and (3.1),
Z tZ
(0)
d(v, x, dy)Q
s
X
Z
=δx (Λ)
t
−
q(v, x) e
Z tZ
(v, y; t, Λ)dv =
Rt
v
s
q(r,x)dr
−
δx (dy) q(v, x) δy (Λ) e
(0)
dv = δx (Λ) − Q
(s, x; t, Λ).
Taking n = 0 in (3.7), using (3.1) and the Fubini’s Theorem we get
(1)
d(v, x, dy)Q (v, y; t, Λ)dv
Z tZ
hZ tZ
i
(0)
d(v, x, dy)
=
Q (v, y; u, dz)q ∗ (u, z; t, Λ)du dv
s
X
v
X
Z tZ
Ru
hZ tZ
i
−
q(r,y)dr ∗
δx (dy) q(v, x)
δy (dz) e v
q (u, z; t, Λ)du dv
=
s
X
s
X
v
X
v
q(r,y)dr
dv
X
s
Z tZ
Rt
(3.12)
8
L.E. YE, X.P. GUO
Z
Z tZ
Ru
−
q(r,x)dr ∗
q (u, z; t, Λ)dudv
q(v, x)
δx (dz) e v
s
v
X
Z tZ
Ru
i
hZ u
q(r,x)dr
−
δx (dz)
q(v, x) e v
dv q ∗ (u, z; t, Λ)du
=
s
X
s
Z tZ
h
n Z u
oi
=
δx (dz) 1 − exp −
q(r, x)dr q ∗ (u, z; t, Λ)du
s
X
s
Z tZ
Z tZ
(0)
=
δx (dz)q ∗ (u, z; t, Λ)du −
Q (s, x; u, dz)q ∗ (u, z; t, Λ)du
s
X
s
X
Z t
(1)
=
q ∗ (u, x; t, Λ)du − Q (s, x; t, Λ)
s
Z tZ
£
¤ − R t q(v,y)dv
(1)
=
q(u, x, dy) + d(u, x, dy) e u
du − Q (s, x; t, Λ)
(by ((3.5))
s
Λ
Z tZ
Rt
£
¤
(1)
−
q(v,y)dv
=
q(u, x, dy) + d(u, x, dy) δy (Λ) e u
du − Q (s, x; t, Λ)
s
X
Z tZ
£
¤ (0)
(1)
q(u, x, dy) + d(u, x, dy) Q (u, y; t, Λ)du − Q (s, x; t, Λ).
=
t
=
s
X
Hence, (3.11) holds for n = 0. Suppose that (3.11) holds for some n ≥ 0. Then, in view of (3.7)
and the induction hypothesis, we obtain
Z tZ
(n+2)
(v, y; t, Λ)dv
d(v, x, dy)Q
s
X
Z tZ
hZ tZ
i
(n+1)
=
d(v, x, dy)
Q
(v, y; u, dz)q ∗ (u, z; t, Λ)du dv
s
X
v
X
Z tZ hZ u Z
i
(n+1)
=
d(v, x, dy)Q
(v, y; u, dz)dv q ∗ (u, z; t, Λ)du
s
X
s
X
Z t Z nZ u Z
£
¤ (n)
=
q(v, x, dy) + d(v, x, dy) Q (v, y; u, dz)dv
s
X
s
X
o
(n+1)
−Q
(s, x; u, dz) q ∗ (u, z; t, Λ)du
(by induction hypothesis)
Z tZ Z uZ
£
¤ (n)
=
q(v, x, dy) + d(v, x, dy) Q (v, y; u, dz)q ∗ (u, z; t, Λ)dvdu
s
X s
X
Z tZ
(n+1)
Q
(s, x; u, dz)q ∗ (u, z; t, Λ)du
−
s
X
Z tZ
Z Z
i
£
¤h t
(n)
=
Q (v, y; u, dz)q ∗ (u, z; t, Λ)du dv
q(v, x, dy) + d(v, x, dy)
s
X
(n+2)
−Q
Z tZ
=
s
v
X
(s, x; t, Λ)
£
¤ (n+1)
(n+2)
q(v, x, dy) + d(v, x, dy) Q
(v, y; t, Λ)dv − Q
(s, x; t, Λ).
X
Consequently, (3.11) holds for n + 1 and this verifies (3.11).
Finally, from (3.11) and (3.8), we can get (iii) as
∞
X
n=0
(n+1)
Q
(s, x; t, Λ) =
∞ Z tZ
X
£
¤ (n)
q(v, x, dy) + d(v, x, dy) Q (v, y; t, Λ)dv
n=0
s
X
Construction and Regularity of Transition Functions on Polish Spaces under Measurability Conditions
−
Z
∞ Z tZ
X
n=0
tZ
s
(n+1)
d(v, x, dy)Q
9
(v, y; t, Λ)dv
X
£
¤
q(v, x, dy) + d(v, x, dy) Q(v, y; t, Λ)dv
s
X
Z tZ
Z tZ
(0)
−
d(v, x, dy)Q(v, y; t, Λ)dv +
d(v, x, dy)Q (v, y; t, Λ)dv
s
X
s
X
Z tZ
(0)
=
q(v, x, dy)Q(v, y; t, Λ)dv + δx (Λ) − Q (s, x; t, Λ)
(by (3.12)).
=
s
X
Hence,
Z tZ
Q(s, x; t, Λ) =
q(v, x, dy)Q(v, y; t, Λ)dv + δx (Λ).
s
X
This equality together with (i) gives (iii). Therefore Lemma 1 is thus verified.
2
Remark 2. The results in Lemma 1 (i)–(ii) are self-contained and does not appear in [2,5,6,11].
Moreover, they are useful in obtaining the Kolmogorov’s equations.
Now we can easily prove Theorem 1 by using Lemma 1.
Proof of Theorem 1.
In Lemma 1, we have constructed two functions P (s, x; t, Λ) and
Q(s, x; t, Λ) which satisfy some properties. Now we want to prove that the function P (s, x; t, Λ)
is a nonhomogeneous Q(t)-transition function. To this end, we need to show that P (s, x; t, Λ)
satisfies the conditions in Definition 1 and also (2.6) is satisfied for P (s, x; t, Λ).
Since q(u, x, ·) + d(u, x, ·) is a nonnegative measure on B (for any fixed u ≥ 0 and x ∈ X),
it is obviously satisfied that P (s, x; t, Λ) ≥ 0 for all t ≥ s ≥ 0, x ∈ X and Λ ∈ B.
To get that P (s, x; t, Λ) satisfies the second inequality of (2.1), we denote
n
X
S (n) (s, x; t, Λ) :=
P
(k)
(s, x; t, Λ),
∀ n ≥ 0.
(3.13)
k=0
(n)
Since P (s, x; t, Λ) ≥ 0 for all n ≥ 0, the sequence {S (n) } is non-negative and non-decreasing,
where S (n) := S (n) (s, x; t, Λ). On the other hand,
S (0) (s, x; t, Λ) = P
(0)
−
(s, x; t, Λ) = δx (Λ) e
Rt
s
q(v,x)dv
≤ 1.
Let us suppose that 0 ≤ S (0) ≤ S (1) ≤ · · · ≤ S (n−1) ≤ 1. Then, by (3.13) and (3.3)
S (n) (s, x; t, Λ) = P
=P
=P
(0)
(0)
(s, x; t, Λ) +
(0)
(s, x; t, Λ) +
n−1
XZ t
k=0
t
(s, x; t, Λ) +
−
e
s
≤P
(0)
Z
t
(s, x; t, Λ) +
−
e
s
=P
(0)
Z
(s, x; t, Λ) +
t
−
e
s
k=0
−
e
Ru
s
Ru
s
Ru
s
Ru
s
P
(k+1)
Z
q(v,x)dv
s
Z
n−1
X
Z
q(v,x)dv
(s, x; t, Λ)
£
¤ (k)
q(u, x, dy) + d(u, x, dy) P (u, y; t, Λ)du
X
£
¤
q(u, x, dy) + d(u, x, dy) S (n−1) (u, y; t, Λ)du
X
Z
q(v,x)dv
£
¤
q(u, x, dy) + d(u, x, dy) du
X
Z
q(v,x)dv
q(u, x, dy)du
X−{x}
10
L.E. YE, X.P. GUO
≤P
(0)
Z
t
(s, x; t, Λ) +
−
Ru
e
s
q(v,x)dv
q(u, x)du ≤ 1
(by (2.5)).
s
Hence, 0 ≤ S (n) (s, x; t, Λ) ↑ P (s, x; t, Λ) ≤ 1, which yields (2.1).
Moreover, P (s, x; t, Λ) satisfies (2.2). It is obvious since
¯
¯
¯P (s, x; t, Λ) − δx (Λ)¯
Rt
∞
X
£
(n+1)
q(v,x)dv ¤
−
=δx (Λ) 1 − e s
+
P
(s, x; t, Λ)
−
≤1 − e
n=0
Z
Rt
q(v,x)dv
s
t
+
−
Ru
e
s
Z
s
−
≤1 − e
−
Z
Rt
q(v,x)dv
s
q(v,x)dv
s
−
=2 − 2e
+
−
Ru
e
s
Rt
≤1 − e
X
t
Z
∞
£
¤X
(n)
q(u, x, dy) + d(u, x, dy)
P (u, y; t, Λ)du
q(v,x)dv
t
+
−
s
Ru
e
s
Z
n=0
£
¤
q(u, x, dy) + d(u, x, dy) du
q(v,x)dv
(by (2.1))
X
q(v,x)dv
q(u, x)du
(by (2.5))
s
Rt
q(v,x)dv
s
−→ 0 (as |t − s| → 0).
Next we will prove that the C-K equation (2.3) holds for P (s, x; t, Λ). In view of (3.4), it is
sufficient and necessary to prove that, for each n ≥ 0 and s ≤ u ≤ t,
n Z
X
(n)
(m)
(n−m)
P (s, x; t, Λ) =
P
(s, x; u, dy)P
(u, y; t, Λ).
(3.14)
m=0
X
In fact, (3.14) follows from (3.2) with n = 0. Suppose now that (3.14) holds for some n ≥ 0.
Then, by (3.3) and the induction hypothesis, we have that for any s ≤ r ≤ t,
(n+1)
P
(s, x; t, Λ)
Z t Ru
Z
£
¤ (n)
−
q(v,x)dv
=
e s
q(u, x, dy) + d(u, x, dy) P (u, y; t, Λ)du
s
Z
=
r
−
Ru
e
s
s
X
Z
q(v,x)dv
X
Z
t
−
+
Ru
e
s
n Z
X
m=0
m=0
Z
X
£
¤ (n)
q(u, x, dy) + d(u, x, dy) P (u, y; t, Λ) du
q(v,x)dv
r
=:
n Z
i
£
¤h X
(m)
(n−m)
q(u, x, dy) + d(u, x, dy)
P (u, y; r, dz)P
(r, z; t, Λ) du
X
P
(m+1)
(s, x; r, dz)P
(n−m)
(r, z; t, Λ) + C,
X
where
Z
Z
n
t
C :=
exp
r
−
u
q(v, x)dv
oZ £
¤ (n)
q(u, x, dy) + d(u, x, dy) P (u, y; t, Λ)du.
s
X
Since (3.14) holds for n = 0, recalling (3.2) we then have
Z
(0)
(n+1)
(0)
(n+1)
C = P (s, x; r, x)P
(r, x; t, Λ) =
P (s, x; r, dz)P
(r, z; t, Λ),
X
and so
P
(n+1)
(s, x; t, Λ) =
n+1
X
m=0
Z
P
X
(m)
(s, x; r, dz)P
(n+1−m)
(r, z; t, Λ).
Construction and Regularity of Transition Functions on Polish Spaces under Measurability Conditions
11
Hence, (3.14) holds for all n, and, as already noted, (2.3) follows for P (s, x; t, Λ).
To end the proof, noting that we have just shown that P (s, x; t, Λ) satisfies (2.1), (2.2) and
(2.3), it only remains to verify that P (s, x; t, Λ) satisfies (2.6). But in fact from Lemma 1 (ii)
and the fundamental theorem of calculus for Lebesgue integrals we can obtain
Z s+∆s h Z
i
P (s, x; s + ∆s, Λ) − δx (Λ)
1
lim
P (s, x; v, dy)q(v, y, Λ) dv
= lim
∆s
∆s→0+
∆s→0+ ∆s s
X
Z
=
P (s, x; s, dy)q(s, y, Λ) = q(s, x, Λ).
X
Moreover, Lemma 1 (ii)–(iii) give the Kolmogorov’s forward and backward equations, namely,
Z
Z
∂P (s, x; t, Λ)
∂P (s, x; t, Λ)
=
P (s, x; t, dy)q(t, y, Λ),
=−
q(s, x, dy)P (s, y; t, Λ)
∂t
∂s
X
X
for a.a. t ≥ s ≥ 0. Theorem 1 is thus proved.
4
2
Regularity of the Q(t)-transition Function
Under suitable assumptions, Theorem 1 shows that there exists a Q(t)-transition function
P (s, x; t, Λ) associated with the given function q(t, x, Λ). It is worth to note that we might have
P (s, x; t, X) < 1. We naturally want to know that under what conditions P (s, x; t, X) ≡ 1.
Moreover, to obtain a regular Q(t)-transition function, we have to impose some conditions on
the transition function itself. Then in the following, we will show that the Q(t)-transition
function P (s, x; t, Λ) in Lemma 1 is minimum; see (4.1) below, and also give a necessary and
sufficient condition for it to be regular; see (4.2) below.
To do these, we first give some notations. Given a function q(t, x, Λ), we denote by Pq
a class of Q(t)-transition functions satisfying that, for each fixed t ≥ 0, x ∈ X and Λ ∈ B,
∂P (s,x;t,Λ)
exists for a.a. 0 ≤ s ≤ t. Theorem 1 shows that Pq is not empty under Assumption
∂s
A, since P (s, x; t, Λ) ∈ Pq .
We have the following results.
Theorem 2. Suppose that the function q(t, x, Λ) satisfies Assumption A. Then
(i) P (s, x; t, Λ) in Lemma 1 is minimum in Pq . It implies that for any P (s, x; t, Λ) ∈ Pq ,
P (s, x; t, Λ) ≥ P (s, x; t, Λ),
∀ x ∈ X, Λ ∈ B, 0 ≤ s ≤ t < ∞;
(ii) P (s, x; t, Λ) is regular if and only if
Z thZ
i
P (s, x; v, dy)q(v, y, X) dv ≡ 0,
s
∀ x ∈ X, 0 ≤ s ≤ t < ∞.
(4.1)
(4.2)
X
(iii) Suppose that q(t, x, Λ) is conservative, i.e., q(t, x, X) ≡ 0 for all t ≥ 0 and x ∈ X. If
there
exist a measurable function w ≥ 0 on X, and constants c, b ≥ 0, and M > 0 such that
R
w(y)q(t, x, dy) ≤ cw(x) + b and q(t, x) ≤ M w(x) for all x ∈ X and t ≥ 0, then P (s, x; t, Λ)
X
is regular.
The proof of Theorem 2 needs a key result which is given in the following lemma.
Lemma 2.
Assume that q(t, x, Λ) satisfies Assumption A. Then for any P (s, x; t, Λ) ∈ Pq
we have, for each x ∈ X, Λ ∈ B and 0 ≤ s < t < ∞,
(i)
Z
∂P (s, x; t, Λ)
≤−
q(s, x, dy)P (s, y; t, Λ);
(4.3)
∂s
X
12
L.E. YE, X.P. GUO
(ii)
Z
t
P (s, x; t, Λ) ≥
−
e
Z
Ru
q(v,x)dv
s
s
−
+ δx (Λ)e
Rt
s
£
¤
q(u, x, dy) + d(u, x, dy) P (u, y; t, Λ)du
X
q(v,x)dv
.
(4.4)
Furthermore, if q(t, x, Λ) is conservative, i.e., q(t, x, X) ≡ 0 for all t ≥ 0 and x ∈ X, then (4.3)
and (4.4) take equality.
Proof. (i) Since P (s, x; t, Λ) ∈ Pq , ∂P (s,x;t,Λ)
exists for a.a. 0 ≤ s ≤ t. Using the C-K
∂s
equation (2.3), we can get that for ∆s ≥ 0,
¤
1 £
P (s + ∆s, x; t, Λ) − P (s, x; t, Λ)
∆s
1 h
=
P (s + ∆s, x; t, Λ) − P (s, x; s + ∆s, {x})P (s + ∆s, x; t, Λ)
∆sZ
i
−
P (s, x; s + ∆s, dy)P (s + ∆s, y; t, Λ)
X−{x}
¤
1 £
=
1 − P (s, x; s + ∆s, {x}) P (s + ∆s, x; t, Λ)
∆s
Z
1
P (s, x; s + ∆s, dy)P (s + ∆s, y; t, Λ).
−
∆s X−{x}
(4.5)
Hence, in view of (2.4), the requirement before (2.2) and the Fatou’s Lemma, we have
Z
Z
1
lim inf+
P (s, x; s + ∆s, dy)P (s + ∆s, y; t, Λ) ≥
q(s, x, dy)P (s, y; t, Λ),
∆s→0 ∆s X−{x}
X−{x}
which together with (2.4) and (4.5) gives (4.3).
On the other hand, if q(t, x, Λ) is conservative, one can easily show that
Z
Z
1
lim sup
P (s, x; s + ∆s, dy)P (s + ∆s, y; t, Λ) ≤
q(s, x, dy)P (s, y; t, Λ)
∆s→0+ ∆s X−{x}
X−{x}
and the last two inequalities imply
Z
Z
1
P (s, x; s + ∆s, dy)P (s + ∆s, y; t, Λ) =
q(s, x, dy)P (s, y; t, Λ).
lim
∆s→0+ ∆s X−{x}
X−{x}
Hence, (4.3) take equality.
(ii) From (4.3) and the fundamental theorem of calculus for Lebesgue integral we have
Z t Ru
Z
−
q(v,x)dv
s
e
[q(u, x, dy) + d(u, x, dy)]P (u, y; t, Λ)du
s
X
Z t Ru
Z
−
q(v,x)dv
=
e s
q(u, x, dy)P (u, y; t, Λ)du
X
s
Z t Ru
Z
−
q(v,x)dv
+ e s
d(u, x, dy)P (u, y; t, Λ)du
s
X
Z t Ru
Z t Ru
−
q(v,x)dv
−
q(v,x)dv
≤−
e s
dP (u, x; t, Λ) +
e s
q(u, x)P (u, x; t, Λ)du
s
s
Rt
−
q(v,x)dv
= − δx (Λ) e s
+ P (s, x; t, Λ),
Construction and Regularity of Transition Functions on Polish Spaces under Measurability Conditions
13
which yields (4.4), and the theorem is thus verified.
With Lemma 2, we can easily prove Theorem 2.
Proof of Theorem 2.
(i) Let P (s, x; t, Λ) be as defined in Lemma 1. Then, from Theorem 1, P (s, x; t, Λ) is a
Q(t)-transition function. Obviously, P (s, x; t, Λ) ∈ Pq . For any P (s, x; t, Λ) ∈ Pq , it follows
from (4.3) that
Z
∂P (u, x; t, {x})
≤−
q(u, x, dy)P (u, y; t, {x}),
∀ x ∈ X, 0 ≤ u ≤ t < ∞.
∂u
X
Since q(u, x, Λ) ≥ 0 for x ∈
/ Λ,
Z
∂P (u, x; t, {x})
≤−
q(u, x, dy)P (u, y; t, {x})
∂u
X
Z
=−
q(u, x, dy)P (u, y; t, {x}) − q(u, x, {x})P (u, x; t, {x})
X−{x}
≤q(u, x)P (u, x; t, {x})
and so
−
dP (u, x; t, {x})
≥ −q(u, x)du.
P (u, x; t, {x})
Integrating from s to t with respect to u on the both sides, we obtain
Z t
ln P (s, x; t, {x}) ≥ −
q(u, x)du,
s
that is, by (3.2),
−
P (s, x; t, {x}) ≥ e
Hence
P (s, x; t, Λ) ≥ P
(0)
Rt
s
q(u,x)du
(s, x; t, Λ),
=P
(0)
(s, x; t, {x}).
∀x ∈ X
and Λ ∈ B.
Suppose that for some n ≥ 0,
P (s, x; t, Λ) ≥
n
X
P
(k)
(s, x; t, Λ).
(4.6)
k=0
Taking (4.4) and the induction hypothesis into accout, we can obtain
Z t Ru
Z
£
¤
−
q(v,x)dv
P (s, x; t, Λ) ≥
e s
q(u, x, dy) + d(u, x, dy) P (u, y; t, Λ)du
s
X
Rt
−
q(v,x)dv
+ δx (Λ) e s
Z t Ru
Z
n
£
¤X
(k)
−
q(v,x)dv
s
e
≥
q(u, x, dy) + d(u, x, dy)
P (u, y; t, Λ)du
s
X
+P
=
n
X
k=0
(0)
P
k=0
(s, x; t, Λ)
(k+1)
(s, x; t, Λ) + P
(0)
(s, x; t, Λ)
(by (3.3)) =
n+1
X
k=0
Thus, (4.6) holds for all n ≥ 0. Letting n → ∞ in (4.6) we obtain (4.1).
P
(k)
(s, x; t, Λ).
14
L.E. YE, X.P. GUO
(ii) By Definition 2, P (s, x; t, Λ) is regular if and only if it is the unique function that
satisfies (2.6) and P (s, x; t, X) ≡ 1. The uniqueness obviously follows from (4.1). Then, by
Lemma 1 (ii), it is equivalent to
Z thZ
s
that is,
i
P (s, x; v, dy)q(v, y, X) dv + δx (X) ≡ 1,
X
Z thZ
s
i
P (s, x; v, dy)q(v, y, X) dv ≡ 0,
X
as required.
(iii) Following the arguments in [8], we see that the result is true.
5
2
Conclusions
In this paper we presented a fairly detailed, self–contained, exposition of the construction of a
nonhomogeneous transition function starting from a nonhomogeneous transition rate q(t, x, Λ)
that satisfies a very mild measurability condition. The constructive transition function verifies
the Kolmogorov’s forward and backward equations. Moreover, we have further presented a
necessary and sufficient condition for the constructive transition function to be unique and
regular. In particular, our arguments are based on the measurability condition which is a
generalization of the traditional continuity condition imposed on the given transition rates.
Acknowledgements.
suggestions.
We thank the anonymous referees for pertinent comments and valuable
References
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
Ash, R.B. Probability and Measure Theory, Second Edition. Academic Press, London, 2000
Breuer, L. From Markov Jump Processes to Spatial Queues. Kluwer, Dordrecht, 2003
Chen, M.F. From Markov Chains to Non-Equilibraum Particle Systems. World Scientific, 1992
Doshi, B.T. Continuous-time control of Markov provesses on an arbitrary atate space: discounted rewards.
Ann. Statist. 4, 1219–1235 (1976)
Feller, W. On the integro-differential equations of purely discontinuous Markoff processes. Trans. Amer.
Math. Soc., 48: 488–515 (1940)
Gihman, I.I., Skorokhod, A.V. The Theory of Stochastic Processes II. Springer-Verlag (This is a reprint of
the First edition 1975), 2004
Guo, X.P., Ulirich, R. Average optimality for continuous-time Markov processes in Polish spaces. Ann.
Appl. Probab., 16, 730–756 (2006)
Guo, X.P. Continuous-time Markov decision processes with discounted rewards: the case of Polish spaces.
Math. Oper. Res., 32(1): 73–87 (2007)
Guo, X.P. Constrained optimizatin for average cost continuous-time Markov decision processes. IEEE
Trans. Automat. Control, 52: 1139–1143 (2007)
Guo, X.P., Hernández-Lerma, O. Zero-sum games for continuous-time jump Markov processes in polish
spaces: discounted payoffs. Adv. Appl. Probab., 39, 645–668 (2007)
Hu, D.H. The Analysis Theory of Markov Process with General State. Wuhan: Hubei Education Press,
China, 1985 (in Chinese)
Hou, Z.T., Guo, Q.F. Time-homogeneous Markov Processes with Countable State Space. Beijing Science
Press and Sprinker-Verlag, 1978
Lai, H.C., Tanaka, K. On an N-person noncooperative Markov game with a metric state space. J. Math.
Anal. Appl., 101, 78–96 (1984)
Stidham, S., Weber, R. A survey of Markov decision models for the control of networks of queues. Queueing
Systems, 13, 291–314 (1993)
Wang, Z.K., Yang, X.Q. Birth and Death Processes and Markov Chains. Beijing Science Press and
Springer-Verlag, 1992