Lumpability of Markov chains and
reward processes
Ana Sokolova, Erik de Vink
{asokolov,evink}@win.tue.nl.
TU/e
Lumpability – p.1/14
Outline
•
What is lumpability all about?
Lumpability – p.2/14
Outline
•
What is lumpability all about?
∗ General lumpability
Lumpability – p.2/14
Outline
•
What is lumpability all about?
∗ General lumpability
∗ Rewards and performance analysis
Lumpability – p.2/14
Outline
•
What is lumpability all about?
∗ General lumpability
∗ Rewards and performance analysis
•
Ordinary lumpability
Lumpability – p.2/14
Outline
•
What is lumpability all about?
∗ General lumpability
∗ Rewards and performance analysis
•
Ordinary lumpability
•
Exact lumpability
Lumpability – p.2/14
Outline
•
What is lumpability all about?
∗ General lumpability
∗ Rewards and performance analysis
•
Ordinary lumpability
•
Exact lumpability
•
Properties of interest
Lumpability – p.2/14
Outline
•
What is lumpability all about?
∗ General lumpability
∗ Rewards and performance analysis
•
Ordinary lumpability
•
Exact lumpability
•
Properties of interest
•
Results
Lumpability – p.2/14
Outline
•
What is lumpability all about?
∗ General lumpability
∗ Rewards and performance analysis
•
Ordinary lumpability
•
Exact lumpability
•
Properties of interest
•
Results
Conclusions
•
Lumpability – p.2/14
What is lumpability all about ?
Lumpability – p.3/14
What is lumpability all about ?
Example:
M = (S, P, π)
@ABC
GFED
1[ ]
= O
aB
1
2
M
||
|
||
|
|| 1
/ 2[ 1 ] 2
o
4 aC
CC
CC 1
CC2
1
1
C
4
4
1
4
@ABC
GFED
@AGF BC
BB 3
BB4
1
BB
B
/ 1
3[ ]
1
={ 4
4
{{
{
{{ 14
{
{
4[0]
@ABC
GFED
@ABC
GFED
S = {1, 2, 3, 4}




P =



0
0
0
1
1
4
3
4
1
2
1
4
1
4
1
4
1
2

0 


0 

0
0
1
4
π = [ 12 , 14 , 14 , 0]
Lumpability – p.3/14
What is lumpability all about ?
Example:
@ABC
GFED
1[ ]
= O
aB
1
2
M
||
|
|
|
|
|| 1
/ 2[ 1 ] 2
o
4 aC
CC
CC 1
CC2
1
1
C
4
4
1
4
@ABC
GFED
@AGF BC
BB 3
BB4
1
BB
B
/ 1
3[ ]
1
={ 4
4
{{
{
{{ 14
{
{
4[0]
@ABC
GFED
@ABC
GFED
Lumpability – p.3/14
What is lumpability all about ?
Example:
@ABC
GFED
1[ ]
= O
aB
1
2
M
||
|
|
|
|
|| 1
/ 2[ 1 ] 2
o
4 aC
CC
CC 1
CC2
1
1
C
4
4
1
4
@ABC
GFED
@AGF BC
BB 3
BB4
1
BB
B
/ 1
3[ ]
1
={ 4
4
{{
{
{{ 14
{
{
4[0]
@ABC
GFED
@ABC
GFED
M̂
?
GF@A/ @ABC
EDGFED
x[1] o
BC
/ GFED
@ABC
GFED
y[0] o
?
?
?
Lumpability – p.3/14
What is lumpability about ?
•
Markov chain, X(n) = M = (S, P, π)
S−
hrhr93 O
h
h
h
hhh rrr
h
h
h
r
hhh
r
h
h
r
h
rr
hhh
r
rr
r
state space rrr
r
rr
r
rr
r
r
rr
P − trans. prob.
π − initial prob.
Lumpability – p.4/14
What is lumpability about ?
•
Markov chain, X(n) = M = (S, P, π)
S−
hrhr93 O
h
h
h
hhh rrr
h
h
h
r
hhh
r
h
h
r
h
rr
hhh
r
rr
r
state space rrr
r
rr
r
rr
r
r
rr
P − trans. prob.
•
π − initial prob.
Partition L on S.
Lumpability – p.4/14
What is lumpability about ?
•
Markov chain, X(n) = M = (S, P, π)
S−
hrhr93 O
h
h
h
hhh rrr
h
h
h
r
hhh
r
h
h
r
h
rr
hhh
r
rr
r
state space rrr
r
rr
r
rr
r
r
rr
P − trans. prob.
•
π − initial prob.
Partition L on S.
⇒ Lumped stochastic process is X̂(n), with
X̂(n) = i ⇔ X(n) ∈ Ci ∈ L.
Lumpability – p.4/14
What is lumpability about ?
•
Markov chain, X(n) = M = (S, P, π)
S−
hrhr93 O
h
h
h
hhh rrr
h
h
h
r
hhh
r
h
h
r
h
rr
hhh
r
rr
r
state space rrr
r
rr
r
rr
r
r
rr
P − trans. prob.
•
π − initial prob.
Partition L on S.
⇒ Lumped stochastic process is X̂(n), with
X̂(n) = i ⇔ X(n) ∈ Ci ∈ L.
•
When is X̂(n) a markov chain ???
Lumpability – p.4/14
General lumpability
Th. The Markov chain M = (S, P, π) is lumpable w.r.t a
partition L = {C1 , . . . , Cm } on S iff there exists a matrix P̂
of order m, such that for all i, j ∈ {1, . . . , m} and for all
k ≥ 0 it holds
P
P
0
k 0 0
π(i
)
P
(i , j )
0
0
i ∈Ci
j ∈Cj
k
P̂ (i, j) =
π̂(i)
Lumpability – p.5/14
General lumpability
Th. The Markov chain M = (S, P, π) is lumpable w.r.t a
partition L = {C1 , . . . , Cm } on S iff there exists a matrix P̂
of order m, such that for all i, j ∈ {1, . . . , m} and for all
k ≥ 0 it holds
P
P
0
k 0 0
π(i
)
P
(i , j )
0
0
i ∈Ci
j ∈Cj
k
P̂ (i, j) =
π̂(i)
L
Notation: M →l M̂ , M̂ = (L, P̂ , π̂) for
P
π̂(i) = i0 ∈Ci π(i0 )
M1 ∼l M2 if they have a common lumping.
Lumpability – p.5/14
General lumpability - rewards
Def. The MRP M = (S, P, π, r) is lumpable w.r.t a partition
L = {C1 , . . . , Cm } on S iff the chain (S, P, π) is.
Notation: M
L
→lp
M̂ , M̂ = (L, P̂ , π̂, r̂) for
Lumpability – p.6/14
General lumpability - rewards
Def. The MRP M = (S, P, π, r) is lumpable w.r.t a partition
L = {C1 , . . . , Cm } on S iff the chain (S, P, π) is.
Notation: M
L
→lp
M̂ , M̂ = (L, P̂ , π̂, r̂) for
P
0
0
π(i
)r(i
)
0
i ∈Ci
r̂(i) = P
0)
π(i
0
i ∈Ci
Lumpability – p.6/14
General lumpability - rewards
Def. The MRP M = (S, P, π, r) is lumpable w.r.t a partition
L = {C1 , . . . , Cm } on S iff the chain (S, P, π) is.
Notation: M
L
→lp
M̂ , M̂ = (L, P̂ , π̂, r̂) for
P
0
0
π(i
)r(i
)
0
i ∈Ci
r̂(i) = P
0)
π(i
0
i ∈Ci
P
Recall: pm(M ) = i∈S r(i)π(i)
Th. If M
L
→lp
M̂ then pm(M ) = pm(M̂ ).
Lumpability – p.6/14
Ordinary lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all i0 , i00 ∈ Ci
X
X
P (i0 , j 0 ) =
P (i00 , j 0 )
j 0 ∈Cj
j 0 ∈Cj
then M is ordinary lumpable w.r.t L.
Lumpability – p.7/14
Ordinary lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all i0 , i00 ∈ Ci
X
X
P (i0 , j 0 ) =
P (i00 , j 0 ) = P̂ (i, j)
j 0 ∈Cj
j 0 ∈Cj
then M is ordinary lumpable w.r.t L.
Lumpability – p.7/14
Ordinary lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all i0 , i00 ∈ Ci
X
X
P (i0 , j 0 ) =
P (i00 , j 0 ) = P̂ (i, j)
j 0 ∈Cj
j 0 ∈Cj
then M is ordinary lumpable w.r.t L.
Notation: M
L
→ol
M̂ , M̂ = (L, P̂ , π̂)
Lumpability – p.7/14
Ordinary lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all i0 , i00 ∈ Ci
X
X
P (i0 , j 0 ) =
P (i00 , j 0 ) = P̂ (i, j)
j 0 ∈Cj
j 0 ∈Cj
then M is ordinary lumpable w.r.t L.
Notation: M
L
→ol
M̂ , M̂ = (L, P̂ , π̂)
M1 ∼ol M2 if they have a common ordinary lumping
Lumpability – p.7/14
Ordinary lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all i0 , i00 ∈ Ci
X
X
P (i0 , j 0 ) =
P (i00 , j 0 ) = P̂ (i, j)
j 0 ∈Cj
j 0 ∈Cj
then M is ordinary lumpable w.r.t L.
Notation: M
L
→ol
M̂ , M̂ = (L, P̂ , π̂)
M1 ∼ol M2 if they have a common ordinary lumping
Prop. M
L
→ol
L
M̂ ⇒ M →l M̂
Lumpability – p.7/14
Example - ordinary lumpability
M = (S, P, π)
1
2
GFED
BC
@ABC
GFED
1[ ] o
= O
aB
S = {1, 2, 3, 4}
1
2
M
BB 1
||
|
1 BB4
|
BB
|
2
|
B
|
| 1
/ 1
/ 2[ 1 ] 2
o
3[
]
1
o
4 aC
4
2
CC
CC 1
CC8
1
1
5
C 4
4
8
/ 4[0]
1
4
@ABC
GFED
@AGF BC
@ABC
GFED
@ABCED
@ABC
GFED
@AGF BC
1
4
L = {{1, 2}, {3, 4}}




P =


1
2
1
4
1
4
1
8

0
0
1
2
1
4
1
2
5
8
1
2
1
4

0 


0 

0
1
4
π = [ 12 , 14 , 14 , 0]
Lumpability – p.8/14
Example - ordinary lumpability
M = (S, P, π)
S = {1, 2, 3, 4}
L = {{1, 2}, {3, 4}}
M̂
1
2
GF@A @ABC
EDGFED
/ x[ 3 ]
4 o
1
4
1
2
3
4
BC
/ GFED
@ABC
GFED
y[ ] o
1
4




P =


1
2
1
4
1
4
1
8

0
0
1
2
1
4
1
2
5
8
1
2
1
4

0 


0 

0
1
4
π = [ 12 , 14 , 14 , 0]
Lumpability – p.8/14
Exact lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all j 0 , j 00 ∈ Cj
X
X
P (i0 , j 0 ) =
P (i0 , j 00 ), π(j 0 ) = π(j 00 )
i0 ∈Ci
i0 ∈Ci
then M is exactly lumpable w.r.t L.
Lumpability – p.9/14
Exact lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all j 0 , j 00 ∈ Cj
X
X
P (i0 , j 0 ) =
P (i0 , j 00 ), π(j 0 ) = π(j 00 )
i0 ∈Ci
i0 ∈Ci
then M is exactly lumpable w.r.t L.
L
Notation: M →el M̂ , M̂ = (L, P̂ , π̂) for
|Cj | P
P̂ (i, j) = |Ci | i0 ∈Ci P (i0 , j 0 )
Lumpability – p.9/14
Exact lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all j 0 , j 00 ∈ Cj
X
X
P (i0 , j 0 ) =
P (i0 , j 00 ), π(j 0 ) = π(j 00 )
i0 ∈Ci
i0 ∈Ci
then M is exactly lumpable w.r.t L.
L
Notation: M →el M̂ , M̂ = (L, P̂ , π̂) for
|Cj | P
P̂ (i, j) = |Ci | i0 ∈Ci P (i0 , j 0 )
M1 ∼el M2 if they have a common ordinary lumping
Lumpability – p.9/14
Exact lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all j 0 , j 00 ∈ Cj
X
X
P (i0 , j 0 ) =
P (i0 , j 00 ), π(j 0 ) = π(j 00 )
i0 ∈Ci
i0 ∈Ci
then M is exactly lumpable w.r.t L.
L
Notation: M →el M̂ , M̂ = (L, P̂ , π̂) for
|Cj | P
P̂ (i, j) = |Ci | i0 ∈Ci P (i0 , j 0 )
M1 ∼el M2 if they have a common ordinary lumping
L
L
Prop. M →el M̂ ⇒ M →l M̂
Lumpability – p.9/14
Exact lumpability
Def. Let M = (S, P, π) and L = {C1 , . . . , Cm } a partition on
S. If for all Ci , Cj ∈ L and all j 0 , j 00 ∈ Cj
X
X
P (i0 , j 0 ) =
P (i0 , j 00 ), π(j 0 ) = π(j 00 )
i0 ∈Ci
i0 ∈Ci
then M is exactly lumpable w.r.t L.
L
Notation: M →el M̂ , M̂ = (L, P̂ , π̂) for
|Cj | P
P̂ (i, j) = |Ci | i0 ∈Ci P (i0 , j 0 )
M1 ∼el M2 if they have a common ordinary lumping
Note: In the exact
P case 0 for rewards
i0 ∈Ci r(i )
r̂(i) =
|Ci |
Lumpability – p.9/14
Example - exact lumpability
M = (S, P, π)
1
2
M
||
|
||
|
|| 1
/ 2[ 1 ] 2
o
4 aC
CC
CC 1
CC2
1
1
C
4
4
1
4
@ABC
GFED
@AGF BC
S = {1, 2, 3, 4}
@ABC
GFED
1[ ]
= O
aB
BB 3
BB4
1
BB
B
/ 1
3[ 4 ]
1
=
4
{
1
4 {{
{{
{
{{
4[0]
@ABC
GFED
@ABC
GFED
L = {{1}, {2, 3}, {4}}




P =



0
0
0
1
1
4
3
4
1
2
1
4
1
4
1
4
1
2

0 


0 

0
0
1
4
π = [ 12 , 14 , 14 , 0]
Lumpability – p.10/14
Example - exact lumpability
M = (S, P, π)
S = {1, 2, 3, 4}
@ABC
GFED
a[ ] B
B
=
1
2
M̂
||
|
||
|
||
/ b[ 1 ] o
1
2
1
2
GF@A @ABC
EDGFED
2
1
2
L = {{1}, {2, 3}, {4}}
`BB BBB1
BB BB
BB B
1
BB
2
c[0]
@ABC
GFED




P =



0
0
0
1
1
4
3
4
1
2
1
4
1
4
1
4
1
2

0 


0 

0
0
1
4
π = [ 12 , 14 , 14 , 0]
Lumpability – p.10/14
Properties
transitivity:
M1
/
M2
/
M3
?
⇒
M1
/
M3
Lumpability – p.11/14
Properties
transitivity:
M1
/
M2
/
M3
?
⇒
M1
/
M3
p-transitivity:
>> L00
>>
>
00
_ _ L̄_ _ _/ M
L0
M1
M
@ 2>
3
Lumpability – p.11/14
Properties
transitivity:
M1
/
M2
/
M3
?
⇒
M1
/
M3
p-transitivity:
>> L00
>>
>
00
_ _ L̄_ _ _/ M
L0
M1
M
@ 2>
3
Prop. p-transitivity implies transitivity
Lumpability – p.11/14
Properties
3 - property:
M >>
>>
>>

M1>
M2
>
>
M0
Lumpability – p.12/14
Properties
3 - property
M >>
>>
>>

M1>
M2
∗3 - property:
M >>
>>L2
>>
L1

M1>
M2
>
>
>
M0
L1 |L∗ >
M0
L2 |L∗
Lumpability – p.12/14
Properties
3 - property
M >>
>>
>>

M1>
M2
∗3 - property:
M >>
>>L2
>>
L1

M1>
M2
>
>
>
L1 |L∗ >
M0
Prop. ∗3 implies 3.
M0
L2 |L∗
Lumpability – p.12/14
Properties
3 - property
M >>
>>
>>

M1>
M2
∗3 - property:
M >>
>>L2
>>
L1

M1>
M2
>
>
>
L1 |L∗ >
L2 |L∗
M0
M0
Prop. ∗3 implies 3.
Prop. 3 and transitivity imply ∼ equivalence.
Lumpability – p.12/14
Results - lumpability relations
ordinary exact general
p-transitivity
yes
no
yes
transitivity
yes
no
yes
3
yes
no
yes
∗3
yes
no
no
∼ is equiv.
yes
no
yes
Lumpability – p.13/14
Results - lumpability relations
ordinary exact general
p-transitivity
yes
no
yes
transitivity
yes
no
yes
3
yes
no
yes
∗3
yes
no
no
∼ is equiv.
yes
no
yes
The same holds with or without rewards.
Lumpability – p.13/14
Conclusion
•
for the properties of interest
Lumpability – p.14/14
Conclusion
•
for the properties of interest
∗ general lumpability - no real practical use
Lumpability – p.14/14
Conclusion
•
for the properties of interest
∗ general lumpability - no real practical use
∗ ordinary lumpability - ”well behaved”
Lumpability – p.14/14
Conclusion
•
for the properties of interest
∗ general lumpability - no real practical use
∗ ordinary lumpability - ”well behaved”
∗ exact lumpability - good for rewards
Lumpability – p.14/14
Conclusion
•
for the properties of interest
∗ general lumpability - no real practical use
∗ ordinary lumpability - ”well behaved”
∗ exact lumpability - good for rewards
•
future aims
Lumpability – p.14/14
Conclusion
•
for the properties of interest
∗ general lumpability - no real practical use
∗ ordinary lumpability - ”well behaved”
∗ exact lumpability - good for rewards
•
future aims
∗ lumpability in simulation
Lumpability – p.14/14
Conclusion
•
for the properties of interest
∗ general lumpability - no real practical use
∗ ordinary lumpability - ”well behaved”
∗ exact lumpability - good for rewards
•
future aims
∗ lumpability in simulation
∗ lumpability and conditional reward
reduction together
Lumpability – p.14/14
Conclusion
•
for the properties of interest
∗ general lumpability - no real practical use
∗ ordinary lumpability - ”well behaved”
∗ exact lumpability - good for rewards
•
future aims
∗ lumpability in simulation
∗ lumpability and conditional reward
reduction together
∗ ...
Lumpability – p.14/14