Deciding the value 1 problem for reachability
in 1-clock Decision Stochastic Timed Automata
Nathalie Bertrand1
1
2
Thomas Brihaye2
Inria/IRISA, Rennes, France
Université Mons, Mons, Belgium
3
CNRS/IRISA, Rennes, France
QEST’14 - 9/9 - Firenze
Blaise Genest3
Non-deterministic and probabilistic timed systems
Two approaches to combine probability, non-determinism and time:
I
I
Probabilistic Timed Automata (à la PRISM)
[KNSS-arts99]
I
Time-delays are chosen non-deterministically,
I
Edges are according to discrete probability distributions.
Decision Stochastic Timed Automata (ext. of CTMDP)
[BS-formats12]
Nathalie Bertrand
I
Time-delays are chosen via continuous probability distributions,
I
Edges are chosen non-deterministically.
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 2/21
Known results on (decision) stochastic timed automata
Stochastic timed automata (STA)
I
The almost-sure model-checking of LTL is decidable
I
I
I
on 1-clock STA.
[BBBBG-lics08]
on reactive n-clock STA.
[BBJM-qest12]
Open problem decidability of the almost-sure reachability
problem on general 2-clock STA.
Decision Stochastic Timed Automata (DSTA)
I
Existence of an optimal scheduler for the time-bounded
reachability problem on reactive DSTA.
[BS-formats12]
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 3/21
Known results on (decision) stochastic timed automata
Stochastic timed automata (STA)
I
The almost-sure model-checking of LTL is decidable
I
I
I
on 1-clock STA.
[BBBBG-lics08]
on reactive n-clock STA.
[BBJM-qest12]
Open problem decidability of the almost-sure reachability
problem on general 2-clock STA.
Decision Stochastic Timed Automata (DSTA)
I
Existence of an optimal scheduler for the time-bounded
reachability problem on reactive DSTA.
[BS-formats12]
This talk: reachability problem on 1-clock DSTA.
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 3/21
Outline of the talk
Introduction
Decision Stochastic Timed Automata
Solving the value 1 problem
The limit corner-point MDP
Correctness of the limit corner-point MDP
Conclusion
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 4/21
One-clock timed automata
≥1
e 2, x
e0 , x := 0
e1
`0
`1
x ≤1
x ≤2
e3 , x
≤1
one-clock timed automaton: A = (L, `0 , E , I)
,
e4
/
e5
Example of execution:
0
1
2
(`0 , 0) −
→ (`0 , .7) −→
(`0 , 0) −
→ (`0 , .8) −→
(`1 , .8) −
→ (`0 , 1.1) −→
,
.7
Nathalie Bertrand
e
.8
e
Value 1 problem for Decision Stochastic Timed Automata
.3
e
QEST’14 – Florence – 9 sept., 5/21
One-clock Decision Stochastic Timed Automaton
unif
e0 , x := 0
`0
x ≤1
unif
e1
≥1
e 2, x
`1
x ≤2
e3 , x
≤1
decision stochastic timed automaton: (A, µ) where
,
/
I
A = (L, `0 , E , I) is a one-clock timed automaton
I
µ = (µ`,ν ) is a family of distributions
µ`,ν : distribution over potential delays from state (`, ν)
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
e4
e5
QEST’14 – Florence – 9 sept., 6/21
Semantics of DSTA
unif
e0 , x := 0
e1
`0
≥1
e 2, x
unif
`1
x ≤1
x ≤2
e3 , x
≤1
Infinite state MDP: from state s
I a delay τ is randomly chosen according to µs ;
I the player chooses an edge e enabled in s + τ .
,
/
e4
e5
0
1
2
h`0 , 0i −
→ [`0 , .7] −→
h`0 , 0i −
→ [`0 , .8] −→
h`1 , .8i −
→ [`0 , 1.1] −→
,
.7
e
.8
e
.3
e
strategy σ: in [ ]-states dictates which edge to choose
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 7/21
Optimal positional strategies
unif
e0 , x := 0
e1
`0
x ≤1
`1
x ≤2
0
(
e0
σε (`0 , ν) =
e1
1−ε/1
if ν < 1 − ε
if ν ≥ 1 − ε
,
;
0 ,0)
P(`
(A, µ) |= 3, ≥
σε
Nathalie Bertrand
≥1
e 2, x
unif
e3 , x
≤1
2
(
σε (`1 , ν) =
e2
e3
,
/
e4
e5
if ν ≥ 1
if ν < 1
1
≥ 1 − ε.
1+ε
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 8/21
Optimal positional strategies
unif
e0 , x := 0
e1
`0
x ≤1
`1
x ≤2
0
(
e0
σε (`0 , ν) =
e1
≥1
e 2, x
unif
1−ε/1
if ν < 1 − ε
if ν ≥ 1 − ε
,
;
0 ,0)
P(`
(A, µ) |= 3, ≥
σε
e3 , x
≤1
2
(
σε (`1 , ν) =
e2
e3
,
/
e4
e5
if ν ≥ 1
if ν < 1
1
≥ 1 − ε.
1+ε
ε-optimal strategies are not region-uniform
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 8/21
Almost-sure vs Limit-sure
DSTA (A, µ), target set
I
, ⊆ L, and initial state s ∈ S
, is almost-surely reachable from s
∃σ
I
Psσ (A, µ) |= 3, = 1.
, is limit-surely reachable from s
∀ε > 0 ∃σ
Nathalie Bertrand
if
if
Psσ (A, µ) |= 3, > 1 − ε.
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 9/21
Almost-sure 6= Limit-sure
unif
e0 , x := 0
`0
x ≤1
I
I
unif
e1
≥1
e 2, x
`1
x ≤2
e3 , x
≤1
, is not almost-surely reachable from (`0 , 0),
,
/
e4
e5
, is limit-surely reachable from (`0 , 0).
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 10/21
Our contribution
Probability 1 problem
Input: A DSTA (A, µ), a target set , ⊆ L and a state s ∈ S.
Question: Is , almost-surely reachable from s?
Value 1 problem
Input: A DSTA (A, µ), a target set , ⊆ L and a state s ∈ S.
Question: Is , limit-surely reachable from s?
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 11/21
Our contribution
Probability 1 problem
Input: A DSTA (A, µ), a target set , ⊆ L and a state s ∈ S.
Question: Is , almost-surely reachable from s?
Value 1 problem
Input: A DSTA (A, µ), a target set , ⊆ L and a state s ∈ S.
Question: Is , limit-surely reachable from s?
Main Result
The probability 1 and value 1 problems are decidable in polynomial
time for one-clock decision stochastic timed automata.
For value 1, ε-optimal strategies are not region-uniform.
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 11/21
Introduction
Decision Stochastic Timed Automata
Solving the value 1 problem
The limit corner-point MDP
Correctness of the limit corner-point MDP
Conclusion
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 12/21
Solving the value 1 problem - Key idea
From a DSTA (A, µ), we build a finite MDP Acp such that
, is limit-surely reachable from s0 in (A, µ)
if and only if
, is almost-surely reachable from s0 in Acp .
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 13/21
Introduction
Decision Stochastic Timed Automata
Solving the value 1 problem
The limit corner-point MDP
Correctness of the limit corner-point MDP
Conclusion
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 14/21
Limit corner-point region MDP
<1
<x
e 4 ,0
e1 ,0<x<1
e0 ,x<1
`0
x:=0
`1
e3 ,0<x<1
e2 ,1<x<2 ; x:=0
`0 ,{0}
`0 ,(0,1)
`0 ,{0}
`0 ,(0,1)
`2
e5 ,1
≤x
<2
/
,
/
,
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 15/21
Limit corner-point region MDP
<1
<x
e 4 ,0
e1 ,0<x<1
e0 ,x<1
`0
x:=0
`1
e3 ,0<x<1
e2 ,1<x<2 ; x:=0
`0 ,(0,1)
`0 ,{0}
`0 ,(0,1)
`2
e5 ,1
≤x
<2
/
,
/
,
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 15/21
Limit corner-point region MDP
<1
<x
e 4 ,0
e1 ,0<x<1
e0 ,x<1
`0
x:=0
`1
e3 ,0<x<1
e2 ,1<x<2 ; x:=0
e0
`0 ,(0,1)
`0 ,{0}
`0 ,(0,1)
e0
Nathalie Bertrand
`2
e5 ,1
≤x
<2
/
,
/
,
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 15/21
Limit corner-point region MDP
<1
<x
e 4 ,0
e1 ,0<x<1
e0 ,x<1
`0
x:=0
`1
e3 ,0<x<1
`2
e5 ,1
≤x
e2 ,1<x<2 ; x:=0
e0
`0 ,(0,1)
`0 ,{0}
`0 ,(0,1)
e1
e1
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
e3
e3
<2
/
,
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
e0
`1 ,(1,2)
e2
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
`2 ,(1,2)
e5
/
,
QEST’14 – Florence – 9 sept., 15/21
Limit corner-point region MDP
<1
<x
e 4 ,0
e1 ,0<x<1
e0 ,x<1
`0
x:=0
`1
e3 ,0<x<1
`2
e5 ,1
≤x
e2 ,1<x<2 ; x:=0
e0
`0 ,(0,1)
`0 ,{0}
`0 ,(0,1)
e1
e1
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
e3
e3
<2
/
,
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
e0
`1 ,(1,2)
`2 ,(1,2)
e2
, is not almost-surely reachable from
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
e5
/
,
h`0 ,{0}i
QEST’14 – Florence – 9 sept., 15/21
Limit corner-point region MDP
<1
<x
e 4 ,0
e1 ,0<x<1
e0 ,x<1
`0
x:=0
`1
e3 ,0<x<1
`2
e5 ,1
≤x
e2 ,1<x<2 ; x:=0
<2
need to take into account limit behaviours.
e0
`0 ,(0,1)
`0 ,{0}
`0 ,(0,1)
e1
e1
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
e3
e3
/
,
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
e0
`1 ,(1,2)
`2 ,(1,2)
e2
, is not almost-surely reachable from
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
e5
/
,
h`0 ,{0}i
QEST’14 – Florence – 9 sept., 15/21
Limit corner-point region MDP
<1
<x
e 4 ,0
e1 ,0<x<1
e0 ,x<1
`0
x:=0
`1
e3 ,0<x<1
`2
e5 ,1
≤x
e2 ,1<x<2 ; x:=0
<2
need to take into account limit behaviours.
e0
e1
`0 ,(0,1)
`0 ,{0}
e1
`0 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,{1}
`1 ,(1,2)
e3
e3
/
,
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,{1}
`2 ,(1,2)
e0
e1limit
e3limit
e2
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
e5
/
,
QEST’14 – Florence – 9 sept., 15/21
Limit corner-point region MDP
<1
<x
e 4 ,0
e1 ,0<x<1
e0 ,x<1
`0
x:=0
`1
e3 ,0<x<1
`2
e5 ,1
≤x
e2 ,1<x<2 ; x:=0
<2
e0
e1
`0 ,(0,1)
`0 ,{0}
e1
`0 ,(0,1)
e0
e1limit
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,{1}
e3
e3
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
limit
`1 ,(1,2) e3
`2 ,{1}
`2 ,(1,2)
, is almost-surely reachable from
Value 1 problem for Decision Stochastic Timed Automata
,
`2 ,(0,1)
e2
Nathalie Bertrand
/
e5
/
,
(`0 ,{0})
QEST’14 – Florence – 9 sept., 15/21
Introduction
Decision Stochastic Timed Automata
Solving the value 1 problem
The limit corner-point MDP
Correctness of the limit corner-point MDP
Conclusion
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 16/21
limit-sure in DSTA ⇒ almost-sure in MDP
Proposition
If , is not almost-surely reachable from s0 in Acp ,
then , is not limit-surely reachable from s0 in (A, µ).
Proof idea
I
if [`, (c, c + 1)] is losing in MDP, then the value is uniformely
bounded away from 1 for all states in (`, (c, c + 1));
I
else, if [`, (c, c + 1)] is losing in MDP, then for all states in
(`, (c, c + 1)) the value is bounded away from 1.
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 17/21
almost-sure in MDP ⇒ limit-sure in DSTA
Proposition
If , is almost-surely reachable from s0 in Acp ,
then , is limit-surely reachable from s0 in (A, µ).
Proof idea
I
Key: strategies that are positional and uniform inside
(`, (c, c + T )) and (`, (c + T , c + 1)) suffice
I
from an almost-surely winning strategy σcp in Acp
Nathalie Bertrand
I
build an abstract family of strategies σT in A such that:
(
σcp (`, (c, c + 1)) if ν ∈ (c, c + T )
σT (`, ν) =
σcp (`, (c, c + 1)) otherwise
I
given ε, tune T to ensure probability ≥ 1 − ε
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 18/21
Solving the limit corner-point MDP
e0
e1
`0 ,(0,1)
`0 ,{0}
e1
`0 ,(0,1)
e0
e1limit
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(1,2)
`1 ,(1,2)
e3
e3
e3limit
e2
Computing winning states
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(1,2)
`2 ,(1,2)
e5
/
,
QEST’14 – Florence – 9 sept., 19/21
Solving the limit corner-point MDP
e0
e1
`0 ,(0,1)
`0 ,{0}
e1
`0 ,(0,1)
e0
e1limit
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(1,2)
`1 ,(1,2)
e3
e3
e3limit
e2
Computing winning states
I
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(1,2)
`2 ,(1,2)
e5
/
,
states that cannot reach , are bad
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 19/21
Solving the limit corner-point MDP
e0
e1
`0 ,(0,1)
`0 ,{0}
e1
`0 ,(0,1)
e0
e1limit
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(1,2)
`1 ,(1,2)
e3
e3
e3limit
e2
Computing winning states
I
I
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(1,2)
`2 ,(1,2)
e5
/
,
states that cannot reach , are bad
actions that lead to bad states with > 0 probability are unsafe
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 19/21
Solving the limit corner-point MDP
e0
e1
`0 ,(0,1)
`0 ,{0}
e1
`0 ,(0,1)
e0
e1limit
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(1,2)
`1 ,(1,2)
e3
e3
e3limit
e2
Computing winning states
I
I
I
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(1,2)
`2 ,(1,2)
e5
/
,
states that cannot reach , are bad
actions that lead to bad states with > 0 probability are unsafe
states that only have unsafe actions are bad
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 19/21
Solving the limit corner-point MDP
e0
e1
`0 ,(0,1)
`0 ,{0}
e1
`0 ,(0,1)
e0
e1limit
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(1,2)
`1 ,(1,2)
e3
e3
e3limit
e2
Computing winning states
I
I
I
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(1,2)
`2 ,(1,2)
e5
/
,
states that cannot reach , are bad
actions that lead to bad states with > 0 probability are unsafe
states that only have unsafe actions are bad
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 19/21
Solving the limit corner-point MDP
e0
e1
`0 ,(0,1)
`0 ,{0}
e1
`0 ,(0,1)
e0
e1limit
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(1,2)
`1 ,(1,2)
e3
e3
e3limit
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(1,2)
`2 ,(1,2)
e2
Computing winning states
I
I
I
e5
/
,
states that cannot reach , are bad
actions that lead to bad states with > 0 probability are unsafe
states that only have unsafe actions are bad
From each winning state: safe edge towards ,
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 19/21
Abstract family of strategies
e0
e1
`0 ,(0,1)
`0 ,{0}
e1
`0 ,(0,1)
e0
e1limit
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(0,1)
`1 ,(1,2)
`1 ,(1,2)
e3
e3
e3limit
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(0,1)
`2 ,(0,1) e4
`2 ,(1,2)
`2 ,(1,2)
e5
e2
e1 ,0<x<1
e0 ,x<1
`0
1
x<
0<
e 4,
e3 ,0<x<1
`1
x:=0
e2 ,1<x<2 ; x:=0
`2
e5 ,
1≤
x<
2
/
,
I
green edges are safe
I
red edges are losing
I
orange edges are risky
; chosen only when “close enough” to the right corner
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
/
,
Details
QEST’14 – Florence – 9 sept., 20/21
Conclusion
Contributions
I
PTIME algorithms on one-clock DSTA for
I
I
I
the almost-sure reachability problem, and
the limit-sure reachability problem
non trivial ε-optimal strategies
I
I
Nathalie Bertrand
not region uniform
cutpoint set according to “distance” to
Value 1 problem for Decision Stochastic Timed Automata
,
QEST’14 – Florence – 9 sept., 21/21
Conclusion
Contributions
I
PTIME algorithms on one-clock DSTA for
I
I
I
the almost-sure reachability problem, and
the limit-sure reachability problem
non trivial ε-optimal strategies
I
I
not region uniform
cutpoint set according to “distance” to
Ongoing and future work
,
I
Value 1 for other properties, and larger class of DSTA.
I
Towards quantitative analysis: value approximation.
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 21/21
Construction of the cutpoint function T
A simple case
unif
e0 ,x:=0
unif
e1
`0
`1
x≤1
0
x≤2
1−ε 1
e0
e1
T (`0 ,(0,1))=1−ε
(
σT (`0 , ν) =
Nathalie Bertrand
e0
e1
if ν < 1 − ε
if ν ≥ 1 − ε
1
e2, x ≥
;
e3 , x ≤
1
,
/
Psσ0T (A, µ) |= 3, ≥ 1 − ε.
Value 1 problem for Decision Stochastic Timed Automata
QEST’14 – Florence – 9 sept., 22/21
Construction of the cutpoint function T
A not that simple case
0
1−ε 1
e0
0
e7 ,x≤1
e1
e1 ,x≤1
`0
e7
e2 ,1≤x≤2
`1
x:=0
e5
e0 ,x≤1
1−ε 1
,x
e3
e3 ,x≤1
`2
≤
1
`3
,x
/
PσT ([`0 , 0] |= 3,) < 2/3
Nathalie Bertrand
≤
1
e6 ,1≤x≤2
e4
Value 1 problem for Decision Stochastic Timed Automata
,
QEST’14 – Florence – 9 sept., 23/21
Construction of the cutpoint function T
A not that simple case
0
1−ε2 1
e0
0
e7 ,x≤1
e1
e1 ,x≤1
`0
e7
e2 ,1≤x≤2
`1
x:=0
e5
e0 ,x≤1
1−ε 1
,x
e3
e3 ,x≤1
`2
≤
1
`3
,x
e4
/
≤
1
e6 ,1≤x≤2
,
, is limit-surely reachable from [`0 , 0] using involved T .
Nathalie Bertrand
Value 1 problem for Decision Stochastic Timed Automata
Back to main
QEST’14 – Florence – 9 sept., 23/21