Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Towards A Real-Time Distributed Computing
Model
Heinrich Moser
Embedded Computing Systems Group
Vienna University of Technology
LVA Distributed Algorithms for Fault-tolerant Real-Time
Systems
1 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
“Real” event-driven execution
p
MAC
MAC
MAC
m1
m3
processing m1
q
MAC
processing m2
processing m3
m2
2 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
“Real” event-driven execution
p
MAC
MAC
MAC
m1
m3
processing m1
q
MAC
processing m2
processing m3
m2
• Complex timing behavior
2 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
“Real” event-driven execution
p
MAC
MAC
MAC
m1
m3
processing m1
q
MAC
processing m2
processing m3
m2
• Complex timing behavior
• Analysis requires simplification (arrival patterns)
2 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Classic “distributed computing” abstraction
p
q
m3
m1
m2
End-to-end delays ∈ [δ − , δ + ], zero processing time
3 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Classic “distributed computing” abstraction
p
q
MAC
MAC
m1
MAC
MAC
m3
m2
End-to-end delays ∈ [δ − , δ + ], zero processing time
3 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Classic “distributed computing” abstraction
p
q
m3
m1
m2
End-to-end delays ∈ [δ − , δ + ], zero processing time
• Simple model, well-studied, easy to analyze
3 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Classic “distributed computing” abstraction
p
q
m3
m1
m2
End-to-end delays ∈ [δ − , δ + ], zero processing time
• Simple model, well-studied, easy to analyze
• Downside: δ is end-to-end delay, including queuing delay
• A priori information about the message pattern needed
• Too much power to the adversary
3 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Our real-time computing model
p
q
m1
m3
processing m1
processing m2
processing m3
m2
Link delays ∈ [δ − , δ + ], job processing times ∈ [µ− , µ+ ]
4 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Our real-time computing model
p
q
MAC
MAC
m1
MAC
MAC
m3
processing m1
processing m2
processing m3
m2
Link delays ∈ [δ − , δ + ], job processing times ∈ [µ− , µ+ ]
4 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Our real-time computing model
p
q
m1
m3
processing m1
processing m2
processing m3
m2
Link delays ∈ [δ − , δ + ], job processing times ∈ [µ− , µ+ ]
• Explicitly models queuing and processing ⇒ no overly
pessimistic end-to-end delay bounds needed
4 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Our real-time computing model
p
q
m1
m3
processing m1
processing m2
processing m3
m2
Link delays ∈ [δ − , δ + ], job processing times ∈ [µ− , µ+ ]
• Explicitly models queuing and processing ⇒ no overly
pessimistic end-to-end delay bounds needed
−
+
+
• δ(`)
, δ(`)
, µ−
(`) , µ(`)
4 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Our real-time computing model
p
q
MAC
MAC
m1
MAC
MAC
m3
processing m1
processing m2
processing m3
m2
Link delays ∈ [δ − , δ + ], job processing times ∈ [µ− , µ+ ]
• Explicitly models queuing and processing ⇒ no overly
pessimistic end-to-end delay bounds needed
−
+
+
• δ(`)
, δ(`)
, µ−
(`) , µ(`)
4 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Our real-time computing model
p
q
m1
m3
processing m1
processing m2
processing m3
m2
Link delays ∈ [δ − , δ + ], job processing times ∈ [µ− , µ+ ]
• Explicitly models queuing and processing ⇒ no overly
pessimistic end-to-end delay bounds needed
−
+
+
• δ(`)
, δ(`)
, µ−
(`) , µ(`)
• Queuing delay ω, end-to-end delay ∆ (not system
parameters)
4 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Our real-time computing model
p
q
MAC
MAC
MAC
MAC
m1
m3
processing m1
processing m2
processing m3
m2
Link delays ∈ [δ − , δ + ], job processing times ∈ [µ− , µ+ ]
• Explicitly models queuing and processing ⇒ no overly
pessimistic end-to-end delay bounds needed
−
+
+
• δ(`)
, δ(`)
, µ−
(`) , µ(`)
• Queuing delay ω, end-to-end delay ∆ (not system
parameters)
4 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Our real-time computing model
p
q
m1
m3
processing m1
processing m2
processing m3
m2
Link delays ∈ [δ − , δ + ], job processing times ∈ [µ− , µ+ ]
• Explicitly models queuing and processing ⇒ no overly
pessimistic end-to-end delay bounds needed
−
+
+
• δ(`)
, δ(`)
, µ−
(`) , µ(`)
• Queuing delay ω, end-to-end delay ∆ (not system
parameters)
• Scheduling policy needed
4 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Formal model
m0 m4 m5
p
ac0
Classic computing model
ac2
m3
m1
q
ac1
m2
ac3
ac4
Execution = Sequence of actions
ex = (ac0 , ac1 , ac2 , ac3 , ac4 )
proc(ac2 ) = p
msg(ac2 ) = m0
time(ac2 ) = t
HC(ac2 ) = T
trans(ac2 ) = [s, m3 , m4 , s0 , m5 , s00 ]
5 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Formal model
Real-time computing model
p
m1
q
m3
processing m1
processing m2
processing m3
m2
Rt-run = Sequence of receive events and jobs
ru = (. . . , R1 , J1 , . . . , R2 , R3 , J2 , J3 )
proc(J2 ) = q
proc(R2 ) = q
msg(R2 ) = m2
time(R2 ) = t
msg(J2 ) = m2
begin(J2 ) = t 0
d(J2 ) = d
HC(J2 ) = T 0
trans(J2 ) = [s, s 0 ]
6 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Specifying problems
• Challenge: Problems should be defined independent of the
chosen model (classic or real-time)
7 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Specifying problems
• Challenge: Problems should be defined independent of the
chosen model (classic or real-time)
• Obvious approach: Predicate on sequence of actions/jobs
(common subset).
7 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Specifying problems
• Challenge: Problems should be defined independent of the
chosen model (classic or real-time)
• Obvious approach: Predicate on sequence of actions/jobs
(common subset).
• → aj-problems
7 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Specifying problems
• Challenge: Problems should be defined independent of the
chosen model (classic or real-time)
• Obvious approach: Predicate on sequence of actions/jobs
(common subset).
• → aj-problems
Example (Terminating Clock Synchronization)
•
Termination: All processors eventually terminate.
∀p : ∃e ∈ ex : is lastevent(e, p)
•
Agreement: After all processors have terminated, all processors have adjusted
clocks within γ of each other.
∀p, q : ∀ep , eq ∈ ex : (is lastevent(ep , p) ∧ is lastevent(eq , q)) ⇒ |HC(ep ) +
newstate(ep ).adj − begin(ep ) − (HC(eq ) + newstate(eq ).adj − begin(eq ))| ≤ γ
7 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Specifying problems
• Challenge: Problems should be defined independent of the
chosen model (classic or real-time)
• Obvious approach: Predicate on sequence of actions/jobs
(common subset).
• → aj-problems
Example (Terminating Clock Synchronization)
•
Termination: All processors eventually terminate.
∀p : ∃e ∈ ex : is lastevent(e, p)
•
Agreement: After all processors have terminated, all processors have adjusted
clocks within γ of each other.
∀p, q : ∀ep , eq ∈ ex : (is lastevent(ep , p) ∧ is lastevent(eq , q)) ⇒ |HC(ep ) +
newstate(ep ).adj − begin(ep ) − (HC(eq ) + newstate(eq ).adj − begin(eq ))| ≤ γ
•
Precondition I: Hardware clocks do not drift.
∀e, e0 ∈ ex : (proc(e) = proc(e0 )) ⇒ (HC(e) − HC(e0 ) = time(e) − time(e0 ))
•
Precondition II: Apart from the init messages, there are no input messages.
7 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Specifying problems
Drawbacks of aj-problems
Example (Mutual exclusion)
8 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Specifying problems
Drawbacks of aj-problems
Example (Mutual exclusion)
p
q
enters
exits
8 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Specifying problems
Drawbacks of aj-problems
Example (Mutual exclusion)
p
q
p
q
enters
exits
enters
exits
8 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Specifying problems
Drawbacks of aj-problems
Example (Mutual exclusion)
p
q
p
q
enters
exits
enters
exits
8 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
How to fix this?
9 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
How to fix this?
• Requirement: Well-defined state at every time t
9 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
How to fix this?
• Requirement: Well-defined state at every time t
• Problem: δ − = 0 and/or µ− = 0
9 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
How to fix this?
• Requirement: Well-defined state at every time t
• Problem: δ − = 0 and/or µ− = 0
• 1st try: state(p, t) is first or last state
9 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
How to fix this?
• Requirement: Well-defined state at every time t
• Problem: δ − = 0 and/or µ− = 0
• 1st try: state(p, t) is first or last state
• want to enter → enter → exit → want to enter
9 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
How to fix this?
• Requirement: Well-defined state at every time t
• Problem: δ − = 0 and/or µ− = 0
• 1st try: state(p, t) is first or last state
• want to enter → enter → exit → want to enter
• 2nd try: state(p, t) is set of all possible states
9 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
How to fix this?
• Requirement: Well-defined state at every time t
• Problem: δ − = 0 and/or µ− = 0
• 1st try: state(p, t) is first or last state
• want to enter → enter → exit → want to enter
• 2nd try: state(p, t) is set of all possible states
• exitp → enterq
9 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
How to fix this?
• Requirement: Well-defined state at every time t
• Problem: δ − = 0 and/or µ− = 0
• 1st try: state(p, t) is first or last state
• want to enter → enter → exit → want to enter
• 2nd try: state(p, t) is set of all possible states
• exitp → enterq
• Solution: Sequence of global states at time t
9 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Example of a st-trace
m1
[s, s 0 , m2 , s 00 ]
p
0
1
2
3
4
5
6
7
8
m2
q
10 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Example of a st-trace
m1
[s, s 0 , m2 , s 00 ]
p
0
1
2
3
4
5
6
7
8
m2
q
80 1
19
0 19
2
2
3 >
>
>
>
>
>
<B C
B s C=
B s C=
sC
B
B
C
B
C ,
···@
(process
:
2,
p,
m
),
,
·
·
·
1
@ sq A
@ s q A>
s q A>
>
>
>
>
>
>
:
:
;
;
{m1 }
{}
{}
80 19
80
1
0
19
3 >
3
4
>
>
>
>
>
>
<B 0 C>
<B 0 C
=
B s 0 C=
s C
s C
B
B
C
(transition : 3, p, s, s 0 ), B
(send
:
3,
p,
m
),
,
·
·
·
,
2
@
@
A
A
@
A
sq >
sq
sq
>
>
>
>
>
>
>
:
:
;
;
{}
{m2 }
{m2 }
80
9
1
4
>
>
>
>
<B 00 C
=
s C
(transition : 4, p, s 0 , s 00 ), B
·
·
·
@ sq A
>
>
>
>
:
;
{m2 }
8
>
>
<
0
10 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Example of a st-trace
m1
[s, s 0 , m2 , s 00 ]
p
0
1
2
3
4
5
6
7
8
m2
q
80 1
19
0 19
2
2
3 >
>
>
>
>
>
<B C
B s C=
B s C=
sC
B
C
B
B
C ,
···@
,
(process
:
2,
p,
m
),
·
·
·
1
@ sq A
@ s q A>
s q A>
>
>
>
>
>
>
:
:
;
;
{m1 }
{}
{}
80 19
80
1
0
19
3 >
3
4
>
>
>
>
>
>
<B 0 C>
=
<B 0 C
B s 0 C=
s C
s C
B
B
C
(transition : 3, p, s, s 0 ), B
,
(send
:
3,
p,
m
),
·
·
·
,
2
@
A
@
A
@
A
sq >
sq
sq
>
>
>
>
>
>
>
:
;
:
;
{}
{m2 }
{m2 }
80
9
1
4
>
>
>
>
<B 00 C
=
s C
(transition : 4, p, s 0 , s 00 ), B
·
·
·
@ sq A
>
>
>
>
:
;
{m2 }
8
>
>
<
0
10 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Example of a st-trace
m1
[s, s 0 , m2 , s 00 ]
p
0
1
2
3
4
5
6
7
8
m2
q
80 1
19
0 19
2
2
3 >
>
>
>
>
>
<B C
B s C=
B s C=
s
C , (process : 2, p, m1 ), B C · · · B C ,
···B
@ sq A
@ s q A>
@ s q A>
>
>
>
>
>
>
:
:
;
;
{}
{m1 }
{}
80
80 19
19
0
1
4
3
3 >
>
>
>
>
>
>
<B 0 C
=
<B 0 C>
B s 0 C=
s C
s C
0
C
B
B
B
,
···@
(transition : 3, p, s, s ), @ A , (send : 3, p, m2 ), @
A
A
sq
sq
sq >
>
>
>
>
>
>
>
;
:
;
:
{m2 }
{m2 }
{}
80
9
1
4
>
>
>
>
<B 00 C
=
s C
(transition : 4, p, s 0 , s 00 ), B
·
·
·
@ sq A
>
>
>
>
:
;
{m2 }
8
>
>
<
0
→ st-problems
10 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
st-problems
Example (Terminating Clock Synchronization)
• Termination: All processors eventually terminate.
∃g ∈ gstates(tr ) : is finalstate(g)
• Agreement: After all processors have terminated, all
processors have adjusted clocks within γ of each other.
∀g ∈ gstates(tr ) : is finalstate(g) ⇒ (∀p, q :
|ACp (g) − ACq (g)| ≤ γ)
• Precondition I: Hardware clocks do not drift:
∀p, t, t 0 : HCp (t) − HCp (t 0 ) = t − t 0
• Precondition II: Apart from the init messages, there are no
input messages.
11 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Real-time alg. on classic system
Simulation algorithm
idle
busy
wait for message
process message
wait for timeout
enqueue incoming messages
Yes
queue empty?
No
12 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Real-time alg. on classic system
Proof
For each admissible execution
ex of S µ− ,A :
m1
m2
fin.proc.
fin.proc.
• create a corresponding admissible rt-run ru of A (satisfying P).
m1
m2
• Every st-trace of ex is a (simulation-invariant V-extension of a)
st-trace of ru ⇒ ex satisfies PV> .
⇒ S µ− ,A solves PV> in s.
13 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Real-time alg. on classic system
Proof
For each admissible execution
ex of S µ− ,A :
m1
m2
fin.proc.
fin.proc.
• create a corresponding admissible rt-run ru of A (satisfying P).
m1
m2
• Every st-trace of ex is a (simulation-invariant V-extension of a)
st-trace of ru ⇒ ex satisfies PV> .
⇒ S µ− ,A solves PV> in s.
⇒ Precision (1 − 1n )(δ + − δ − ) still optimal for clock synchronization
13 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Classic alg. on real-time system
m
∆(m)
Circular dependency
between end-to-end delays
and message pattern:
´
`
[∆− , ∆+ ] = F n, [δ − , δ + ], [µ− , µ+ ], A([δ − , δ + ])
[δ − , δ + ] = [∆− , ∆+ ]
m
⇒ Real-time scheduling
analysis necessary
δ(m)
14 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Classic alg. on real-time system
Proof
For each s-admissible rt-run ru
of SA :
• create a corresponding
s-admissible execution ex
of A (satisfying P).
• Every st-trace of ru is a
µ+ -shuffle of a st-trace of
ex ⇒ ru satisfies Pµ∗+ .
⇒ SA solves Pµ∗+ in s.
m
∆(m)
m
δ(m)
15 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Classic alg. on
real-time system
+
µ -shuffle?
Example (τ -gap mutual exclusion)
µ+ = 3 seconds
If classic model
algorithm A solves
3-second gap mutual
exclusion
then real-time model
algorithm SA solves
0-second gap mutual
exclusion.
exits
p
q
enters
> 3s
exits
p
q
enters
> 0s
16 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Classic alg. on
real-time system
+
µ -shuffle?
Example (τ -gap mutual exclusion)
µ+ = 3 seconds
If classic model
algorithm A solves
3-second gap mutual
exclusion
then real-time model
algorithm SA solves
0-second gap mutual
exclusion.
exits
p
q
enters
> 3s
exits
p
q
enters
> 0s
Example (Causal mutual exclusion)
compatible problem
16 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Terminating drift-free clock synchronization
Classic model results:
(Lundelius and Lynch, 1984)
• Optimal precision (tight): (1 − n1 )(δ + − δ − )
• Worst-case time complexity: O(1)
17 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Terminating drift-free clock synchronization
Optimal precision
Classic model: (1 − n1 )(δ + − δ − )
Real-time model: (1 − n1 )(δ + − δ − )
• Lower bound: Transformation
• Upper bound: “Stretched” Lundelius-Lynch algorithm
(avoid queuing)
18 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Terminating drift-free clock synchronization
Worst-case time complexity (1/2)
Classic model: O(1)
Real-time model:
• Within γ < (δ + − δ − ): O(n)
• Within γ ≥ (δ + − δ − ):
• If broadcasting is possible in constant time: O(1)
+
+
• Otherwise: lower bound of minM
k =0 (k · µ(0) + δ(M−k ) ),
√
M = d 2c+1 ne.
⇒ Classic model abstracts too much away!
19 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Terminating drift-free clock synchronization
Worst-case time complexity (2/2)
Lemma
When synchronizing clocks to within γ = c · (δ + − δ − ), there is
a real-time run whose message graph has diameter 2c or less.
20 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Terminating drift-free clock synchronization
Worst-case time complexity (2/2)
Lemma
When synchronizing clocks to within γ = c · (δ + − δ − ), there is
a real-time run whose message graph has diameter 2c or less.
Proof.
By contradiction. Assume diameter > 2c and use shifting
argument.
p
dist. 1
dist. 2
20 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Terminating drift-free clock synchronization
Worst-case time complexity (2/2)
Lemma
When synchronizing clocks to within γ = c · (δ + − δ − ), there is
a real-time run whose message graph has diameter 2c or less.
Proof.
By contradiction. Assume diameter > 2c and use shifting
argument.
p
dist. 1
dist. 2
20 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Terminating drift-free clock synchronization
Worst-case time complexity (2/2)
Lemma
When synchronizing clocks to within γ = c · (δ + − δ − ), there is
a real-time run whose message graph has diameter 2c or less.
For c < 1: message complexity O(n 2 ) ⇒ time complexity O(n).
Otherwise:
Lemma
In an undirected graph with n > 2 nodes and diameter
D or
√
D+1
n.
less, there is at least one node with degree ≥
√
2c+1
⇒ At least one processor exchanging M = d
ne messages
+
+
+
δ
).
⇒ Time complexity: minM
(k
·
µ
k =0
(0)
(M−k )
20 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Terminating drift-free clock synchronization
Algorithms
Precision
(1 − n1 )(δ + − δ − )
(δ + − δ − )
Message complexity
O(n2 )
O(n)
Time complexity
O(n)
O(1) or O(n)
21 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Summary (1/3)
p
MAC
MAC
MAC
m1
MAC
“Real” event-driven execution
m3
processing m1
q
processing m2
processing m3
m2
p
q
p
q
m3
m1
Classic “distributed computing” abstraction (executions)
m2
m1
m3
processing m1
m2
processing m2
processing m3
Our real-time computing model (rtruns)
22 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Summary (2/3)
Formal model
• Sequence of receive events and jobs
(vs. sequence of actions in the classic model)
• State transition traces (st-traces): alternating events and
sets of global states
Problem specification
• aj-problems: straightforward
• st-problems: powerful
23 / 24
Introduction
RT Computing Model
Problems
Transformations
Clocksync
Summary
Summary (3/3)
Transformations
• Real-time alg. on classic system
• simulation algorithm
• Classic alg. on real-time system
• circular dependency
• µ+ -shuffle
Terminating drift-free clock synchronization
• Best precision: (1 − n1 )(δ + − δ − )
(confirmed)
• Best precision
⇒ msg. complexity Ω(n2 ), time complexity Ω(n) (vs. O(1))
• Best msg. complexity O(n)
⇒ precision not better than (δ + − δ − )
24 / 24