Reusable Models for Timing and Liveness
Analysis of Middleware for Distributed RealTime and Embedded Systems
Venkita Subramonian
and Christopher Gill*
{venkita,cdgill}@cse.wustl.edu
Dept. of Computer Science and Engineering
Washington University, St. Louis, MO
César Sánchez
and Henny Sipma+
{cesar, henny}@cs.stanford.edu
Department of Computer Science
Stanford University, Stanford, CA
EMSOFT 2006
October 23-25, 2006, Seoul, Korea
*Research supported in part by NSF CAREER award CCF-0448562
+Research supported in part by NSF grants CCR-01-21403, CCR-02-20134,
CCR-02-09237, CNS-0411363, and CCF-0430102, ARO grant DAAD19-01-1-0723,
and NAVY/ONR contract N00014-03-1-0939
Thanks also to Joseph Sifakis, Marius Bozga and Iulian Ober for valuable advice about the IF tool set,
and to Doug Schmidt and the hundreds of ACE+TAO+CIAO developers/users world-wide
A Motivating Example: Real-Time Image Transmission


Soft real-time distributed
system (over wireless)
Many end-to-end activities
must be managed at once
» image (de-)compression,
transmission, analysis, display

image
source
client
side
transmission
middleware
adaptation
middleware
low (and variable) bandwidth radio link
Needs adaptive enforcement
Console
Many applications are like this
» Is correct reuse of middleware possible?
2
[Gill et al., “Integrated Adaptive QoS
Management in Middleware: An Empirical
Case Study” (RTAS ‘04)]
[Wang et al., “CAMRIT: Control-based
Adaptive Middleware for Real-time Image
Transmission” (RTAS ‘04)]
displays
virtual folder,
images
» Task scheduling, rate adaptation,
feedback control

server
side
Camera
Motivation for Formal Models of Middleware
…a main obstacle to the application of rigorous development
techniques is the lack of methodology for relating application software
and functional design to physical architecture and implementation…
Joseph Sifakis, “Modeling real-time systems-challenges and work directions”, EMSOFT 2001

The formal verification community needs accurate middleware models
ORB configurations have a substantial impact on performance, e.g.,
TAO's fastest concurrency model is thread-per-connection, rather than
thread pool…
…what's ultimately most important is to determine how things perform
for applications that are representative of actual user configurations,
rather than micro-benchmarks alone…
Doug Schmidt, ICE/CORBA newsgroup discussion 3/29/05

3
The real-time embedded systems community needs formal ways to
evaluate the semantics of different middleware configurations
A Brief Survey of Other Related Work

Vanderbilt University and UC Irvine
» GME, CoSMIC, PICML, DREAM

UC Berkeley
» Ptolemy, E-machine, Giotto

Kansas State University and University of Nebraska
» Bogor, Cadena

University of Pennsylvania
» VERSA

University of Brest
» Cheddar
4
ACE+TAO+CIAO COTS Middleware Architecture
CODES+ISSS (Tuesday, session A10)
system software engineering focus
A “Distributed System Software Stack”
TAO+CIAO
EMSOFT (Monday, session B1)
protocol design and proof focus
EMSOFT (Wednesday, session B9)
modeling and verification focus
5
ACE
Goal: Correct (Re-)use of DRE Middleware
Thesis: formal models of fine-grain middleware building blocks provide a principled
approach to analyze timing and liveness properties of middleware-based DRE systems
System Implementation
System Modeling
CADENA
DREAM
RMA
Ptolemy
GME
CoSMIC
UPPAAL
IF-toolkit
SPIN
Bogor
6
Formal
approach
to
composition
Reusable
software
building
blocks
Bridging the Gap:
composable, reusable
and executable timed
automata models of ACE
building blocks
Another Gap: Between High Level Models and Lower-Level
Consequences of their Configuration Choices

Modeling at a high-level of
abstraction is helpful
» Focus on nuances of the application

Low-level design choices matter too
Application Models
Implementation Platform
CCM
Collocation strategies
Buffer flushing strategies
Communication strategies
Connection strategies
Concurrency strategies
» May invalidate (implicit) assumptions

We focus on low-level frameworks
» Capture fundamental concurrency and
timing details of the middleware
7
CORBA
EJB
Pattern-based frameworks
OS
Model Architecture in IF for ACE



8
Network/OS layer: interprocess communication abstractions
Middleware layer: ACE pattern-oriented abstractions
Application layer: application-specific semantics within ACE event handlers
Modeling Threads

Challenge
» No native constructs for threads in model checkers that currently
support timed automata

Option 1: model all thread actions as a single automaton
» Suitable for high level modeling of application semantics

Option 2: model a thread as multiple interacting automata
» Interactions model the flow of control
» This option better abstracts the nuances of ACE-level mechanisms
Foo
9
Bar
output method_request
input method_request
input method_result
output method_result
Modeling Thread Scheduling Semantics (1/4)

Easy to achieve with one automaton
per thread
» Specify to model checker directly


Activity1
Update
Display
Activity2
Control
Flow Rate
E.g., using IF priority rules
More difficult with more than one
automaton per thread
» Thread of control spans interactions
among automata
» Need to express thread scheduling in
terms of execution control primitives
provided by the model checker
1 automaton per thread
2 automata per thread
Foo
10
prio_rule: pid1 < pid2
if pid1 instanceof Activity1
and pid2 instanceof Activity2
Bar
output m_req
input m_req
input m_result
output m_result
Modeling Thread Scheduling Semantics (2/4)

Solution
» Introduce a thread id that is propagated along automata interactions
» Thread id acts as an index to a storage area which holds each
thread’s scheduling parameters
Resulting Behavior
Foo1
Bar1
1
Prio=5
Foo2
1
2
Foo2
Prio=8
Bar2
Bar1
Bar1
Foo1
Bar2
2
Hint to the model checker
Give higher preference to the
automaton whose “thread” (pointed
to by thread id) has higher priority
11
Foo1
thread_schedule: pid1 < pid2
if
pid1 instanceof Foo1 and
pid2 instanceof Bar1 and
({Foo1}pid1).threadid <>
({Bar1}pid2).threadid and
({Thread}(({Foo1}pid1).threadid)).prio <
({Thread}(({Bar1}pid2).threadid)).prio )
Modeling Thread Scheduling Semantics (3/4)



What if two threads have the same priority?
In an actual implementation, run-to-completion
(SCHED_FIFO) may control the possible interleavings
How can we model run-to-completion?
Foo1
Foo2
Foo1
Foo3
Foo2
Foo3
Bar1
Bar2
Bar3
Bar1Foo1
Bar1
Bar2
Bar1
Foo1
Bar1
Bar2
Bar3
12
Bar3
How do we prune
out this space?
Foo2
Foo3
Modeling Thread Scheduling Semantics (4/4)

Solution
» Record id of
currently
executing thread
» Update when
executing actions
in each automaton
Current=nil
Foo1
Current=1
Foo2
Bar1
Current=2
Bar1
Current=1
Foo3
Bar1
Current=1
Bar1
Current=2
Bar2
Current=2
Bar3
Current=2
13
Hint to the model checker
Give higher preference to the
automaton whose thread is
the currently running thread.
Non-deterministic choice if
Current is nil
Problem: Over-constraining Concurrency
Hint to the model checker
Current=nil
Give higher preference to the
automaton whose thread is the
currently running thread. Nondeterministic choice if Current
is nil
Foo1
Current=1
Foo2
Bar1
Current=2
Bar1
Current=1
Foo3
Bar1
Current=1
Bar1
Current=2
Bar2
Bar3 always chosen to run
Current=2
Bar3
Current=2
Time progresses
14
Bar3
Current=2
Foo3
Solution: Idle Catcher Automaton
Current=2
Foo3, Bar3 blocked
at this point
Idle catcher runs

Current=nil
Time progress
» E.g., catcher thread in
middleware group scheduling
(RTAS ‘05)
Current=nil
Foo3
Bar3
Foo3 or Bar3 could be chosen
to run. Over-constraining eliminated
15
Key idea: lowest priority
“catcher” runs when all
others are blocked

Here, idle catcher automaton
» runs when all other automata
are idle (not enabled), but
before time progresses
» Resets value of current id to nil
Problem: Tractability
“maybe tomorrow?”

“go for an espresso”
“get coffee”
“in a minute” “right away”
Model checking can suffer from state space explosion
» State space reduction, live variable analysis can help
» But even good model checkers don’t fully solve this

16
Need to think of modeling as a design issue, too
» Does the model represent what it needs to represent?
» Can the model be re-factored to help the checker?
» Can domain specific information help avoid unnecessary checking?
Optimization 1: Leader Election

Leader/Followers concurrency
» Threads in a reactor thread pool
take turns waiting on the reactor
» One thread gets the “token” to
access the reactor - leader
» All other threads wait for the
token – followers



17
It does not matter which
thread gets selected as leader
in a threadpool
Model checker not aware of this
domain specific semantics
For BASIC-P protocol example,
saved factor of ~50 in state
space, and factor of ~20 in time
Token to access the
reactor is available
T1
T2
T2
T3
T3
T3
Prune this out
Optimization 2: System Initialization

S1 creates R
S2 creates R

S1
R1
S2
S2 creates R
R1
» E.g., 2 server automata,
each of which creates a
reactor automaton
S1 creates R
S1
R1
S2
R1
S2
R2
S1
R2


Prune this out
18
Similar idea, but different
technique
Iff ok to establish initial
object relations in any
order, can optimize away
Useful when modeling object
systems in model checkers
with dynamic automaton
creation capability (e.g., IF)
State space reduction
depends on application
» Factor of ~250 for a
deadlock scenario with 2
reactors and 3 threads in
each reactor
Verification of a Real-Time Gateway
Consumer1
Supplier1
Gateway
Supplier2
Consumer2
Consumer3
Consumer4


An exemplar of many realistic ACE-based applications
We modified the Gateway example to add new capabilities
» E.g., Real time, Reliability, Control-Push-Data-Pull
» Value added service in Gateway before forwarding a to consumer


E.g. Using consumer specific information to customize data stream
Different design, configuration choices become important
» E.g., number of threads, dispatch lanes, reply wait strategies
19
Model Checking/Experiment Configuration
Gateway
20
Period
100ms
S1
S2
10

20
C2
100ms
C3
50ms
C4
50ms
Relative
Deadline
Value-added
execution
(and its cost)

100ms
20
10
50ms
C1
Gateway is theoretically schedulable under RMA
» Utilization = 80%
» Schedulable utilization = 100% for harmonic periods
» Assumption – Messages from 50ms supplier is given
higher preference than 100ms supplier
ACE models let us verify scheduling enforcement
» IN THE ACTUAL SYSTEM IMPLEMENTATION
Deadline
Exec time
C1
100ms
20ms
C2
100ms
20ms
C3
50ms
10ms
C4
50ms
10ms
Real-time Gateway – Single Thread
Gateway
SupplierHandler
ConsumerHandler
ConsumerHandler
Supplier
SupplierHandler
ConsumerHandler
Consumer
ConsumerHandler
Reactor



21
Single reactor thread dispatches incoming events
I/O (reactor) thread same as dispatch thread
I/O thread responsible for value added service
Real-time Gateway – Dispatch Lanes
Gateway
SupplierHandler
ConsumerHandler
ConsumerHandler
Supplier
SupplierHandler
ConsumerHandler
Consumer
ConsumerHandler
Reactor




22
Single reactor thread again dispatches events to gateway handlers
I/O (reactor) thread puts message into dispatch lanes
Lane threads perform value added service, dispatch to consumers
DO QUEUES HELP OR HURT TIMING PREDICTABILITY?
Model/Actual Traces for Real-time Gateway
Execution in the context of lane threads
Execution in the context of reactor thread
Single threaded Gateway
S1,S2
Model
C1
C2
Actual
C1
C2
Time
Period
C3 C4
Deadline miss
for Consumer4
because of blocking
delay at reactor
C3 C4
Exec
time
Deadline
Gateway with dispatch lanes
S2
C3 C4
C1
C3 C4
C1
Time
10 20 30 40 50 60
100ms
20ms
100ms
C2
100ms
20ms
100ms
C3
50ms
10ms
50ms
C4
50ms
10ms
50ms
C2 C3 C4 C2
C2 C3 C4 C2
10 20 30 40 50 60 70 80 90 100
Expected execution timeline with RMS
C3, C4
C1
23
S1,S2
S2
C3 C4
C1
C1, C2, C3, C4
C2 C3 C4 C2
10 20 30 40 50 60 70
80 90 100
Concluding Remarks

Timed automata models of middleware building blocks

Domain specific model checking optimizations

We have run multiple case studies using these models

Future work
» Can be used to verify middleware concurrency and timing semantics
» Can achieve significant reductions in state space
» Allow comparison of model checking & software execution traces
» Demonstrate reusability and fidelity of our models
» Continue to develop reusable models and optimizations
» Incorporate domain information directly into model checking tools
» Extend our approach beyond real-time concerns


24
Model mechanism faults and failure modes
Power usage (yet another ~real valued “clock”)
Questions?
25
Backup Slides
26
Timing Traces for BASIC-P Protocol
EH11
R1
R2
EH31
Flow1
EH12
EH22
R1
R2
EH32
Model checking & actual timing traces
show BASIC-P protocol’s regulation of
threads’ use of resources (no deadlock)
EH13
R1
EH33
27
EH21
Flow2
EH23
R2
Flow3
BASIC-P Blocking Delay Comparison
Model Execution
Blocking delay
for Client2
28
Actual Execution
Blocking delay
for Client3
Overhead of ACE TP/DA reactor with BASIC-P
Negligible overhead with no DA protocol
Overhead increases linearly with
# of event handlers due
suspend/resume actions on
handlers at BASIC-P entry/exit
29