The New Dyninst Event Model
James Waskiewicz
University of Maryland
Motivation: the series



Several years in the making:
Paradyn separation requires…
DyninstAPI support for threads
– Among other things

Supporting threads (a la Paradyn) requires…
– Asynchronous event handling, which requires…

Solving the multi-event-source problem
– Eg. Listen for and handle events coming from
• Socket / file descriptor
• System call ( wait() )
– Solution: use threads
• Just a can of worms …
• Or Pandora’s box?
University of Maryland
Overall Design Concerns

Keep thread stuff out of the way
– Don’t want to “pollute” dyninst code with
synchronization primitives
– Dyninst is already hard to fix when broken
• Keep threading issues modular

Try to keep existing code threadcontext free
– In other words, most dyninst code should be
able to run successfully no matter what thread
executes it.
– There can be some tricky situations here
(inferior RPCs for example)
– Obvious exceptions in signal handling
University of Maryland
The Plan: The 3 thread model

3 threads
– (1) User Interface Thread
• API user’s mutator
– (1) Synchronous Event Thread
• Eg. Loop on waitpid()
– (1) Asynchronous Event Thread
• Eg. Loop on select()

And a mailbox
– Hide internal threads by forcing non UI
threads to “mail” events to the UI thread.
• Eg. ensure that user-supplied callbacks are
only executed on the UI thread.
University of Maryland
The Plan: The 3 thread model
UI Thread
Synchronous
Event Thread
Mutatee 1
waitpid()
poll()
Mutatee 2
Asynchronous
Event Thread
select()
Mutatee N
University of Maryland
But there’s a problem (linux)

Ptrace() may only be called by one
thread, period.
– But we really want all threads to be able to
issue ptrace() commands
• Process::pause()
• Write/readDataSpace()

Solution: Add another thread
– Solely responsible for accepting and handling
requests for calling ptrace()
– And sometimes waitpid() and fork() …
University of Maryland
The Plan: The 4 thread model
UI Thread
Ptracer
(DBI)
Thread
Synchronous
Event Thread
Mutatee 1
waitpid()
poll()
Mutatee 2
Asynchronous
Event Thread
select()
Mutatee N
University of Maryland
But there’s another problem (windows)

Restrictive OS level debugging API
– The thread that starts the mutatee process is
the only thread that is allowed to call
WaitNextDebugEvent()

Solution: add N threads !
– N = number of mutatees

This is actually a good idea
– One “listener” thread per process
• Elegant in terms of code modularity
• A nice simple concept: a thread that starts
a process and receives events from it until
termination/detach
• Handles events as they occur.
University of Maryland
The Plan: The (n+3) thread model
UI Thread
Ptracer
(DBI)
Thread
Synchronous
Event Thread 1
waitpid()
Poll()
Synchronous
Event Thread 2
waitpid()
Poll()
Synchronous
Event Thread N
waitpid()
Poll()
Asynchronous
Event Thread
select()
University of Maryland
Mutatee 1
Mutatee 2
Mutatee N
But there’s another problem: recursion

Dyninst is full of recursive event patterns
– ie. An event that causes another event, and needs to wait for
its completion

Best solution will involve encapsulating this behavior into a
“natural” structure
– Either that or re-architect many functional elements of dyninst
– And don’t forget: add more threads!

Solution: decouple event handling concept into:
– Event generation
• One dedicated thread (per process)
– Event handling
• Multiple possible threads (thread pool)
• Number of threads depends on recursion depth

This is a bit nasty
– Could probably be done using the mailbox’s recursive callback
features, but…
• They’re complicated too, and need more thought
University of Maryland
The Plan: The (2n+3+x) thread model
UI Thread
Ptracer
(DBI)
Thread
Signal Generator
Thread 1
SH
SH
SH
Signal Generator
Thread 1
SH
SH
SH
Asynchronous
Event Thread
select()
University of Maryland
Mutatee 2
SH
Signal Generator
Thread 1
SH
Mutatee 1
SH
Mutatee N
A simple example: full stop
UI Thread issues stopExecution()
– Sends SIGSTOP to mutatee
– Blocks inside waitForStop()
kill (SIGSTOP)
UI Thread
block
Mutatee
Wait for event:
evtProcessStop
Event Gate

Signal
Generator
Thread
Signal
Handler
Thread
University of Maryland
A simple example: full stop
Mutatee Stops
–
–
Notifes parent process of SIGSTOP
This is received by SignalGenerator, which is listening for mutatee events
kill (SIGSTOP)
UI Thread
block
Mutatee
Wait for event:
evtProcessStop
Event Gate

Signal
Generator
Thread
Signal
Handler
Thread
University of Maryland
SIGSTOP
A simple example: full stop
Signal Generator decodes SIGSTOP and assigns event
– Creates an Event with platform independent type evtProcessStop
– Assigns Event to available SignalHandler Thread
kill (SIGSTOP)
UI Thread
block
Mutatee
Wait for event:
evtProcessStop
Signal
Generator
Thread
SIGSTOP
evtProcessStop
Event Gate

Signal
Handler
Thread
University of Maryland
A simple example: full stop
Signal Hander handles evtProcessStop
–
–
Platform independent code for handling evtProcessStop
Signal waiting EventGates that evtProcessStop happened
kill (SIGSTOP)
UI Thread
block
Mutatee
Wait for event:
evtProcessStop
Signal
Generator
Thread
SIGSTOP
evtProcessStop
Event Gate

Signal event:
evtProcessStop
Signal
Handler
Thread
University of Maryland
A simple example: full stop
Event Gate receives target event
– Release UI Thread from block
kill (SIGSTOP)
UI Thread
block
Mutatee
Wait for event:
evtProcessStop
Signal
Generator
Thread
SIGSTOP
evtProcessStop
resume
Event Gate

Signal event:
evtProcessStop
Signal
Handler
Thread
University of Maryland
A (slightly) more complex example

User registers a DynLibrary Callback
– When a library is loaded, insert some instrumentation
UI Thread
CB Manager
register
callback
Mailbox
Mutatee
Signal
Generator
Thread
SH - 0
SH - 1
University of Maryland
A (slightly) more complex example

Continue mutatee execution
– Wait for it to exit
– Mutatee runs
kill (SIGCONT)
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
Block until
exit event
Event
Gate
SH - 0
SH - 1
University of Maryland
A (slightly) more complex example

Eventually, mutatee executes dlopen()
– Stops, sending SIGTRAP to mutator
kill (SIGCONT)
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
SIGTRAP
Block until
exit event
Event
Gate
SH - 0
SH - 1
University of Maryland
dlopen()
A (slightly) more complex example

SignalGenerator Decodes SIGTRAP
– Analyzes runtime link maps, finds new library
– Sends an evtLoadLibrary Event to a Signal Handler
kill (SIGCONT)
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
SIGTRAP
Block until
exit event
Event
Gate
evtLoadLibrary
SH - 0
SH - 1
University of Maryland
dlopen()
A (slightly) more complex example

SignalHandler handles evtLoadLibrary
– Adjusts Dyninst Internal State (platform indep)
– Obtains relevant Callback from CBManager
– “Executes” loadLibrary callback (zoom in to mailbox)
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
SIGTRAP
Block until
exit event
Event
Gate
evtLoadLibrary
SH - 0
SH - 1
University of Maryland
dlopen()
A (slightly) more complex example

Callback Execution (zoom in):
– Registers instance of callback with Mailbox (target = UI Thread)
– Signals Event Gates that evtLoadLibrary happened
– Blocks until completion of callback
UI Thread
Block until
exit event
SH - 0
Mailbox
CB
Event Gate
Register CB
Signal
evtLoadLibrary
Block until
CB complete
University of Maryland
A (slightly) more complex example

UI Thread, signalled, wakes up
– Checks mailbox and finds callback with target_thread == UIThread
– But the Event does not match target
– Executes callback (zoom back out… )
UI Thread
Block until
exit event
Wake up, Execute
Callbacks
SH - 0
Mailbox
CB
Event Gate
Register CB
Signal
evtLoadLibrary
Block until
CB complete
CB
University of Maryland
A (slightly) more complex example

But the callback triggers another event
– InsertSnippet needs to allocate memory before insertion
– Issues inferiorMalloc RPC and blocks for its completion
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
dlopen()
Block until
exit event
Insert:
Inferior
Malloc RPC
block
resume
Event
Gate
SH - 0
SH - 1
block
Event
Gate
University of Maryland
A (slightly) more complex example


Mutatee issues malloc() and traps
SignalGenerator receives SIGTRAP
– Discovers that we just completed a RPC
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
dlopen()
SIGTRAP
Block until
exit event
Insert:
Inferior
Malloc RPC
block
resume
Event
Gate
SH - 0
SH - 1
block
Event
Gate
University of Maryland
malloc()
A (slightly) more complex example

SignalGenerator sees that SH-0 is busy (blocked)
– Assigns evtRPCSignal event to SH-1
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
dlopen()
SIGTRAP
Block until
exit event
Insert:
Inferior
Malloc RPC
block
resume
evtRPCSignal
Event
Gate
SH - 0
SH - 1
block
Event
Gate
University of Maryland
malloc()
A (slightly) more complex example

SH -1 handles RPC completion event
– Manages Dyninst internal state (RPC Manager)
– Signals Event Gates that evtRPCSignal happened
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
dlopen()
SIGTRAP
Block until
exit event
Insert:
Inferior
Malloc RPC
block
resume
evtRPCSignal
Event
Gate
SH - 0
SH - 1
block
Event
Gate
evtRPCSignal
University of Maryland
malloc()
A (slightly) more complex example

EventGate waiting for evtRPCSignal wakes up
– Releases UI thread from RPC block
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
dlopen()
SIGTRAP
Block until
exit event
Insert:
Inferior
Malloc RPC
block
resume
evtRPCSignal
Event
Gate
SH - 0
SH - 1
block
Event
Gate
evtRPCSignal
University of Maryland
malloc()
A (slightly) more complex example

Callback finishes executing
– Inserts instrumentation in newly malloc()’d space
– SH-0 is released from blocking state
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
dlopen()
SIGTRAP
Block until
exit event
Insert:
Inferior
Malloc RPC
block
resume
evtRPCSignal
Event
Gate
SH - 0
SH - 1
resume
Event
Gate
evtRPCSignal
University of Maryland
malloc()
A (slightly) more complex example

UI Thread resumes blocking for process exit
– Inserts instrumentation in newly malloc()’d space
– SH-0 is released from blocking state
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
dlopen()
malloc()
Block until
exit event
Insert:
Inferior
Malloc RPC
block
resume
Event
Gate
SH - 0
SH - 1
University of Maryland
A (slightly) more complex example

Finally, Mutatee Exits
– evtProcessExit propagates through Event Handling
– UI Thread is released from block
UI Thread
CB Manager
register
callback
Continue
execution
Mailbox
Mutatee
Signal
Generator
Thread
dlopen()
malloc()
Block until
exit event
Event
Gate
resume
evtProcessExit
SH - 0
SH - 1
Signal event: evtProcessExit
University of Maryland
exit()
Thread Inflation aside…


Good things have emerged too…
Unified Dyninst event concept
– All event oriented subsystems operate on the
same generic event concept
• Well, its really just a new c++ class

Unified Dyninst event handling
– Event handling framework is now platform
independent
– Dyninst takes uniform actions when specific
events occur, across platforms
• More or less
• But more than before
University of Maryland
Implications (future features)

Finer grained Callback APIs
– We currently have callbacks for events like
• LoadLibrary
• Process Exit
• Thread Creation
– Now possible (and easy) to expose generic event
callback to user
• Eg. Execute func() when mutatee receives
signal S
• Eg. Execte func() when mutatee issues
dlclose()
University of Maryland
Implications (future features)

Spin-off “Debugger API” library
– Separate out code for process control, signal
handling, other debugger-like operations into
shared library
– Requires solving some complex problems:
• Eg. How to deal with RPCs

How much gets spun off?
– This is a hard question to answer
– Related RPC and shared-object subsytems are
heavily intertwined.
– Where to draw the line?
University of Maryland