A Binary Agent Technology for COTS
Software Integrity
Richard Schooler
Anant Agarwal
InCert Software
The Mission Critical Environment
Input
COTS
Binary
Operating
System
COTS
Binary
SAP
The development
environment
Output
The deployment
environment
Objective
To improve the
integrity of the
deployment
environment
with COTS software
in the presence of
attacks, bugs
Input
Operating
System
COTS
Binary
SAP
Output
Our Focus
Outer security defences will
be breached by attackers
Input
Operating
System
Use a practical, systems
level approach – executiontime monitoring
On COTS program or data
corruption, rapidly
d- detect problems
a- trigger an alarm
p- try to protect
r- recover
COTS
Binary
SAP
Output
Our Approach: Execution-Time Monitoring of
COTS through Binary Instrumentation
d- Policy specs for detection
d- Heartbeat insertion
d- Argument range checks
d- Rare code execution/sigs.
a- Alarm messages to console
p- Defaults for fault tolerance
p- Access constraints, redund.
r- Logging
COTS Binary
COTS New
Legacy
Missing
source
The development
environment
COTS New
Legacy
Missing
source
COTS New
Legacy
The deployment
environment
Missing
source
Our Approach to Integrity
 A pragmatic system level approach
 Works with COTS binaries, even legacy codes
 Allows a user to establish desired security levels and
to some extent modify policy on the fly
 Allows monitoring of all user code execution, not
selected OS calls
 Can work completely at the user’s deployment site
Drawbacks of Binary Insertion
 Specific to a single platform, needs new technology
development for different platform
 Challenging to relate low-level observable events
back to high-level user actions
 hard to detect some types of intrusions that only
affect data corruption
 hard to protect or correct problems at higher
semantic levels
Three Major Components in the Prototype,
Three Major Tasks
 Core technology for customizable agent
insertion into PC/NT, PC/Linux
 Anomaly detection and reporting
 Rapid recovery and problem pinpointing
Selected Challenges and Ideas



Core technology for agent insertion into binary
 How to deal with unknown relocations, e.g., for dusty decks – incremental
control and dataflow analysis; an integrated static and dynamic method
 Dealing with multithreading, in particular, time syncing and monitoring
events in a distributed environment
Anomaly detection
 Runtime comparison against execution path signatures?
 State machines for control flow checks (e.g., Abraham)
 How to minimize runtime overhead – borrow compiler optimization
techniques (e.g., steal registers, inline code, sampling, multilevel checks)
Rapid recovery and problem pinpointing technology
 Instrumentation records trace in circular buffer to pinpoint problem and
facilitate recovery from crash/alert following an actual or suspected attack
 Can we get data values? Use dataflow analysis and offline simulation to
obtain intermediate data values
Measures of Success
 Core technology for agent insertion into binary:
Can we handle all binaries, even dusty decks?
 Target: Performance degradation to be under 1 percent
 Anomaly detection
 What fraction of injected problems can we detect
 automatically
 with user spec
 Rapid recovery technology
 Performance degradation to be under 1 percent
 Can we cut recovery time significantly? We will measure
recovery time with and without
 As a bonus, can we catch problems before system goes
down?
 We are building a prototype system, work with real users, and
measure

Current Progress
 Work on NT binary insertion prototype ongoing
 Demo of early capability showing
instrumentation
 simple recovery log
 detecting application has crashed
 taking control and
 writing out log
 Handling multithreading, DLLs imminent
 Ongoing thinking on detection capability

Instrumentation for Recovery: Basic Idea
 At each program block, record the progress of program
execution.
1
AGENT
test al,0x3
jnz 0x1143
...
2
3
2
4
5
AGENT
test al,0x3
jnz 0x1143
AGENT
add ebx,ecx
jc 0x1101
5
1
4
AGENT
shr edx,0x1
add ebx,edx
AGENT
inc eax
add ecx,edi
add edx,esi
cmp eax,0xa
while ((c = ++ci)) {
INSTRUCTION_ITERATOR ii = c->Instructions();
while ((inst = ++ii))
inst->Lift(null_state);
while ((inst = ++ii))
inst->Lift(null_state);
Instrumentation Runtime
Naïve implementation:
•Write block id
DAG Header
•Increment index (in
memory)
•setup new
trace record
•Check for wrap-around
Too big/slow: many
instructions per block;
large buffer
DAG id
DAG Block 1
2
•set bit in
current record
3
4
DAG id
DAG id
5
Instrumentation Example 1
test al,0x3
jnz 0x1143
add ebx,ecx
jmp short 0x1156
shr edx,0x1
add ebx,edx
inc eax
add ecx,edi
add edx,esi
cmp eax,byte 0xa
jc 0x1101
Original
Instrumentation Example 2
mov eax,[0x300c]
mov ecx,[byte eax+0xc]
mov [byte eax+0xc],dword 0x1
mov edx,[byte eax+0x8]
mov [dword eax+edx+0xffe],ecx
add edx,0x4
and edx,[byte eax+0x4]
mov [byte eax+0x8],edx
test al,0x3
jnz 0x1143
or [byte ebp+0xe],byte 0x2
add ebx,ecx
jmp short 0x1156
or [byte ebp+0xe],byte 0x1
shr edx,0x1
add ebx,edx
or [byte ebp+0xe],byte 0x4
inc eax
add ecx,edi
add edx,esi
cmp eax,byte 0xa
jc 0x1101
Original
Instrumentation
Instrumentation Example 3
mov eax,[0x300c]
mov ecx,[byte eax+0xc]
mov [byte eax+0xc],dword 0x1
mov edx,[byte eax+0x8]
mov [dword eax+edx+0xffe],ecx
add edx,0x4
and edx,[byte eax+0x4]
mov [byte eax+0x8],edx
test al,0x3
jnz 0x1147
or [byte ebp+0xe],byte 0x2
add ebx,ecx
jmp short 0x1164
or [byte ebp+0xe],byte 0x1
shr edx,0x1
add ebx,edx
or [byte ecp+0xe],byte 0x4
inc eax
add ecx,edi
add edx,esi
cmp eax,byte 0xa
jc 0x1103
Original
Instrumentation
Displacement Fixup
Instrumentation Example 4
mov eax,[0x300c]
mov ecx,[byte eax+0xc]
mov [byte eax+0xc],dword 0x1
mov edx,[byte eax+0x8]
mov [dword eax+edx+0xffe],ecx
add edx,0x4
and edx,[byte eax+0x4]
mov [byte eax+0x8],edx
test al,0x3
jnz 0x1147
or [byte ebp+0xe],byte 0x2
add ebx,ecx
jmp short 0x1164
or [byte ebp+0xe],byte 0x1
shr edx,0x1
add ebx,edx
mov ecp,ebp
or [byte ecp+0xe],byte 0x4
inc eax
add ecx,edi
add edx,esi
cmp eax,byte 0xa
jc 0x1103
Original
Instrumentation
Displacement Fixup
Register Transfer
Related Approaches
 Applied at source level during development
e.g., type based safety; work of Lee et al.
 Applied at link time with special object formats
 e.g., software fault isolation; work of Pandey et al.
 Applied through interpretor
 e.g., safe Java interpreters
 Applied during program execution – works with existing COTS
 e.g., sandboxing through middleware software
 e.g., modify OS with extensions to catch OS calls, Ghosh
 e.g., wrap application to steer OS calls to intermediary,
Ghosh

The Current Commercial Solution: Middleware
Input
Middleware
wrappers
COTS
Binary
Operating
System
• Slow
• Cannot deal with viruses
• Cannot improve fault
tolerance of COTS
package itself
SAP
Output
Sandboxing through OS Call Interception
(OS extensions or wrapping executable)
Input
Operating
System
• This is the right first step, and
less intrusive than binary
instrumentation
• Cannot improve fault tolerance
of COTS package itself
• Cannot handle communication
through shared memory
• Cannot observe internal app
behavior
• OS extensions can slow OS for
others
COTS
Binary
SAP
Output
Summary
 A systems approach to COTS Integrity
 Approach based on execution-time
monitoring using binary insertion
 We have an early prototype version of NT
binary insertion implemented