OmniUmpack: Fast, Generic, and
Safe Unpacking of Malware
Authors: Lerenzo Martignoni, Mihai Christodorescu and
Somesh Jha
Computer Security Applications Conference 2007 (ACSAC’07)
Presented by LIU Limin
1
Agenda
•
•
•
•
•
Introduction
Design
Implementation
Evaluation
Conclusion
2
Agenda
•
•
•
•
•
Introduction
Design
Implementation
Evaluation
Conclusion
3
Packed malware
• Term packed refers to malware whose payload
is either compressed or encrypted.
– E.g. polymorphic malware
• An unpacking routine that restores the original
code from memory and then executes it during
runtime.
– Self-decompression or self-decryption
– Several layers of unpacking
4
Why malware packed?
• Anti-analysis
– Reverse engineering
– Experts can make the malware hard to analyze
• Anti-virus tools evasion
– Existing packers are modified to evade anti-virus
tools at the rate of 10-15 per month.
– 80% new malware are packed with various packers.
– 50% new malware samples are repacked versions of
existing malware.
How to effectively recover the real malicious code?
5
Traditional unpacking approaches
• Algorithmic unpacking
– Using specific unpacking routines to recover the
original program, one routine per packing algorithms.
• ClamAV
– Limited to a fixed set of known packers.
6
Traditional unpacking approaches (Cont.)
• Generic unpacking
– Emulation/tracing of the execution until the
unpacking routine terminates.
• PolyUnpack ([ACSAC 06]), Renovo ([Worm 07])
– Limitations
• Unpacking is slow because of instruction-level tracing.
• Effectiveness depends on the fidelity of the emulation
environment.
• The detection of the termination of the unpacking routine
is undecidable.
7
Goals of OmniUnpack
• Generic unpacking with low-overhead by
tracking memory access at the page level.
• Precise unpacking by running the program on
native OS.
• An in-memory malware detection strategy
independent of packing and self-modification.
8
Agenda
•
•
•
•
•
Introduction
Design
Implementation
Evaluation
Conclusion
9
OmniUnpack: simple strategy
• Tracking all memory writes and the program
counter.
• The execution of a previously written memory
locations indicates the end of an unpacking
stage.
• All written-then-executed (or written and
about to be executed) memory locations should
then be analyzed by a malware detector.
Starting from this strategy, designing an unpacking
algorithm that achieves low overhead yet does not
compromise the security of the system.
10
Execution traces for a packed executable
• Ideally
– The detection of the termination of the unpacking routine is
undecidable.
• Coarse-grained memory access tracking
– using hardware mechanism at page level.
– Written-then-executed pages are indicative of unpacking but
not indicative of the end of unpacking.
– Increasing the chance to detect spurious unpacking stages (up
to hundreds of thousands stages) .
– The overhead introduced by invoking the malware detector
11
every time a written memory page is executed is prohibitive.
OmniUnpack-enhanced detection
• Heuristic: An unpacking stage is completed when the
execution of previous written page is followed by a
dangerous system call.
• Choice of dangerous system calls
– A dangerous system call is a system call whose execution can
leave the system in an unsafe state.
12
Continuous monitoring of the execution
• OmniUnpack implements a continuous
monitoring approach, where the execution is
observed in its entirety.
– Multiple unpacking stages
– Approximation of the end of an unpacking stage
• A solution for the undecidability of the
problem of detecting the real end of the
unpacking stage.
• The low overhead of OmniUnpack allows for
continuous monitoring in an end-user
environment.
13
Algorithm
14
Example
• Execution Trace :
<x(0), w(1), s1, w(2), x(1), s2, x(2), s3, …>
–
–
–
–
–
–
–
–
x(0): W = Ø, WX = Ø
s1 is not dangerous
w(1): W= {1}, WX = Ø
and WX is empty.
s1 (NtOpenFile): W = {1}, WX = Ø
w(2): W = {1,2}, WX= Ø
x(1): W = {1,2}, WX= {1}
s2 is not dangerous.
s2 (NtOpenKey): W = {1,2}, WX= {1}
x(2): W = {1,2), WX = {1,2}
s3 (NtDeleteFile):
The system call is dangerous, malware detector is
invoked to scan all the memory pages in W.
15
Agenda
•
•
•
•
•
Introduction
Design
Implementation
Evaluation
Conclusion
16
Architecture
• Kernel driver
– Tracking memory
access
– Detecting when the
dangerous system
call occurs
– Triggering malware
detection
• User-space
Component
– Malware detector
responsible for ondemand scanning of
memory of the
suspicious program.
17
Monitoring memory accesses
• The W⊕X policy is enforced on the memory pages of
the suspicious program.
– A memory page can be either writable or executable.
• Page-fault exceptions are trapped by OmniUnpack.
– When an unpacking routine writes the unpacked code to
memory, the destination page is marked writable but not
executable.
– When an unpacking stage ends, the access to this page for
execution will cause a protection exception because of the
lack of execution permission.
– Intercept and process such exceptions.
• Non-executable pages can be emulated via software
mechanism with a minimal overhead.
– PaX PAGEEXEC
18
Monitoring system calls
• Intercepting the system calls of interest (dangerous
ones) through interposition on the system-call
dispatch table.
• If newly generated code is present in the program
memory, the code is executed, and the execution is
followed by a dangerous system call, then consider
the unpacking stage concluded and trigger malware
detection.
• At the end of scanning, write permissions are revoked
from all scanned pages.
19
User-space malware detector
• Any malware detection strategy can be used to scan
the code generated during the previous stage.
• OmniUnpack only provides a detector with data to
analyze and then authorizes, or denies, the execution
of such data according to the response of the
detector.
• A bug in the malware detector will not compromise
the stability of the whole system since the detector
is running in user-space.
20
Agenda
•
•
•
•
•
Introduction
Design
Implementation
Evaluation
Conclusion
21
Experimental evaluation
• Successfully monitored
all the analyzed
samples and detected
the end of each
unpacking stage.
• Handled 80% of
packed malware, while
ClamAV handled only
about 15%.
• Armadillo and CExe
unpack the original
payload on disk and
then execute this
executable.
•
•
N/A: programs that did not execute correctly after packing, with or
without OmniUnpack.
22
IMPL: a limitation of the current OmniUnpack implementation.
Time to unpack
• OmniUnpack was significantly faster than PolyUnpack.
(e.g., with UPX OmniUnpack was 20.56 times faster
than PolyUnpack.)
• In 42% of the cases PolyUnpack timeout (300 secs).
• OmniUnpack was within one order of magnitude of
ClamAV (e.g., with UPX OmniUnpack was only 5 times
slower).
• In 75% cases, ClamAV failed to unpack.
23
Overhead for benign programs
• About 6% average overhead on non-packed
applications.
• About 11% average overhead on packed
applications (with UPX).
24
Agenda
•
•
•
•
•
Introduction
Design
Implementation
Evaluation
Conclusion
25
Conclusion
• A new unpacking technique that addresses the
shortcomings of previous unpackers.
– Generic unpacker
• Able to deal with known and unknown unpacking algorithms.
• resilient to anti-debugging, anti-VM and anti-emulation
techniques.
• Integrated with any operating system and any malware detector
without trading off efficiency or unpacking capability.
– Suitable for use on end-user environment.
• Minimizing the overhead by using hardware mechanism and
working directly in the OS.
– Safe in the presence of multiple unpacking stages.
– Improved detection rate.
• Any malware detector can delegate to OmniUnpack the recovery
of the packed code.
26
Limitations
• Imprecision of page-level tracking.
– The memory monitoring at page level caused the
undecidability of the specific addresses which are
really accessed on the page.
– The assumption that memory accesses potentially
affect all addresses on a page will cause the
identification of spurious unpacking stages.
•
Incomplete extracted instructions.
– The content of written-then-executed page is
modified before invoking a dangerous system call.
27
Thank you!
Questions?
28
PolyUnpack ([ACSAC 06])
• Performing static analysis over malware instance to acquire its
static code model.
• The statically derived model and malware instance are fed into
the dynamic analysis component.
• The execution of malware paused after each instruction and its
context is compared with the model.
• When the first instruction of a sequence not found in the static
model is detected, representations of that unknown instructions
sequences are written out and the malware’s execution is halted.
29
Renovo ([Worm 07])
• Heuristic: the original program code and data should be present
in memory to be executed, and also the instruction point should
jump to the OEP of the restored program code which has been
written in memory at runtime.
• A technique dynamically extract the hidden original code and the
OEP from the packed executable by examining whether the
current instruction has been generated at run-time, after the
program binary was loaded.
• Instruction level.
30