A Taxonomy of Software Assurance
Tools and the Security Bugs they
Catch
Michael Kass
Information Technology Laboratory
OWASP
AppSec
DC
October 2005
National Institute of Standards and Technology
http://samate.nist.gov
[email protected]
This is a work of the U.S. Government and is not subject to copyright
protection in the United States.
The OWASP Foundation
http://www.owasp.org/
SAMATE Goals:
A taxonomy of software assurance tools and
their functions
A taxonomy of the software flaws that those
tools find
A reference dataset (RDS) of tests to measure if
SA tools correctly detect those flaws
OWASP AppSec DC 2005
2
Why a Tool Taxonomy?
Provide a common reference/classification of
functions for evaluation of a tool’s effectiveness
A “first step” in defining a tool functional
specification
Tool specification becomes basis for creating tool
reference dataset/benchmarks
A common specification and reference dataset
for comparing SA tools of the same class
OWASP AppSec DC 2005
3
Tool Taxonomy and Follow-on Products
Workshop
SA tool
classes
tool function
taxonomy
select
tool class
and
functions
focus
group
functional
specification
test plan
test suite
OWASP AppSec DC 2005
4
Tool Survey and Taxonomy
 ”External” Tools
 Network Scanners
 Web Application Scanners
 Web Services Scanners
 Dynamic Analysis Tools
 “Internal” Tools
 Software Requirements Verifiers
 Software Design/Model Verifiers
 Compilers
 Static Source Code Scanners
 Static Byte Code Scanners
 Static Binary Code Scanners
 Database Scanners
Much of the information in the Software Security Tool Taxonomy is derived from the DISA Application Security Assessment Tool Survey,
July 2004, to be published as a DISA STIG.
OWASP AppSec DC 2005
5
Network Scanners
Remotely scan targeted machines, performing port scans, and probing
for vulnerabilities known in operating systems and third party network
software.
 Functions - scan for known vulnerabilities in:
 operating systems
 applications
 web servers
 network devices
 network protocols
 RDS requirements:
 networked testbed
 customize-able deployment scenarios for vulnerability environment
 test case documentation against a functional specification for general
network scanner SA tools
OWASP AppSec DC 2005
6
Web Application Scanners
A more “specialized” class of SA tool that focuses specifically on web
applications only, and are not considered generalized network
scanners.
 Scan for vulnerabilities by:
 simulating a web browser client
 performing field manipulation/cookie poisoning
 exploiting relative URL paths
 injecting malicious data via form fields to exploit server
 RDS requirements:
 HTTP(S) server with applications having known security flaws and
vulnerabilities
 networked testbed
 customize-able deployment environment for vulnerability scenarios
 test case documentation against a functional specification for web
OWASP AppSec DC 2005
application scanner SA tools
7
Web Services Scanner Functions
A relatively new class of SA tool whose purpose is the
analysis of web service applications. Web service scanners
have functions of the following type:
 Scan for vulnerabilities by:
 functional test of SOAP server
 test XML encryption, signatures and signature verification
 test WS-Security
 perform XML validation of received messages
 RDS requirements:
 a networked testbed of web services with known security flaws and
vulnerabilities
 networked testbed
 customize-able deployment environment for vulnerability scenarios
 test case documentation against a functional specification for web
services scanner SA tools
OWASP AppSec DC 2005
8
Dynamic Analysis Tool Functions
Dynamic analysis tools generate runtime vulnerability scenarios
through the following functions:
 Scan for vulnerabilities by:
 performing file corruption
 resource/network/system/interface fault injection
 design attacks
 implementation attacks
 RDS requirements:
 a testbed of applications with known security flaws and vulnerabilities
 virtual environment (sandbox)
 customize-able operating environment to create vulnerability scenarios
 test case documentation against a functional specification for dynamic
analysis SA tools
OWASP AppSec DC 2005
9
Software Requirements Verification Tool Functions
Functions of some requirements verification tools that have been
developed include determining whether requirements are: (from
NASA’s Automated Requirements Measurement Tool )
 Scan for vulnerabilities by verifying that application requirements
are:
 verifiable
 complete
 consistent
 unambiguous
 RDS requirements:
 a corpus of application requirement documents that introduce security
vulnerabilities into application design and/or implementation
 test case documentation against a functional specification for
requirements verification SA tools
OWASP AppSec DC 2005
10
Design/Model Verification Tool Functions
Design/model verification tools provide are a vital
“first line of defense“ in software assurance.
 Identify potential flaws/vulnerabilities by performing:
 application simulation
 exhaustive verification
 proof of design
 RDS requirements:
 a corpus of design documents that introduce security vulnerabilities into
application design and/or implementation
 test case documentation against a functional specification for
design/model verification SA tools
OWASP AppSec DC 2005
11
Compilers
Choosing “type safe” compilers, or extending the capability of
some compilers can provide an additional level of software
security (although not guaranteed). Some security functions of
“extended compilers” include:

Identify potential flaws/vulnerabilities by injecting run-time
code to identify:




buffer overflows
race conditions
un-initialised variables
RDS requirements:




a testbed of applications with known security flaws and vulnerabilities
A virtual test environment
customize-able operating environment to create vulnerability
scenarios
test case documentation against functional specification for dynamic
OWASP AppSec DC 2005
binary analysis SA tools
12
Source Code Scanners
A relatively “new” class of SA tool that can perform data-flow,
control-flow, inter-procedural analysis of code and identify software
security flaws and potential vulnerabilities.
 Some key functions include:
Identify security flaws and vulnerabilities
filter results based upon flaw type/priority
scale to large code sets
generate code metrics
 RDS requirements:
a corpus of source code examples with known security flaws and
vulnerabilities
test case documentation against functional specification for source code
analysis SA tools
OWASP AppSec DC 2005
13
Static Bytecode Analysis Tools
A step below “source code scanners’”, but above binary analysis
tools, byte code analysis tools have similar functions to source code
scanning tools.
 Some key functions include:
Identify security flaws and vulnerabilities
filter results based upon flaw type/priority
scale to large code sets
 RDS requirements:
a corpus of byte code examples with known security flaws and
vulnerabilities
test case documentation against functional specification for static
bytecode analysis tools
OWASP AppSec DC 2005
14
Static Binary Analysis Tools
The “last line of defense” in SA tools. Performs similar SA functions
as source code analysis tools, but does so at “runtime”.
 Some key functions include:
perform boundary checks
identify memory leaks
identify unreachable code
identify race conditions
identify type mismatches
 RDS requirements:
a corpus of binary executable examples with known security flaws and
vulnerabilities
test case documentation against functional specification for static binary
code analysis tools
OWASP AppSec DC 2005
15
Defining a Software Security Flaw Taxonomy
Because of the nature of software development (new technologies, new
languages and language features) a taxonomy of software security flaws
will be a living and changing entity. Additional characteristics that must be
considered include:
 Program vulnerabilities are usually a combination of security flaws
 Mutual flaw exclusion will be difficult to deal with (examples :
authentication vs. logic flaw problem)
 Some of the flaws in the taxonomy cannot be identified by tools today
 Some flaws have never been seen in real world code… yet
 Some flaws can be introduced at multiple points in the SDLC
OWASP AppSec DC 2005
16
A Taxonomy of Software Security Flaws and their
Point of Introduction in the SDLC
One of the first steps in comparing tools of the same class is to determine
what software flaws all tools of that class should be able to identify. The
following hierarchical classification of software flaw root causes is taken
from Secure Software’s CLASP V1.1 Training Manual
• identify range and type errors
• Stack overflow – (requirements, design, implementation)
• Command injection (design, implementation)
• Double Free (implementation)
• identify environmental problems
• Resource exhaustion (design, implementation)
• identify synchronization and timing errors
• TOCTOU (time-of-check time-of-use) race condition (design,
implementation)
• identify protocol errors
• Use of a broken/risky cryptographic algorithm (design)
• identify general logic errors
 Performing a “chroot” without a “chdir” (implementation)
OWASP AppSec DC 2005
17
Reference Dataset
The SAMATE team is creating a reference dataset of test cases to measure
SA tool performance against the functions defined for that particular type of
tool. Currently, we are focusing on source code scanning tools.
 Existing source code material for a reference dataset include:





OWASP WebGoat
Foundstone HacmeBook and HacmeBank
CVE listing of known vulnerabilities
MIT Lincoln Lab Source Code Contribution (1000+ test cases)
Fortify Software Source Code Contribution (80+ test cases)
 Both “In the Wild” and “Manufactured” code will be part of the reference dataset
 Dataset will include representation of software flaws from the SA security flaw
taxonomy
 The SAMATE Reference Dataset will include testing artifacts from requirements,
modelling and design, implementation and deployment phases of the SDLC
OWASP AppSec DC 2005
18
Ongoing Work

NIST is developing prototype reference dataset for SA tools, for review by
SAMATE working group members

November 7,8 -- Workshop on Software Security Assurance Tools,
Techniques, and Metrics , Long Beach, CA (http://samate.nist.gov)
 DAY #1 – Papers
 DAY #2 - “Target Practice” for source code scanner tool developers
against SAMATE SA Tool Reference Dataset

Continue work through SAMATE focus groups (online and at follow-on
workshops) to refine:
 a taxonomy of common functions for source code scanners
 a common taxonomy of software security flaws
 content of a reference dataset for testing SA tools
OWASP AppSec DC 2005
19