TEST METHODOLOGY
Next Generation Firewall (NGFW)
v7.0
NSS Labs
Test Methodology – Next Generation Firewall v7.0
Table of Contents
1
1.1
1.2
1.3
2
2.1
2.2
2.3
3
Introduction .................................................................................................................... 5
The Need for Next Generation Firewalls (NGFWs) ............................................................................ 5
About This Test Methodology ............................................................................................................ 5
Inclusion Criteria ................................................................................................................................ 6
Product Guidance ............................................................................................................ 8
Recommended ................................................................................................................................... 8
Neutral ............................................................................................................................................... 8
Caution ............................................................................................................................................... 8
Security Effectiveness ...................................................................................................... 9
3.1 Firewall Policy Enforcement............................................................................................................... 9
3.1.1
Baseline Policy........................................................................................................................... 10
3.1.2
Simple Policies ........................................................................................................................... 10
3.1.3
Complex Policies........................................................................................................................ 10
3.1.4
Static NAT (Network Address Translation) ............................................................................... 10
3.1.5
Dynamic/Hide NAT (Network Address Translation).................................................................. 10
3.1.6
SYN Flood Protection................................................................................................................. 10
3.1.7
IP Address Spoofing .................................................................................................................. 10
3.1.8
TCP Split Handshake Spoof ....................................................................................................... 10
3.2 Application Control .......................................................................................................................... 11
3.2.1
Block .......................................................................................................................................... 11
3.2.2
Block Specific Action (Depends on the application) .................................................................. 11
3.3 Intrusion Prevention ........................................................................................................................ 11
3.3.1
False Positive Testing ................................................................................................................ 12
3.3.2
Exploit Library ........................................................................................................................... 12
3.3.3
Coverage by Attack Vector........................................................................................................ 13
3.3.4
Coverage by Impact Type.......................................................................................................... 13
3.3.5
Coverage by Date ...................................................................................................................... 14
3.3.6
Coverage by Vendor .................................................................................................................. 15
3.3.7
Coverage by Result.................................................................................................................... 15
3.4 Evasions ............................................................................................................................................ 16
3.4.1
IP Packet Fragmentation .......................................................................................................... 16
3.4.2
Stream Segmentation ............................................................................................................... 16
3.4.3
RPC Fragmentation ................................................................................................................... 17
3.4.4
URL Obfuscation ....................................................................................................................... 17
3.4.5
HTML Obfuscation .................................................................................................................... 18
3.4.6
HTTP Compression .................................................................................................................... 19
3.4.7
Payload Encoding ...................................................................................................................... 19
3.4.8
FTP/Telnet Evasion.................................................................................................................... 19
3.4.9
Payload Padding ....................................................................................................................... 19
2
NSS Labs
3.4.10
4
Test Methodology – Next Generation Firewall v7.0
Layered Evasions ....................................................................................................................... 20
Performance ................................................................................................................. 21
4.1 Raw Packet Processing Performance (UDP Throughput)................................................................. 21
4.1.1
64 Byte Packets ......................................................................................................................... 21
4.1.2
128 Byte Packets ....................................................................................................................... 21
4.1.3
256 Byte Packets ....................................................................................................................... 21
4.1.4
512 Byte Packets ....................................................................................................................... 21
4.1.5
1024 Byte Packets ..................................................................................................................... 21
4.1.6
1514 Byte Packets ..................................................................................................................... 21
4.2 Latency ............................................................................................................................................. 22
4.2.1
64 Byte Frames ......................................................................................................................... 22
4.2.2
128 Byte Frames ....................................................................................................................... 22
4.2.3
256 Byte Packets ....................................................................................................................... 22
4.2.4
512 Byte Packets ....................................................................................................................... 22
4.2.5
1024 Byte Packets ..................................................................................................................... 22
4.2.6
1514 Byte Packets ..................................................................................................................... 22
4.3 Maximum Capacity .......................................................................................................................... 22
4.3.1
Theoretical Maximum Concurrent TCP Connections ................................................................ 23
4.3.2
Theoretical Maximum Concurrent TCP Connections with Data ................................................ 23
4.3.3
Maximum TCP Connections per Second.................................................................................... 23
4.3.4
Maximum HTTP Connections per Second ................................................................................. 23
4.3.5
Maximum HTTP Transactions per Second ................................................................................ 23
4.3.6
Maximum SSL Handshakes per Second..................................................................................... 24
4.4 HTTP Capacity with No Transaction Delays ..................................................................................... 24
4.4.1
44 KB HTTP Response Size – 2,500 Connections per Second..................................................... 24
4.4.2
21 KB HTTP Response Size – 5,000 Connections per Second..................................................... 24
4.4.3
10 KB HTTP Response Size – 10,000 Connections per Second................................................... 25
4.4.4
4.5 KB HTTP Response Size – 20,000 Connections per Second.................................................. 25
4.4.5
1.7 KB HTTP Response Size – 40,000 Connections per Second.................................................. 25
4.5 Application Average Response Time: HTTP ..................................................................................... 25
4.6 HTTP Capacity with Transaction Delays ........................................................................................... 25
4.6.1
21 KB HTTP Response with Delay .............................................................................................. 25
4.6.2
10 KB HTTP Response with Delay .............................................................................................. 25
4.7 HTTP Capacity with HTTP Persistent Connections ........................................................................... 26
4.7.1
250 Connections per Second ..................................................................................................... 26
4.7.2
500 Connections per Second ..................................................................................................... 26
4.7.3
1000 Connections per Second ................................................................................................... 26
4.8 HTTPS Capacity with No Transaction Delay HTTP Persistent Connections ...................................... 26
4.8.1
250 Connections per Second ..................................................................................................... 27
4.8.2
500 Connections per Second ..................................................................................................... 27
4.8.3
1000 Connections per Second ................................................................................................... 27
3
NSS Labs
Test Methodology – Next Generation Firewall v7.0
4.9 “Real-World” Traffic Mixes .............................................................................................................. 27
4.9.1
“Real-World” Protocol Mix (Enterprise Perimeter) ................................................................... 27
4.9.2
“Real-World” Protocol Mix (Financial) ...................................................................................... 28
4.9.3
“Real-World” Protocol Mix (US Mobile Carrier) ........................................................................ 28
4.9.4
“Real-World” Protocol Mix (European Mobile Carrier)............................................................. 28
4.9.5
“Real-World” Internal Segmentation Mix ................................................................................. 29
4.10
IPsec Site-to-Site VPN ................................................................................................................... 29
4.10.1 21 KB HTTP Response Size......................................................................................................... 29
5
Stability and Reliability .................................................................................................. 30
5.1 Blocking Under Extended Attack...................................................................................................... 30
5.2 Passing Legitimate Traffic under Extended Attack .......................................................................... 30
5.3 Behavior of the State Engine Under Load ........................................................................................ 30
5.3.1
Attack Detection/Blocking – Normal Load................................................................................ 31
5.3.2
State Preservation – Normal Load ............................................................................................ 31
5.3.3
Pass Legitimate Traffic – Normal Load ..................................................................................... 31
5.3.4
State Preservation – Maximum Exceeded ................................................................................ 31
5.3.5
Drop Legitimate Traffic – Maximum Exceeded ......................................................................... 31
5.4 Protocol Fuzzing and Mutation ........................................................................................................ 31
5.5 Power Fail ......................................................................................................................................... 31
5.6 Persistence of Data .......................................................................................................................... 32
6
Total Cost of Ownership and Value ................................................................................ 33
Appendix A: Change Log ...................................................................................................... 34
Contact Information ............................................................................................................ 35
4
NSS Labs
Test Methodology – Next Generation Firewall v7.0
1 Introduction
1.1 The Need for Next Generation Firewalls (NGFWs)
Firewall technology is one of the largest and most mature security markets. Firewalls have undergone several
stages of development, from early packet filtering and circuit relay firewalls to application layer (proxy-based) and
dynamic packet filtering firewalls. Throughout their history, however, the goal has been to enforce an access
control policy between two networks, and they should therefore be viewed as an implementation of policy.
A firewall is a mechanism used to protect a trusted network from an untrusted network, while allowing authorized
communications to pass from one side to the other, thus facilitating secure business use of the Internet. With the
emergence of new web applications and security threats, however, firewalls are evolving further. Next generation
firewalls (NGFWs) traditionally have been deployed to defend the network on the edge, but enterprises have
expanded deployment options to include internal segmentation.
As Web 3.0 trends push critical business applications through firewall ports that previously were reserved for a
single function, such as HTTP, legacy firewall technology is effectively blinded. It is unable to differentiate between
actual HTTP traffic and non-HTTP services tunneling over port 80, such as VoIP or instant messaging. Today
application-level monitoring must be performed in addition to analysis of port and destination. Firewalls are
evolving to address this increased complexity.
It is no longer possible to rely on port and protocol combinations alone to define network applications. The NGFW
must be capable of performing deep packet inspection on all packets, on all ports, and over all protocols in order
to determine which applications are running over which ports and thus secure them effectively. In addition, with
the expanded use of SSL/TLS in much of the traffic traversing the modern network, inspection of encrypted
content is required.
1.2 About This Test Methodology
NSS Labs’ test reports are designed to address the challenges faced by enterprise security and IT professionals in
selecting and managing security products. The scope of this particular methodology includes:




Security effectiveness
Performance
Stability and reliability
Total cost of ownership (TCO)
As NGFWs are deployed at critical choke points in the network, their stability and reliability is imperative.
Therefore, regardless of any new deep inspection capabilities, the main requirement of any NGFW is that it must
be as stable, as reliable, as fast, and as flexible as the firewall that it is replacing.
5
NSS Labs
Test Methodology – Next Generation Firewall v7.0
The following capabilities are considered essential in an NGFW:

Traditional “first generation firewall” features, including:

o Basic packet filtering
o Stateful multi-layer inspection
o Network Address Translation (NAT)
o Virtual private network (VPN)
“Next generation firewall” features, including:
o
o
o
o
o
o
o
Application awareness/control
User/group control
Integrated intrusion prevention system (IPS)
Ability to operate at OSI Layer 3 (“traditional”)
External intelligence to enhance blocking decisions (i.e., “reputation services”)
Anti-malware
SSL inspection ability
Tuning: Security engineers typically will tune an IPS to ensure its protection coverage matches the needs of the
environment it is being placed in. Though this strategy works well for data centers and demilitarized zones (DMZs),
protecting desktops is a different matter. In surveying enterprises, NSS researchers discovered that many
enterprises do not strictly control the desktop, and that in larger enterprises, there can be a wide range of
applications running on the typical endpoint. As such, enterprises are expecting IPS and NGFW vendors to provide
maximum security for desktop client applications with their out-of-the-box recommended or default policies. In
addition, research indicates that enterprises are not ready to replace their dedicated IPS solutions in the data
center with NGFWs.
This leads us to conclude that the intrusion prevention functionality within an NGFW typically will be used to
protect desktop clients, with optimal protection predefined via a vendor-supplied recommended/default policy,
and this is how the devices will be tested. Anti-malware technology is allowed in NGFW 7.0 testing, as many
vendors have integrated core security engine functions into their anti-malware functionality.
1.3 Inclusion Criteria
In order to encourage the greatest participation, and to allay any potential concerns of bias, NSS invites all security
vendors claiming NGFW capabilities to submit their products at no cost. Vendors with major market share, as well
as challengers with new technology, will be included.
The NGFW should be supplied as a single appliance, where possible (cluster controller solutions are also
acceptable), with the appropriate number of physical interfaces capable of achieving the required level of
connectivity and performance (minimum of one inline port pair per Gigabit of throughput, or one inline 10 Gbps
port pair per 10 Gbps of throughput).
Firewall products in a traditional edge deployment must be implemented as Layer 3 (routing) devices. 1 Gbps
(copper) or 10 Gbps (fiber) connections will be made from the external to internal switches via the device under
test (DUT), subject to a minimum of one inline port pair per Gigabit (copper) or per 10 Gigabits (fiber) of
throughput. Thus, an 8 Gbps device with only four 1Gbps port pairs will be limited to 4 Gbps. The maximum
number of port pairs will be connected to determine the overall throughput of the DUT. The management
interface must be 1Gb copper, or 10Gb fiber.
6
NSS Labs
Test Methodology – Next Generation Firewall v7.0
Once installed in the test lab, the DUT will be configured for the use case appropriate to the target deployments
(corporate network perimeter and internal segmentation). The DUT must also be configured to block all traffic
when resources are exhausted or when traffic cannot be analyzed for any reason.
7
NSS Labs
Test Methodology – Next Generation Firewall v7.0
2 Product Guidance
NSS issues summary product guidance based on evaluation criteria that is important to information security
professionals. The evaluation criteria are as follows:





Security effectiveness – The purpose of an NGFW is to separate internal trusted networks from external
untrusted networks through policy and routing, and to identify and block attacks against assets while allowing
select controlled traffic to flow between trusted and untrusted networks.
Resistance to evasion – Failure in any evasion class permits attackers to circumvent protection.
Stability and reliability – Long-term stability is particularly important for an inline device, where failure can
produce network outages.
Performance – Correctly sizing an NGFW is essential.
Value – Customers should seek low TCO and high effectiveness and performance rankings.
Products are listed in rank order according to their guidance rating.
2.1 Recommended
A Recommended rating from NSS indicates that a product has performed well and deserves strong consideration.
Only the top technical products earn a Recommended rating from NSS, regardless of market share, company size,
or brand recognition.
2.2 Neutral
A Neutral rating from NSS indicates that a product has performed reasonably well and should continue to be used
if it is the incumbent within an organization. Products that earn a Neutral rating from NSS deserve consideration
during the purchasing process.
2.3 Caution
A Caution rating from NSS indicates that a product has performed poorly. Organizations using one of these
products should review their security posture and other threat mitigation factors, including possible alternative
configurations and replacement. Products that earn a Caution rating from NSS should not be short-listed or
renewed.
8
NSS Labs
Test Methodology – Next Generation Firewall v7.0
3 Security Effectiveness
This section verifies that the DUT is capable of enforcing a specified security policy effectively. NSS firewall analysis
is conducted by incrementally building upon a baseline configuration (simple routing with no policy restrictions) to
a complex, real-world, multiple-zone configuration supporting many addressing modes, policies, applications, and
inspection engines.
At each level of complexity, test traffic is passed across the firewall to ensure that only specified traffic is allowed,
and the rest denied, and that the appropriate log entries are recorded. Administrative visibility is critical; to
facilitate debugging, NSS recommends logging any function that results in dropped traffic.
The NGFW must support stateful firewalling either by managing state tables to prevent “traffic leakage,” or as a
stateful proxy. The NGFW must be able to manage firewall policy across multiple interfaces/zones. NSS also
requires that a single security policy be applied to all interfaces under test. At a minimum, the firewall must
provide a “trusted” internal interface, an “untrusted” external/Internet interface, and (optionally) one or more
DMZ interfaces. In addition, a dedicated management interface (virtual or otherwise) is preferred.
3.1 Firewall Policy Enforcement
Policies are rules that are configured on a firewall to permit or deny access from one network resource to another,
based on identifying criteria such as source, destination, and service. A term typically used to define the
demarcation point of a network where policy is applied is a demilitarized zone (DMZ). Policies are typically written
to permit or deny network traffic from one or more of the following zones:



Untrusted – This is typically an external network and is considered to be unknown and not secure. An example
of an untrusted network would be the Internet.
DMZ – This is a network that is being isolated by the firewall
restricting network traffic to and from hosts contained within the
isolated network.
Trusted – This is typically an internal network; a network that is
considered secure and protected.
NSS’ firewall tests verify performance and the ability to enforce policy
between the following:



Trusted to Untrusted
Untrusted to DMZ
Trusted to DMZ
Note: Firewalls must provide at a minimum one DMZ interface in order to
provide a DMZ or “transition point” between untrusted and trusted
networks
9
NSS Labs
3.1.1
Test Methodology – Next Generation Firewall v7.0
Baseline Policy
Routed configuration with an “allow all” policy.
3.1.2
Simple Policies
Simple outbound and inbound policies allow basic browsing and email access for internal clients and no external
access.
3.1.3
Complex Policies
Complex outbound and inbound policies consist of many rules, objects, and services.
3.1.4
Static NAT (Network Address Translation)
Inbound NAT using fixed IP address translation with one-to-one mapping.
3.1.5
Dynamic/Hide NAT (Network Address Translation)
Outbound NAT (from internal to external), where all outbound traffic “hides” behind the IP address of the external
interface of the firewall, utilizing a pool of high ports to manage multiple connections.
3.1.6
SYN Flood Protection
The basis of a SYN flood attack is to fail to complete the three-way handshake necessary to establish a legitimate
session. The objective of SYN flooding is to disable one side of the TCP connection, which will result in one or more
of the following:



The server is unable to accept new connections.
The server crashes or becomes inoperative.
Authorization between servers is impaired.
The DUT is expected to protect against SYN floods.
3.1.7
IP Address Spoofing
This test attempts to confuse the firewall into allowing traffic to pass from one network segment to another. By
forging the IP header to contain a source address that is different from the address that transmitted the packet, an
attacker can make it appear that the packet was sent from a different (trusted) machine. The endpoint that
receives successfully spoofed packets will respond to the forged source address (the attacker).
The DUT is expected to protect against IP address spoofing.
3.1.8
TCP Split Handshake Spoof
This test attempts to confuse the firewall into allowing traffic to pass from one network segment to another. The
TCP split handshake blends features of both the three-way handshake and the simultaneous-open connection.
10
NSS Labs
Test Methodology – Next Generation Firewall v7.0
The result is a TCP spoof attack that allows an attacker to bypass the firewall by instructing the target to “initiate”
the session back to the attacker. Popular TCP/IP networking stacks respect this handshaking method with no
modification, including Microsoft, Apple, and Linux stacks. 1
The DUT is expected to protect against TCP split handshake spoofing.
3.2 Application Control
These are complex outbound and inbound policies, which consist of many rules, objects, and applications, and
which verify that the DUT is capable of accurately determining the correct application from deep packet inspection
(regardless of port/protocol used), and then taking the appropriate action.






Popular social networking web sites (web applications)
Instant messaging (IM)
Skype and other VoIP
Torrents
Facebook
Other basic legacy applications (e.g., FTP, Telnet)
For each application, NSS will test the DUT’s ability to perform the following functions:
3.2.1
Block
The DUT should be able to correctly identify and then block the application.
3.2.2
Block Specific Action (Depends on the application)
For example, in the case of instant messaging, the DUT should allow text communications while blocking file
transfers.
3.3 Intrusion Prevention
These are policies consisting of threat protection signatures, which verify that the DUT is capable of correctly
blocking malicious traffic based on a comparison of packet/session contents against signatures/filters/protocol
decoders.
The latest signature pack is acquired from the vendor’s support site, and the DUT is deployed with an out-of-thebox recommended or default security profile. The vendor may not tune the product. Once deployed, the device's
inspection capabilities are governed solely through firmware and signature updates. All signatures used must be
available to the general public at the time of testing; no custom signatures are permitted.NSS research has found
that this approach reflects a typical enterprise deployment and will align results from testing with product
performance in the field. The NGFW is required to block and log exploit attempts and hostile traffic.
1
“The TCP Split Handshake: Practical Effects on Modern Network Equipment,” Tod Alien Beardsley & Jin Qian,
http://www.macrothink.org/journal/index.php/npa/article/view/285
11
NSS Labs
Test Methodology – Next Generation Firewall v7.0
NSS considers it unacceptable for a product of this nature to be sold without at least one recommended or default
profile. If multiple preconfigured profiles exist, vendors may use the highest-security profile available as their
recommended profile, as long as that profile meets the following criteria:


The security profile must be available to all of its customers
With the security profile installed, the device must experience zero false positive events during testing
If a device does experience false positive events for the selected security profile, the vendor will be required to
select a lower-security profile, and the test process will be repeated until a profile is found that produces zero false
positives.
Note: NSS will make a distinction between “false positive” and “true positive” events, where a device legitimately
identifies a problem with the test traffic. The device will not be required to be reconfigured as a result of “true
positive” events.
3.3.1
False Positive Testing
The ability of the DUT to identify and allow legitimate traffic while maintaining protection against threats and
exploits is just as important as its abilility to protect against malicious content. This test will include a varied
sample of legitimate application traffic, which should be identified and allowed, or blocked, based on policy rules.
3.3.2
Exploit Library
NSS’ Security Effectiveness testing leverages the deep expertise of our engineers who utilize multiple commercial,
open-source, and proprietary tools, including NSS’ network live stack test environment2 as appropriate. With
thousands of exploits, this is the industry’s most comprehensive test to date. Most notably, all of the live exploits
and payloads in the NSS exploit test have been validated in our lab such that one or more of the following is true:






A reverse shell is returned
A bind shell is opened on the target allowing the attacker to execute arbitrary commands
Arbitrary code is executed
A malicious payload is installed
A system is rendered unresponsive
Etc.
This test goes far beyond replaying packet captures or pressing the button on a test tool. In short, NSS engineers
trigger vulnerabilities for the purpose of validating that an exploit was able to pass through the DUT.
2
For more information on the NSS “Live Testing™” harness and methodology, please refer to the latest Security Stack (IPS): Test Methodology
located at https://www.nsslabs.com/research-advisory/library/search/category/library/methodologies/
12
NSS Labs
3.3.3
Test Methodology – Next Generation Firewall v7.0
Coverage by Attack Vector
Threats and exploits can be initiated by either the target or the attacker targeting local or remote vulnerabilities.
Common examples of attacker-based threats and target-based threats for both local and network vulnerabilities
are prsented below:
Attacker
Target
Network
Local
RPC Exploit
Root Kit
Browser Exploit
Trojan
*Example exploits included above for reference purposes.
3.3.3.1
Attacker Initiated
Also referred to as “server-side” exploits, the threat/exploit is executed remotely by the attacker against a
vulnerable application and/or operating system.
3.3.3.2
Target Initiated
The threat/exploit is initiated by the vulnerable target. The attacker has little or no control as to when the target
user or application will execute the threat. Since NGFW devices are typically deployed to protect end users, this
class of exploit is the main focus of the NGFW Security Effectiveness test.
3.3.3.3
Network
Threats/exploits that are initiated as a result of network communication.
3.3.3.4
Local
Local execution that requires existing access to the target (not applicable to the NGFW Test Methodology).
Protective ratings are reported in raw percentages of mitigated attacks and their resulting impact: system, service,
fault, reconnaissance. Although a system or service exploit may be partially mitigated by the DUT, the service could
have crashed because of the residual communications resulting in a fault impact on the service or operating
system.
3.3.4
Coverage by Impact Type
The NSS Exploit Library contains thousands of publicly available exploits (including multiple variants of each
exploit), from which groups of exploits are carefully selected to test based on appropriate usage. Each exploit has
been validated to impact the target vulnerable host(s).
3.3.4.1
System Exposure
Attacks resulting in remote system compromise, which provide the attacker with the ability to execute arbitrary
system-level commands. Most of the exploits in this class are weaponized and offer the attacker a fully interactive
remote shell on the target client or server.
13
NSS Labs
3.3.4.2
Test Methodology – Next Generation Firewall v7.0
Service Exposure
Such attacks result in an individual service compromise, but they do not provide the attacker with the ability to
execute arbitrary, system-level commands, nor do they immediately result in full system-level access to the
operating system and all services. However, by using additional localized system attacks, it may be possible for the
attacker to move from the service level to the system level.Typical attacks in this category include service-specific
attacks, such as SQL injection, that enable the attacker to execute arbitrary SQL commands within the database
service.
3.3.4.3
System or Service Fault
Attacks resulting in a system or service-level fault that crashes the targeted service or application and requires
administrative action to restart the service or reboot the system. These attacks do not enable the attacker to
execute arbitrary commands. However, the resulting impact to the business could be severe given that the
attacker could crash the protected system or service.
3.3.5
Coverage by Date
The typical enterprise will run a mix of both old and new applications, and NSS research shows that crimeware kits
will frequently include exploits that date back several years. Therefore, NSS security effectiveness testing will
include exploits current at the time of the test, as well as targeting vulnerabilities covering multiple years dating
backwards from the time of the test. Results will be reported by year, beginning in 2005, extending to current year
of the NGFW test. Results prior to that time period, where applicable, will be aggregated into the oldest “bucket.”
Exploits are added to the NSS Strike Packs according to the year the strike was added to the NSS exploit harness,
not according to the year that the CVE was discovered and documented. For example, NSS may add an exploit with
a CVE indicating a date of 2013 to the 2016 NSS Strike Pack.
14
NSS Labs
3.3.6
Test Methodology – Next Generation Firewall v7.0
Coverage by Vendor
NSS’ exploit test contains many vendors, including but not limited to the following.






















3.3.7
3Com
Apache
Avast
Borland
Citrix
Facebook
HP
ISC
Lighttpd
MacroVision
Mercury
Mozilla
MySQL
Nullsoft
OPenSSH
Other misc.
Samba
Sophos
Sun Microsystems
Trillian
VideoLAN
WinFTP






















Adobe
Apple
BEA
CA
ClamAV
GNU
IBM
Kaspersky
Linux
Mailenable
Microsoft
Mplayer
NOD32
OpenLDAP
OPenSSL
Panda
SAP
SpamAssassin
Symantec
UltraVNC
VMWare
Winzip






















Alt-N
Atrium
BitDefender
Cisco
EMC
Google
IPSwitch
LanDesk
Macromedia
McAfee
MIT
Multiple vendors
Novell
OpenOffice
Oracle
RealNetworks
Snort
Squid
Trend Micro
Veritas
Winamp
Yahoo
Coverage by Result
The following results of exploitation are represented in NSS’ exploit test.
3.3.7.1
Arbitrary Code Execution
A software bug that allows an attacker to execute any commands of the attacker's choice on a target machine or in
a target process.
3.3.7.2
Buffer Overflow
The exploitation of a software bug due to improperly established memory bounds allows an attacker to overwrite
adjacent memory and execute a command.
3.3.7.3
Code Injection
The exploitation of a software bug that allows the processing of invalid data within a program. Code injection can
be used by an attacker to introduce code into a computer program to change the course of execution.
3.3.7.4
Cross-Site Script
The exploitation of a web application that enables attackers to insert malicious script into web pages, which can
then be executed by other users.
15
NSS Labs
3.3.7.5
Test Methodology – Next Generation Firewall v7.0
Directory Traversal
The exploitation of a lack of security in an application (as opposed to exploiting a bug in the code), which allows
user-supplied input with characters representing “traverse to parent directory” to be passed to the file APIs.
The goal of this attack is to order an application to access a file or executable that would not normally be
accessible.
3.3.7.6
Privilege Escalation
This exploit type allows an attacker to gain access to resources that would not normally be available.
3.3.7.7
Target Type
The following web target types are represented in NSS’ exploit test:
Web server
ActiveX
Browser plug-ins/add-ons
Web browser
JavaScript
3.4 Evasions
Attackers can modify basic attacks to evade detection in a number of ways. If a DUT fails to detect a single form of
evasion, any exploit can pass through the device, rendering it ineffective. NSS verifies that the DUT is capable of
detecting and blocking basic exploits when subjected to varying common evasion techniques. Wherever possible,
the DUT is expected to successfully decode the obfuscated traffic to provide an accurate alert relating to the
original exploit, rather than alerting purely on anomalous traffic detected as a result of the evasion technique
itself.
A number of common exploits are executed across the DUT to ensure that they are detected in their unmodified
state. These will be chosen from a suite of older/common basic exploits for which NSS is certain that all vendors
will have signatures. None of the exploits that were used in section 3.3 will be used as evasion baselines. This
ensures that vendors are not provided with any information on the content of any part of the main NSS exploit
library in advance of the test.
3.4.1
IP Packet Fragmentation
These tests determine the effectiveness of the fragment reassembly mechanism of the DUT.





Fragments from 8 – 32 bytes in size
Ordered, out-of-order, or reverse order fragments
Fragment overlap, favoring new and favoring old data
Interleaved, duplicate, duplicate with or without incrementing DWORD, duplicate packets with random
payload, or duplicate packets scheduled for later delivery
Any combination of the above methods
It is a requirement of the test that the DUT submitted should have all IP fragmentation reassembly options enabled
out of the box.
3.4.2
Stream Segmentation
These tests determine the effectiveness of the stream reassembly mechanism of the DUT.
16
NSS Labs








Test Methodology – Next Generation Firewall v7.0
Segments from 1 – 2048 bytes in size
Ordered, reverse ordered, or out-of-order segments, with “favor old” or “favor new”
Duplicate, duplicate interleaved, duplicate last packet, or overlapping segments
Invalid or NULL TCP control flags
Sequence resync requests, random initial sequence number, or out-of-window sequence numbers
Faked retransmits, PAWS elimination, or segments containing random data
Endianness interchanged
Any combination of the above methods
It is a requirement of the test that the DUT submitted should have all TCP stream reassembly options enabled by
default out of the box.
3.4.3
RPC Fragmentation
Both Sun/ONC RPC and MS-RPC allow the sending application to fragment requests, and all MS-RPC services have a
built-in fragmentation reassembly mechanism.
An attacker can transmit the BIND followed by a single request fragmented over a hundred actual requests with
small fragments of the malicious payload. Alternatively, the attacker could transmit both the BIND and request
fragments in one large TCP segment, thus foiling any signatures that use a simple size check.
NSS uses test tools that combine large writes with many tiny MS-RPC fragments and provide multiple levels of
fragmentation.
These tests determine the effectiveness of the RPC reassembly mechanism of the DUT:





Sun/Open Network Computing (ONC) RPC byte fragmentation
Fragments sent in one or more TCP segments, with or without last fragment
RPC fragmentation may or may not be performed in a single segment
Etc.
Any combination of the above methods
3.4.4
URL Obfuscation
Random URL encoding techniques are employed to transform simple URLs that are often used in pattern-matching
signatures into apparently meaningless strings of escape sequences and expanded path characters, using a
combination of the following techniques:



Escape encoding (% encoding)
Microsoft %u encoding
Path character transformations and expansions ( /./ , //, \ )
These techniques are combined in various ways for each URL tested, ranging from minimal to extreme
transformation (every character transformed). All transformed URLs are verified to ensure they still function as
expected:





URL encoding – Levels 1 – 8 (minimal to extreme)
Premature URL ending
Long URL
Fake parameter
TAB separation
17
NSS Labs




Test Methodology – Next Generation Firewall v7.0
Case sensitivity
Windows\delimiter
Session splicing
Any combination of the above methods
3.4.5
HTML Obfuscation
The ability to recognize malicious HTML documents is becoming increasingly important when protecting the
enterprise. Malicious HTML documents exploit flaws in common web browsers, browser plug-ins, and add-ons to
gain control of the client system and silently install malware such as Trojans, rootkits, and key loggers.
Therefore, it is becoming increasingly important that security products charged with protecting end systems
correctly interpret HTML documents. Many security products use simple pattern-matching systems with very little
semantic or syntactic understanding of the data they are analyzing. This leaves them vulnerable to evasion through
the use of redundant, but equivalent, alternative representations of malicious documents.
This test suite uses a number of malicious HTML documents that are transferred from server to client through the
DUT. Each malicious HTML document is served with a different form of obfuscation, as follows:









UTF-16 and UTF-32 character set encoding (big-endian)
UTF-16 and UTF-32 character set encoding (little-endian)
UTF-7 character set encoding
Chunked encoding (random chunk size, fixed 8-byte chunk size, or chaffing/arbitrary numbers inserted
between chunks)
Compression (deflate)
Compression (Gzip)
Base-64 character set encoding with or without bit shifting or chaffing
HTTP Evader – A publicly available web-based tool will be used to send a malicious test file, which will be
obfuscated in a rare or uncommon web server response.
Any combination of the above methods
The UTF-16 character set specifies a 2-byte sequence for most characters and a 4-byte sequence for the others (a
small percentage). Recoding an HTML document in UTF-16 significantly changes its appearance. A document that
contains just the ASCII subset of characters will appear to have a null byte between every one of the original
characters. There are also two different forms of the UTF-16 encoding, depending on whether the null high byte
comes first (big-endian) or second (little-endian). This test uses big-endian byte ordering
The UTF-32 character set specifies a 4-byte sequence. As with the UTF-16 character set encoding, there are two
variations –big-endian and little-endian. This test uses big-endian byte ordering.
The UTF-7 character set encodes most ASCII characters as themselves. However, in addition to recoding nonEnglish characters as other encodings do, it also recodes many punctuation symbols, including many of the
symbols that are important to the HTML specification. Therefore, recoding an HTML document in UTF-7
significantly changes its appearance
Chunked encoding allows the server to break a document into smaller chunks and transmit them individually. The
server needs only to specify the size of each chunk before it is transmitted and then indicate when the last chunk
has been transmitted. Since chunked encoding intersperses arbitrary numbers (chunk sizes) with the elements of
the original document, it can be used to greatly change the appearance of the original document as observed “on
the wire.” In addition, the server can choose to break the document into chunks at arbitrary points. This makes it
18
NSS Labs
Test Methodology – Next Generation Firewall v7.0
difficult for simple pattern-matching systems to reliably identify the original HTML document from the raw data on
the network.
Per RFC 2616, the HTTP protocol allows the client to request and the server to use several compression methods.
These compression methods not only improve performance in many circumstances, they completely change the
characteristic size and appearance of HTML documents. Furthermore, small changes in the original document can
greatly change the final appearance of the compressed document. This property of these algorithms could be used
to obfuscate hostile content for the purpose of evading detection. The deflate compression method is a Lempel-Ziv
coding (LZ77), specified in RFC 1951. The gzip compression method is specified in RFC 1952.
For each of the above, it is verified that a standard web browser (such as Internet Explorer) is capable of rendering
the results of the evasion.
3.4.6
HTTP Compression
This test will use well-known exploits in a compressed HTTP response. All NGFW devices must decompress via
deflate and/or gzip, and they must inspect all compressed HTTP content.
3.4.7
Payload Encoding
This test attempts to confuse the NGFW into allowing an otherwise blocked exploit to pass using various encoding
options that are standard within the Metasploit Framework and/or other evasion tools. A partial list includes:






x86/call4_dword_xor – This encoder implements a Call+4 Dword XOR Encoder
x86/countdown – This encoder uses the length of the payload as a position-dependent encoder key to
produce a small decoder stub.
x86/fnstenv_mov – This encoder uses a variable-length mov equivalent instruction with fnstenv for getip.
x86/jmp_call_additive – This encoder implements a Jump/Call XOR Additive Feedback Encoder
x86/shikata_ga_nai – This encoder implements a Polymorphic XOR Additive Feedback Encoder. The decoder
stub is generated based on dynamic instruction substitution and dynamic block ordering. Registers are also
selected dynamically.
x86/Veil Evasion Framework – This encoder aggregates various shellcode injection methods across many
evasion implementations. This test will focus on non-encrypted stagers.
3.4.8
FTP/Telnet Evasion
When attempting FTP exploits, it is possible to evade some products by inserting additional spaces and Telnet
control sequences in FTP commands.
These tests insert a range of valid Telnet control sequences that can be parsed and handled by IIS FTP server and
wu-ftpd, and which also conform to Section 2.3 of RFC 959. Control opcodes are inserted at random, ranging from
minimal insertion (only one pair of opcodes), to extreme (opcodes between every character in the FTP command):



Inserting spaces in FTP command lines
Inserting non-text Telnet opcodes – Levels 1 – 8 (minimal to extreme)
Any combination of the above methods
3.4.9
Payload Padding
This test will verify that a DUT is scanning the entire frame for exploit payloads. Various amounts of padding will be
applied to known exploits. The offset of the actual payload will be randomized over several exploits also contained
in the NSS Exploit Library.
19
NSS Labs
3.4.10
Test Methodology – Next Generation Firewall v7.0
Layered Evasions
This test attempts to bypass the DUT by performing any legitimate combination of the evasion techniques
specified in section 3.4. It will be verified that the target machine’s standard network stack is capable of decoding
the evasion correctly while maintaining the exploit viability.
20
NSS Labs
Test Methodology – Next Generation Firewall v7.0
4 Performance
This section measures the performance of the DUT using various traffic conditions that provide metrics for realworld performance. Individual implementations will vary based on usage; however, these quantitative metrics
provide a gauge as to whether a particular DUT is appropriate for a given environment.
4.1 Raw Packet Processing Performance (UDP Throughput)
This test uses UDP packets of varying sizes generated by traffic generation appliances. A constant stream of the
appropriate packet size—with variable source and destination IP addresses transmitting from a fixed source port
to a fixed destination port—is transmitted bi-directionally through each port pair of the DUT. Each packet contains
dummy data, and is targeted at a valid port on a valid IP address on the target subnet. The percentage load and
frames per second (fps) figures across each inline port pair are verified by network monitoring tools before each
test begins. Multiple tests are run and averages are taken where necessary.
This traffic does not attempt to simulate any form of real-world network condition. No TCP sessions are created
during this test, and there is very little for the detection engine to do. However, each vendor will be required to
write a signature to detect the test packets to ensure that they are being passed through the detection engine and
are not being “fast-tracked” from the inbound port to the outbound port.
The goal of this test is to determine the raw packet processing capability of each inline port pair of the DUT, as well
as its effectiveness at forwarding packets quickly in order to provide the highest level of network performance and
with the lowest latency.
4.1.1
64 Byte Packets
Maximum 1,488,000 frames per second per Gigabit of traffic. This test determines the ability of a device to process
packets from the wire under the most challenging packet processing conditions.
4.1.2
128 Byte Packets
Maximum 844,000 frames per second per Gigabit of traffic
4.1.3
256 Byte Packets
Maximum 452,000 frames per second per Gigabit of traffic.
4.1.4
512 Byte Packets
Maximum 234,000 frames per second per Gigabit of traffic. This test provides a reasonable indication of the ability
of a device to process packets from the wire on an “average” network.
4.1.5
1024 Byte Packets
Maximum 119,000 frames per second per Gigabit of traffic.
4.1.6
1514 Byte Packets
Maximum 81,000 frames per second per Gigabit of traffic. This test has been included to demonstrate how easy it
is to achieve good results using large packets. Readers should use caution when taking into consideration those
test results that only quote performance figures using similar packet sizes.
21
NSS Labs
Test Methodology – Next Generation Firewall v7.0
4.2 Latency
The goal of the latency and user response time tests is to determine the effect the DUT has on traffic passing
through it under various load conditions. Test traffic is passed across the infrastructure switches and through all
inline port pairs of the DUT simultaneously (the latency of the basic infrastructure is known and is constant
throughout the tests).
Packet loss and average latency (s) are recorded for each packet size (64, 128, 256, 512, 1024, and 1514 bytes) at
a load level of 90% of the maximum throughput with zero packet loss as previously determined in section 4.1.
4.2.1
64 Byte Frames
Maximum 1,488,000 frames per second per Gigabit of traffic
4.2.2
128 Byte Frames
Maximum 844,000 frames per second per Gigabit of traffic
4.2.3
256 Byte Packets
Maximum 452,000 frames per second per Gigabit of traffic.
4.2.4
512 Byte Packets
Maximum 234,000 frames per second per Gigabit of traffic.
4.2.5
1024 Byte Packets
Maximum 119,000 frames per second per Gigabit of traffic.
4.2.6
1514 Byte Packets
Maximum 81,000 frames per second per Gigabit of traffic.
4.3 Maximum Capacity
The use of traffic generation appliances allows NSS engineers to create true “real-world” traffic at multi-Gigabit
speeds as a background load for the tests.
The goal of these tests is to stress the inspection engine and determine how it handles high volumes of TCP
connections per second, application layer transactions per second, and concurrent open connections. All packets
contain valid payload and address data, and these tests provide an excellent representation of a live network at
various connection/transaction rates.
Note that in all tests, the following critical “breaking points” – where the final measurements are taken – are used:



Excessive concurrent TCP connections – Latency within the NGFW is causing an unacceptable increase in open
connections.
Excessive concurrent HTTP connections – Latency within the NGFW is causing excessive delays and increased
response time.
Unsuccessful HTTP transactions – Normally, there should be zero unsuccessful transactions. Once these appear, it
is an indication that excessive latency within the NGFW is causing connections to time out.
22
NSS Labs
4.3.1
Test Methodology – Next Generation Firewall v7.0
Theoretical Maximum Concurrent TCP Connections
This test is designed to determine the maximum concurrent TCP connections of the DUT with no data passing
across the connections. This type of traffic would not typically be found on a normal network, but it provides the
means to determine the maximum possible concurrent connections figure.
An increasing number of Layer 4 TCP sessions are opened through the device. Each session is opened normally and
then held open for the duration of the test as additional sessions are added up to the maximum possible. Load is
increased until no more connections can be established, and this number is recorded.
4.3.2
Theoretical Maximum Concurrent TCP Connections with Data
This test is identical to section 4.3.1, except that once a connection has been established, data is transmitted (in 1
KB segments). This ensures that the DUT is capable of passing data across the connections once they have been
established.
4.3.3
Maximum TCP Connections per Second
This test is designed to determine the maximum TCP connection rate of the DUT with one byte of data passing
across the connections. This type of traffic would not typically be found on a normal network, but it provides the
means to determine the maximum possible TCP connection rate.
An increasing number of new sessions are established through the DUT, ramped slowly to determine the exact
point of failure. Each session is opened normally, one byte of data is passed to the host, and then the session is
closed immediately. Load is increased until one or more of the breaking points defined earlier is reached.
4.3.4
Maximum HTTP Connections per Second
This test is designed to determine the maximum TCP connection rate of the DUT with a 1-byte HTTP response size.
The response size defines the number of bytes contained in the body, excluding any bytes associated with the
HTTP header. A 1-byte response size is designed to provide a theoretical maximum HTTP connections per second
rate.
Client and server are using HTTP 1.0 without keep-alive, and the client will open a TCP connection, send one HTTP
request, and close the connection. This ensures that all TCP connections are closed immediately upon the request
being satisfied; and thus any concurrent TCP connections will be caused purely as a result of latency the DUT
introduces on the network. Load is increased until one or more of the breaking points defined earlier is reached.
4.3.5
Maximum HTTP Transactions per Second
This test is designed to determine the maximum HTTP transaction rate of the DUT with a 1-byte HTTP response
size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the
HTTP header. A 1-byte response size is designed to provide a theoretical maximum connections per second rate.
Client and server are using HTTP 1.1 with persistence, and the client will open a TCP connection, send 10 HTTP
requests, and close the connection. This ensures that TCP connections remain open until all 10 HTTP transactions
are complete, thus eliminating the maximum connection per second rate as a bottleneck (one TCP connection = 10
HTTP transactions). Load is increased until one or more of the breaking points defined earlier is reached.
23
NSS Labs
4.3.6
Test Methodology – Next Generation Firewall v7.0
Maximum SSL Handshakes per Second
This test is designed to measure a vendor’s handshake performance using TLS v1.2 with various key sizes and
ciphers. This is purely to determine the number of handshakes that can be performed per second. No data will be
transferred after the handshake is sucessfully completed, and the session will be terminated immediately.
4.4 HTTP Capacity with No Transaction Delays
The aim of these tests is to stress the HTTP detection engine and determine how the DUT copes with network
loads of varying average packet size and varying connections per second. By creating genuine session-based traffic
with varying session lengths, the DUT is forced to track valid TCP sessions, thus ensuring a higher workload than for
simple packet-based background traffic. This provides a test environment that is as close to real-world conditions
as it is possible to achieve in a lab environment, while ensuring absolute accuracy and repeatability.
Each transaction consists of a single HTTP GET request ,and there are no transaction delays (i.e., the web server
responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and
address data, and this test provides an excellent representation of a live network (albeit one biased towards HTTP
traffic) at various network loads.
1,000
1,000
1,000
1,000
1,000
45,000
40,000
35,000
800
30,000
25,000
600
20,000
400
15,000
10,000
200
Connections / Second
Megabits per Second
1,000
5,000
0
4.4.1
44 KB
Response
21 KB
Response
10 KB
Response
4.5 KB
Response
1.7 KB
Response
CPS
2,500
5,000
10,000
20,000
40,000
Mbps
1,000
1,000
1,000
1,000
1,000
0
44 KB HTTP Response Size – 2,500 Connections per Second
Maximum 2,500 new connections per second per Gigabit of traffic with a 44 KB HTTP response size—maximum
140,000 packets per second per Gigabit of traffic. With relatively low connection rates and large packet sizes, all
hosts should be capable of performing well throughout this test.
4.4.2
21 KB HTTP Response Size – 5,000 Connections per Second
Maximum 5,000 new connections per second per Gigabit of traffic with a 21 KB HTTP response size—maximum
185,000 packets per second per Gigabit of traffic. With average connection rates and average packet sizes, this is a
good approximation of a real-world production network, and all hosts should be capable of performing well
throughout this test.
24
NSS Labs
4.4.3
Test Methodology – Next Generation Firewall v7.0
10 KB HTTP Response Size – 10,000 Connections per Second
Maximum 10,000 new connections per second per Gigabit of traffic with a 10 KB HTTP response size—maximum
225,000 packets per second per Gigabit of traffic. With smaller packet sizes coupled with high connection rates,
this represents a very heavily used production network.
4.4.4
4.5 KB HTTP Response Size – 20,000 Connections per Second
Maximum 20,000 new connections per second per Gigabit of traffic with a 4.5 KB HTTP response size—maximum
300,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this
is an extreme test for any host.
4.4.5
1.7 KB HTTP Response Size – 40,000 Connections per Second
Maximum 40,000 new connections per second per Gigabit of traffic with a 1.7 KB HTTP response size—maximum
445,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this
is an extreme test for any host.
4.5 Application Average Response Time: HTTP
Test traffic is passed across the infrastructure switches and through all inline port pairs of the DUT simultaneously
(the latency of the basic infrastructure is known and is constant throughout the tests). The results are recorded at
each response size (44 KB, 21 KB, 10 KB, 4.5 KB, and 1.7 KB HTTP responses) load level of 90% of the maximum
throughput with zero packet loss as previously determined in section 4.4, section 4.7, and section 4.8.
4.6 HTTP Capacity with Transaction Delays
Typical user behavior introduces delays between requests and reponses, for example, as users read web pages and
decide which links to click next. This following tests are identical to the tests in section 4.4 except that these
include a 5-second delay in the server response for each transaction. This has the effect of maintaining a high
number of open connections throughout the test, thus forcing the sensor to utilize additional resources to track
those connections.
4.6.1
21 KB HTTP Response with Delay
Maximum 5,000 new connections per second per Gigabit of traffic with a 21 KB HTTP response size—maximum
185,000 packets per second per Gigabit of traffic. 5-second transaction delay resulting in an additional 50,000 open
connections per Gigabit over the test described in section 4.4.2. With average connection rates and average packet
sizes, this is a good approximation of a real-world production network, and all sensors should be capable of
performing well throughout this test.
4.6.2
10 KB HTTP Response with Delay
Maximum 10,000 new connections per second per Gigabit of traffic with a 10 KB HTTP response size—maximum
225,000 packets per second per Gigabit of traffic. Repeated with background traffic loads of 25%, 50%, 75%, and
100% of maximum throughput of NGFW. 5-second transaction delay resulting in an additional 100,000 open
connections over the test described in section 4.4.3. With large average packet sizes coupled with very high
connection rates, this represents a very heavily used production network, and is a strenuous test for any sensor.
25
NSS Labs
Test Methodology – Next Generation Firewall v7.0
4.7 HTTP Capacity with HTTP Persistent Connections
The aim of these tests is to determine how the DUT copes with network loads of varying average packet size,
varying connections per second while inspecting all traffic. By creating genuine session-based traffic with varying
session lengths, the DUT is forced to track valid TCP sessions, thus ensuring a higher workload than for simple
packet-based background traffic. This provides a test environment that is as close to real-world conditions as it is
possible to achieve in a lab environment, while ensuring absolute accuracy and repeatability.
This test will use HTTP persistent connections, with each TCP connection containing 10 HTTP GETs and associated
responses. All packets contain valid payload (a mix of binary and ASCII objects) and address data, and this test
provides an excellent representation of a live network at various network loads. The stated response size is the
total of all HTTP responses within a single TCP session.
4.7.1
250 Connections per Second
This test will simulate HTTP persistent connections, each containing a total of 10 HTTP GET/responses of various
sizes. The total HTTP response size for each persistent connection will be equal to four megabits, transmitted over
a maximum of 250 connections per second for each gigabit of traffic.
4.7.2
500 Connections per Second
This test will simulate HTTP persistent connections, each containing a total of HTTP 10 GET/responses of various
sizes. The total HTTP response size for each persistent connection will be equal to two megabits, transmitted over
a maximum of 500 connections per second for each gigabit of traffic.
4.7.3
1000 Connections per Second
This test will simulate HTTP persistent connections, each containing a total of 10 HTTP GETs/responses of various
sizes. The total HTTP response size for each persistent connection will be equal to one megabit, transmitted over a
maximum of 1000 connections per second, for each gigabit of traffic.
4.8 HTTPS Capacity with No Transaction Delay HTTP Persistent Connections
The aim of these tests is to stress the HTTPS detection engine and determine how the DUT copes with network
loads of varying average packet size and varying connections per second while decrypting and inspecting all traffic.
SSL/TLS inspection must be enabled for this test. SSL/TLS inspection can be disabled for the remainder of
performance testing for NGFW v7.0; however, in the next version of this Test Methodology, all vendors will be
required to enable SSL/TLS inspection for all testing. By creating genuine session-based traffic with varying session
lengths, the DUT is forced to track valid TCP sessions, thus ensuring a higher workload than for simple packetbased background traffic. This provides a test environment that is as close to real-world conditions as it is possible
to achieve in a lab environment, while ensuring absolute accuracy and repeatability.
This test will use HTTP persistent connections, with each TCP connection containing ten HTTP GETs and associated
responses. All packets contain valid payload (a mix of binary and ASCII objects) and address data, and this test
provides an excellent representation of a live network (albeit one biased towards HTTPS traffic) at various network
loads. The stated response size is the total of all HTTP responses within a single TCP session.
26
NSS Labs
4.8.1
Test Methodology – Next Generation Firewall v7.0
250 Connections per Second
This test will simulate an HTTP persistent connection containing a total of 10 HTTP GETs/responses of various sizes.
The total HTTP response size for each persistent connection will be equal to four megabits, transmitted over a
maximum of 250 connections per second for each gigabit of traffic. In addition, the session is encrypted in order to
ascertain performance degradation (the session is not encrypted in test 4.7.1). This test will use a 2048-byte
certificate and key size.
4.8.2
500 Connections per Second
This test will simulate an HTTP persistent connection containing a total of 10 HTTP GETs/responses of various sizes.
The total HTTP response size for each persistent connection will be equal to two megabits, transmitted over a
maximum of 500 connections per second for each gigabit of traffic. In addition, the session is encrypted in order
to ascertain performance degradation (the session is not encrypted in test 4.7.2). This test will use a 2048-byte
certificate and key size.
4.8.3
1000 Connections per Second
This test will simulate an HTTP persistent connection containing a total of 10 HTTP GETs/responses of various sizes.
The total HTTP response size for each persistent connection will be equal to one megabit, transmitted over a
maximum of 1000 connections per second for each gigabit of traffic. In addition, the session is encrypted in order
to ascertain performance degradation (the session is not encrypted in test 4.7.3). This test will use a 2048-byte
certificate and key size.
4.9 “Real-World” Traffic Mixes
Where previous tests provide a pure HTTP environment with varying connection rates and average packet sizes,
the goal of this test is to simulate a real-world environment by introducing additional protocols and real content
while still maintaining a precisely repeatable and consistent background traffic load.
The result is a background traffic load that is closer to what may be found on a heavily-utilized “normal”
production network.
4.9.1
“Real-World” Protocol Mix (Enterprise Perimeter)
Traffic is generated across the DUT comprising a protocol mix typical of that seen in an enterprise perimeter:
HTTP
Simulated HTTPS
BitTorrent
FTP
Facebook
18.69%
9.66%
10.82%
5%
5.8%
Gmail
9.66%
Yahoo Mail
9.66%
Twitter
3.09%
Amazon S3
7.73%
SMTP
1.93%
Gtalk
4.64%
YouTube
AOL Chat
1.16%
SSH, Oracle DB
11.59%
< 1%
27
NSS Labs
4.9.2
Test Methodology – Next Generation Firewall v7.0
“Real-World” Protocol Mix (Financial)
Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large financial institution:
HTTP
31%
FIX
7.13%
Simulated HTTPS
6.3%
RDP
1.58%
IMAP
1.5%
LDAP
1.58%
POP3
2.4%
YouTube
SMTP
1.6%
Facebook
SMB
7.13%
Citrix
5.5%
4.9.3
SIP
SSH, TFTP, Syslog, RTSP, DB2, LPD,
NetBIOS, NTP, Telnet, SNMPv2, SCCP
11.88%
5.66
1%
<1%
“Real-World” Protocol Mix (US Mobile Carrier)
Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large US mobile carrier:
HTTP
31.8%
iTunes Music Store
SIP
2.6%
Facebook Mobile
AOL Instant Messenger
6.4%
FaceTime Video Call
1%
1.2%
5.3
SMTP Email
10.62%
Pandora iPhone
< 1%
POP3 Email
4.3%
YouTube Mobile
10.6%
IMAPv4
4.2%
RTSP
2.8%
NNTP
4.2%
Google Play Store
< 1%
1%
Instagram Mobile
1%
iTunes App Store
4.9.4
“Real-World” Protocol Mix (European Mobile Carrier)
Traffic is generated across the DUT comprising a protocol mix typical of that seen in a European mobile carrier:
HTTP
50.6%
Google Maps
2.34
H.323
2.4%
Google Play Store
1.9%
POP3 Email
4.2%
Skype v5
2.3%
IMAP Email
2.81
iTunes Mobile Music
Mobile Video
9.8%
Facebook Mobile
2.8%
SMTP Email
4.2%
YouTube Mobile
11.3%
BBC iPlayer Radio
4.2%
1.41%
28
NSS Labs
4.9.5
Test Methodology – Next Generation Firewall v7.0
“Real-World” Internal Segmentation Mix
Traffic is generated across the DUT compromising a protocol mix typical of that seen in internally segmented
deployments:
NFSv3
2.4%
SIP voice call
7.3%
Outlook Calendar/Mail/Web Mail
5.8%
SSH
2.4%
MS Exchange
17%
Windows Live Mail
< 1%
BitTorrent Enterprise
1.5%
Facebook
1%
Enterprise FTP
< 1%
Skype
6%
Oracle
1.2%
YouTube
17%
Pinterest
< 1%
HTTP Chrome
21%
RSS Feed
2.4%
HTTP Firefox
12%
4.10 IPsec Site-to-Site VPN
A critical function of an NGFW is the encryption of traffic between remote networks. IPsec VPN is the dominant
technology for securing site-to-site connections. A vendor’s default out-of-box policy must account for inspection
over the VPN tunnel. Due to its increased simplicity and efficiency, Internet Key Exchange version 2 (IKEv2) will be
utilized to build the security association—where it is supported. If an NGFW vendor does not support IKEv2 at the
time of testing, IKEv1 can also be used, and this will be noted in the test report.
Preferred configuration for IPSec site-to-site VPN:





IKEv2
Pre-shared key
IKE & ESP encryption – AES256
IKE & ESP integrity – SHA1
IKE Diffie-Hellman – Group five
4.10.1
21 KB HTTP Response Size
This test will measure the speed of which encrypted traffic can pass through the NGFW while inspecting all
content. Maximum 5,000 new connections per second per Gigabit of traffic with a 21 KB HTTP response size—
maximum 185,000 packets per second per Gigabit of traffic. A well-known exploit will accompany the HTTP 21 KB
traffic to verify that the vendor is inspecting traffic over the VPN tunnel.
29
NSS Labs
Test Methodology – Next Generation Firewall v7.0
5 Stability and Reliability
Long-term stability is particularly important for an inline device, where failure can produce network outages. These
tests verify the stability of the DUT along with its ability to maintain security effectiveness while under normal load
and while passing malicious traffic. Products that are not able to sustain legitimate traffic (or that crash) while
under hostile attack will not pass.
The DUT is required to remain operational and stable throughout these tests, and to block 100% of previously
blocked traffic, raising an alert for each. If any prohibited traffic passes successfully, caused by either the volume of
traffic or by the DUT failing open for any reason, this will result in a FAIL.
5.1 Blocking Under Extended Attack
The DUT is exposed to a constant stream of security policy violations over an extended period of time. The device
is configured to block and alert, and thus this test provides an indication of the effectiveness of both the blocking
and alert handling mechanisms.
A continuous stream of security policy violations mixed with legitimate traffic is transmitted through the DUT for 8
hours at a maximum of 100 Mbps, with no additional background traffic. This is not intended as a stress test in
terms of traffic load (covered in the previous section); it is merely a reliability test in terms of consistency of
blocking performance.
The DUT is expected to remain operational and stable throughout this test and to block 100% of recognizable
violations, raising an alert for each. If any recognizable policy violations are passed, caused by either the volume of
traffic or the DUT failing open for any reason, this will result in a FAIL.
5.2 Passing Legitimate Traffic under Extended Attack
This test is identical to test 5.1 where the external interface of the DUT is exposed to a constant stream of exploits
over an extended period of time.
The DUT is expected to remain operational and stable throughout this test, and to pass most/all of the legitimate
traffic. If an excessive amount of legitimate traffic is blocked throughout this test, caused by either the volume of
traffic or the DUT failing for any reason, this will result in a FAIL.
5.3 Behavior of the State Engine Under Load
This test determines whether the DUT is capable of preserving state across a large number of open connections
over an extended time period.
At various points throughout the test (including after the maximum has been reached), it is confirmed that the
DUT is still capable of inspecting and blocking traffic that is in violation of the currently applied security policy,
whilst confirming that legitimate traffic is not blocked (perhaps as a result of exhaustion of the resources allocated
to state tables). The DUT must be able to apply policy decisions effectively based on inspected traffic at all load
levels.
30
NSS Labs
5.3.1
Test Methodology – Next Generation Firewall v7.0
Attack Detection/Blocking – Normal Load
This test determines if the DUT is able to detect and block policy violations as the number of open sessions reaches
75% of the maximum determined in Test 4.3.1.
5.3.2
State Preservation – Normal Load
This test determines if the sensor maintains the state of pre-existing sessions as the number of open sessions
reaches 75% of the maximum determined in Test 4.3.1.
A legitimate HTTP session is opened, and the first packet of a two-packet exploit is transmitted. As the number of
open connections approaches the maximum, the initial HTTP session is then completed with the second half of the
exploit, and the session is closed. If the sensor is still maintaining state on the original session, the exploit will be
recorded. If the state tables have been exhausted, the exploit string will be seen as a non-stateful attack and will
thus be ignored. Both halves of the exploit are required to trigger an alert. A product will fail the test if it fails to
generate an alert after the second packet is transmitted, or if it raises an alert on either half of the exploit on its
own.
5.3.3
Pass Legitimate Traffic – Normal Load
This test ensures that the DUT continues to pass legitimate traffic as the number of open sessions reaches 75% of
the maximum determined in Test 4.3.1.
5.3.4
State Preservation – Maximum Exceeded
This test determines whether the DUT maintains the state of pre-existing sessions as the number of open sessions
exceeds the maximum determined in Test 4.3.1. The method of execution is identical to Test 5.3.2.
5.3.5
Drop Legitimate Traffic – Maximum Exceeded
This test ensures that the DUT continues to drop all traffic as the number of open sessions exceeds the maximum
determined in Test 4.3.1.
Note: If a DUT allows traffic to “leak” due to the way it expires old connections, this will result in an automatic fail
for the entire test.
5.4 Protocol Fuzzing and Mutation
This test stresses the protocol stacks of the DUT by exposing it to traffic from various protocol randomizer and
mutation tools. Several of the tools in this category are based on the ISIC test suite and other well-known test
tools/suites.
Traffic load is a maximum of 350 Mbps and 60,000 packets per second. Results are presented as a simple
PASS/FAIL—the DUT is expected to remain operational and capable of detecting and blocking exploits throughout
the test.
5.5 Power Fail
Power to the DUT is cut whilst passing a mixture of legitimate and disallowed traffic. Firewalls should always be
configured to fail closed—no traffic should be passed once power has been cut.
31
NSS Labs
Test Methodology – Next Generation Firewall v7.0
5.6 Persistence of Data
The DUT should retain all configuration data, policy data, and locally logged data once it has been restored to
operation following power failure.
32
NSS Labs
Test Methodology – Next Generation Firewall v7.0
6 Total Cost of Ownership and Value
Implementation of security solutions can be complex, with several factors affecting the overall cost of deployment,
maintenance, and upkeep. All of the following should be considered over the course of the useful life of the
solution:

Product Purchase – The cost of acquisition

Product Maintenance – The fees paid to the vendor, including software and hardware support, maintenance,
and other updates

Installation – The time required to take the device out of the box, configure it, put it into the network, apply
updates and patches, and set up desired logging and reporting

Upkeep – The time required to apply periodic updates and patches from vendors, including hardware,
software, and other updates
33
NSS Labs
Test Methodology – Next Generation Firewall v7.0
Appendix A: Change Log
Version 7.0 – 15 July, 2016








Added requirements for false positive testing to section 3.3.
Added the Veil Evasion Framework to section 3.4.6
Added sections 4.7, “HTTP Capacity with HTTP Persistent Connections,” and 4.8 “HTTPS Capacity with HTTP
Persistent Connections.”
In section 4.7, noted that SSL/TLS inspection will only be required for this particular section in NGFW v7.0. For
the next methodology release, inspection will be required for all NSS NGFW testing.
Changed all occurrences of “Anti-X” to “Anti-malware.”
Added application protocol percentages to all real-world tests in section 4.9
Removed Datacenter and Education traffic mixes from the real-world testing in section 4.9. Added Internal
Segmentation traffic mix.
Added section 4.10 for IPSec functionality testing, which includes language around IKEv1 or IKEv2.
Version 6.0 – 24 February, 2015













Removed IPv6 section
Removed sections referencing centralized management
Removed sections referencing Active Directory integration and NGFW DB
Clarified section 3.3
Clarified section 3.4
Moved information regarding Live Exploit Testing from section 1.2 to section 3.4
Added Arbitrary Code Execution to list of fail criteria in section 3.4.1
Removed sections referencing SMB & NetBIOS evasions
Clarified section 4.3
Removed references to average packet size in sub-sections of 4.4 and 4.6
Removed section referencing Redundancy
Removed section referencing High Availability
Adjusted contact information and legal information
Version 5.4 – 30 September, 2013




Capitalized SYN in subsection title
Removed SSL section
Clarified section 3.4.4
Removed references to specific test tools
Version 5.3 – 04 December, 2012
No Change Log available. Change Log Appendix added with version 5.4.
34
NSS Labs
Test Methodology – Next Generation Firewall v7.0
Contact Information
NSS Labs, Inc.
206 Wild Basin Road
Building A, Suite 200
Austin, TX 78746 USA
[email protected]
www.nsslabs.com
This and other related documents available at: www.nsslabs.com. To receive a licensed copy or report misuse,
please contact NSS Labs.
© 2016 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval
system, emailed or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. (“us” or “we”).
Please read the disclaimer in this box because it contains important information that binds you. If you do not agree to these
conditions, you should not read the rest of this report but should instead return the report immediately to us. “You” or “your”
means the person who accesses this report and any entity on whose behalf he/she has obtained this report.
1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it.
2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed.
All use of and reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses
of any nature whatsoever arising from any error or omission in this report.
3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED
BY US. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT
DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE
POSSIBILITY THEREOF.
4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or
software) tested or the hardware and/or software used in testing the products. The testing does not guarantee that there are
no errors or defects in the products or that the products will meet your expectations, requirements, needs, or specifications, or
that they will operate without interruption.
5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned
in this report.
6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of
their respective owners.
35