CSE 4905
Firewalls & Intrusion Detection
Systems
8-1
Protecting an organization’s network





Prevent, detect and stop denial of service attacks
Detect attempted and successful break-ins
Detect and stop worms, botnets …
Protect servers that need to be accessed from outside
…
8-2
Firewalls
firewall
isolates organization’s internal net from larger Internet,
allowing some packets to pass, blocking others
administered
network
public
Internet
Firewall: combination of hardware & software system
8-3
Firewall design goals
All traffic from inside to outside, and vice versa,
must pass through the firewall
 Only authorized traffic as defined by the local
security policy will be allowed to pass
 The firewall itself is immune to penetration (e.g.,
hardened system with secure operating system)

8-4
Firewall access policy




lists the types of traffic authorized to pass through the
firewall
Includes address ranges, protocols, applications and
content types
Should be developed from the organization’s
information security risk assessment and policy
Based on a broad specification of which traffic types the
organization needs to support
 Then refined to detail the filter elements which can then be
implemented within an appropriate firewall topology
8-5
Types of firewalls
Stateless packet filtering firewall
 Stateful inspection firewall
 Application-level firewall

8-6
Stateless packet filtering
Should arriving
packet be allowed in?
Departing packet let
out?


internal network connected to Internet via router firewall
router filters packet-by-packet, decision to forward/drop
packet based on:
 source IP address, destination IP address
 TCP/UDP source and destination port numbers
 ICMP message type
 TCP SYN and ACK bits
 …
8-7
Stateless packet filtering: example


example 1: block incoming and outgoing datagrams with
IP protocol field = 17 and with either source or dest
port = 23
 result: all incoming, outgoing UDP flows and telnet
connections are blocked
example 2: block inbound TCP segments with ACK=0.
 result: prevents external clients from making TCP
connections with internal clients, but allows internal
clients to connect to outside.
8-8
Stateless packet filtering: more examples
Policy
Firewall Setting
No outside Web access.
Drop all outgoing packets to any IP
address, port 80
No incoming TCP connections,
except those for institution’s
public Web server only.
Drop all incoming TCP SYN packets
to any IP except 130.207.244.203,
port 80
Prevent Web-radios from eating
up the available bandwidth.
Drop all incoming UDP packets except DNS and router broadcasts.
Prevent your network from being
used for a smurf DoS attack.
Drop all ICMP packets going to a
“broadcast” address (e.g.
130.207.255.255).
Prevent your network from being
tracerouted
Drop all outgoing ICMP TTL expired
traffic
8-9
Access Control Lists
 ACL: table of rules, applied top to bottom to incoming
packets: (action, condition) pairs
action
source
address
dest
address
allow
222.22/16
outside of
222.22/16
allow
outside of
222.22/16
222.22/16
outside of
222.22/16
allow
222.22/16
allow
outside of
222.22/16
222.22/16
deny
all
all
protocol
source
port
dest
port
flag
bit
TCP
> 1023
80
TCP
80
> 1023
ACK
UDP
> 1023
53
---
UDP
53
> 1023
----
all
all
all
all
any
8-10
Stateless packet filtering: netsec




Only ssh is allowed to netsec.engr.uconn.edu for
external hosts
Everything is allowed for internal hosts (i.e.,
within UConn)
Can the above access policy be realized using
firewall? How?
IP address of netsec.engr.uconn.edu:
137.99.3.201
8-11
Weaknesses of packet filtering

Do not prevent application-specific attacks
 For example, if there is a buffer overflow in the Web
server, firewall will not block an attack string


IP spoofing: router can’t know if data “really” comes
from claimed source
Vulnerable to misconfiguration
8-12
Stateless filtering is not enough

In TCP connections, ports with numbers less than
1024 are permanently assigned to servers
 20, 21 - FTP, 23 - telnet, 25 - SMTP, 80 - HTTP…

Clients use ports numbered from 1024 to 65535
 They must be available for clients to receive responses

What should a firewall do if it sees an outgoing
request to some client’s port 5151?
 This could be a server’s response in a previously established
connection, OR it could be malicious traffic
 Can’t tell without keeping state for each connection
8-13
Stateless filtering is not enough - II

admits packets that “make no sense,” e.g., dest
port = 80, ACK bit set, even though no TCP
connection established:
action
allow
source
address
dest
address
outside of
222.22/16
222.22/16
protocol
source
port
dest
port
flag
bit
TCP
80
> 1023
ACK
8-14
Stateful packet filtering
 track
status of every TCP connection
 track connection setup (SYN), teardown (FIN):
determine whether incoming, outgoing packets
“makes sense”
 timeout inactive connections at firewall: no longer
admit packets
8-15
Stateful packet filtering

ACL augmented to indicate need to check connection
state table before admitting packet
action
source
address
dest
address
proto
source
port
dest
port
allow
222.22/16
outside of
222.22/16
TCP
> 1023
80
allow
outside of
222.22/16
TCP
80
> 1023
ACK
allow
222.22/16
UDP
> 1023
53
---
allow
outside of
222.22/16
222.22/16
UDP
53
> 1023
----
deny
all
all
all
all
all
all
222.22/16
outside of
222.22/16
flag
bit
check
conxion
any
x
x
8-16
Application gateways


filter packets on
application data as well as
on IP/TCP/UDP fields.
example: allow select
internal users to telnet
outside
host-to-gateway
telnet session
application
gateway
router and filter
gateway-to-remote
host telnet session
1. require all telnet users to telnet through gateway.
2. for authorized users, gateway sets up telnet connection to
dest host. Gateway relays data between 2 connections
3. router filter blocks all telnet connections not originating
from gateway.
8-17
Limitations of firewalls, gateways



IP spoofing
if multiple apps. need
special treatment, each
has own app. gateway
client software must
know how to contact
gateway.
 e.g., must set IP
address of proxy in
web browser

Don’t solve many real
problems
 Buggy software (think
buffer overflow exploits)
 Bad protocol design (think
WEP in 802.11b)


Generally don’t prevent
denial of service
Don’t prevent insider
attacks
8-18
Intrusion detection systems (IDS)

packet filtering:
 operates on TCP/IP headers only
 no correlation check among sessions

IDS: intrusion detection system
 deep packet inspection: look at packet contents (e.g.,
check character strings in packet against database of
known virus, attack strings)
 examine correlation among multiple packets
• port scanning
• network mapping
• DoS attack
8-19
Definitions from RFC 2828
(Internet Security Glossary)

Security Intrusion: A security event, or a combination
of multiple security events, that constitutes a security
incident in which an intruder gains, or attempts to gain,
access to a system (or system resource) without
having authorization to do so.

Intrusion Detection: A security service that monitors
and analyzes system events for the purpose of finding,
and providing real-time or near real-time warning of,
attempts to access system resources in an
unauthorized manner.
8-20
Types of IDS

Host-based intrusion detection
 Monitors the characteristics of a single host for suspicious
activity

Network-based intrusion detection
 Monitors network traffic and analyzes network, transport,
and application protocols to identify suspicious activity

Three logic components in IDS
 Sensors: collect data
 Analyzers: determine if intrusion has occurred
 User interface: view output or control system behavior
8-21
Assumption of intrusion detection



Behavior of intruders
differ from that of
legitimate users
In practice, their
behavior can overlap
What to hope for: IDS
with high detection
rate and low false
alarm rate
8-23
IDS requirements









Run continually w/ minimal human intervention
Fault tolerant
Resist subversion
Impose a minimal overhead on the system that it’s running
Configured according to system security policies
Adapt to changes in systems and users
Scale to monitor large numbers of systems
Provide graceful degradation of service
Allow dynamic reconfiguration
8-24
Analysis approaches

Anomaly detection
 Collecting data relating to the behavior of legitimate users over a
period of time
 Current observed behavior is analyzed to determine whether
this behavior is that of a legitimate user or not
 Can detect unknown attacks
 Statistical, knowledge based or machine learning techniques

Signature or heuristic detection
 Use know malicious data patterns (signatures) or attack rules
(heuristics)
 Can only detect known attacks
8-25
Host-based intrusion detection
system (HIDS)



Security software on vulnerable or sensitive systems (e.g.,
database servers or administrative systems)
Can detect both internal and external intrusions
Sensor data





System call traces
Audit (log file) records: user activity
File integrity checksums (cryptographic checksums)
Registry access
Techniques
 Anomaly detection
 Signature or heuristic rules
8-26
Network-based IDS: types of network
sensors

Inline sensor: inserted into
network segment
 Stand-alone or with another
network device (e.g., firewall or
LAN switch)
 Traffic pass through sensor
 Can detect and block an attack;
may cause extra latency

Passive sensors
 Tap to network transmission
medium
Passive sensor
8-28
Network-based IDS: sensor placement
8-29
Network-based IDS: sensor placement II

In external firewall
 See all incoming (filtered by firewall) and outgoing traffic
 Highlight problems with firewall policy or performance
 See attacks that might target service servers (e.g., web or mail
servers)
 Recognize outgoing traffic from compromised servers

Outside external firewall
 See all incoming and outgoing traffic (filtered by firewall)
 Document # and type of attacks targeting the network
 High processing burden

Inside network (tuned to specific attack types)
 Protect major backbone networks (internal servers and database
resources) in organization
 Protect critical subsystems
8-30
Network-based IDS: techniques
Many commercial products
 many techniques

 Signature detection
• On different protocols (application, transport, IP,
ICMP, etc.)
 Anomaly detection
• DoS attacks
• Network scanning
• worms
 Stateful protocol analysis
8-31
Example system: Snort



Open-source, highly configurable and portable
host-based or network-based IDS
Light-weight IDS
 Small amount of memory and CPU, easily configured by
experienced administrators

Functions
 Real-time packet capture
 Protocol analysis
 Content search and matching

Can be configured to be inline sensor (intrusion
prevention) or passive sensor (intrusion detection)
8-32
Snort: Rule-based detection engine
8-33
Snort: example
Alert tcp $EXTERNAL_NET any -> $HOME_NET any \
(msg: “SCAN SYN FIN” flags: SF 12;\
reference: arachnids, 198; \
classtype: attempted-recon;)


Rule options in the format of
 option-keyword: option-arguments
Example rule options
 msg: defines the message to be sent when a packet
generates an event
 flags: test the TCP flags for specified settings
 reference: defines a link to an external attack
identification system, which provides additional info.
 classtype: what type of attack the packet attempted 8-34
Why intrusion detection is hard?

Signature matching
 Detecting attack strings is hard, even if signature is
known
…

Anomaly detection




What is anomaly?
Training is difficult
Can have high false positives
…
8-35
Detecting attack strings is hard
[Vitaly Shmatikov]

Suppose want to detect “USER root” in packet stream

Scanning for it in every packet is not enough
 Attacker can split attack string into several packets; this will
defeat stateless NIDS

Recording previous packet’s text is not enough
 Attacker can send packets out of order

Full reassembly of TCP state is not enough
 Attacker can use TCP tricks so that certain packets are
seen by NIDS but dropped by the receiving application
• Manipulate checksums, TTL (time-to-live), fragmentation
8-36
TCP Attacks on NIDS
[Vitaly Shmatikov]
Insertion attack
S
t
R
U
Insert packet with
bogus checksum
S
R
E
R
NIDS
TTL attack
10 hops
S
o
o
t
Dropped
8 hops
U
TTL=20
r
S
E
R
r
o
o
t
TTL=12
Short TTL to
ensure this packet
doesn’t reach
destination
t
TTL=20
NIDS
Dropped
(TTL expired)
8-37
Anomaly detection is hard

Training is difficult
 Lack of training data with real attacks
 Network traffic is very diverse, the definition of “normal” is
constantly evolving
• What is the difference between a flash crowd and a
denial of service attack?


Protocols are finite-state machines, but current state
of a connection is hard to see from network
False positive rate can be high
 False identifications are very costly because sys admin will
spend many hours examining evidence
8-38
Intrusion detection errors

False negatives: attack is not detected
 Big problem in signature-based misuse detection

False positives: harmless behavior is classified as an
attack
 Big problem in statistical anomaly detection


All intrusion detection systems (IDS) suffer from
errors of both types
Which is a bigger problem?
 Attacks are fairly rare events, thus IDS often suffer from the
base-rate fallacy
8-39
Conditional Probability



Suppose two events A and B occur with probability
Pr(A) and Pr(B), respectively
Let Pr(AB) be probability that both A and B occur
What is the conditional probability that A occurs
assuming B has occurred?
Pr(A | B) =
Pr(AB)
Pr(B)
8-40
Base-Rate Fallacy

1% of traffic is SYN floods; IDS accuracy is 90%
 IDS classifies a SYN flood as attack with prob. 90%, classifies
a valid connection as attack with prob. 10%

What is the probability that a connection flagged by
IDS as a SYN flood is actually valid?
Pr(alarm | valid)  Pr(valid)
Pr(valid | alarm) =
Pr(alarm)
=
Pr(alarm | valid)  Pr(valid)
Pr(alarm | valid)  Pr(valid) + Pr(alarm | SYN flood)  Pr(SYN flood)
0.10  0.99
=
0.10  0.99 + 0.90  0.01
= 92% chance raised alarm
is false!!!
8-42
Unified threat management (UTM)
system

Definition (by IDC)
 Products that include multiple security features integrated into
one box
 Be able to perform network firewalling, network intrusion
detection and prevention and gateway anti-virus
 All of the capabilities need not be used concurrently, but the
functions must exist inherently

Performance is big issue
 How to keep up with throughput and reduce latency?
 Typical throughput loss of current commercial products (50%,
2006)
8-43
UTM architecture
8-44
Network Function Virtualization (NFV)

initiative to virtualize the network services
that are now being carried out by proprietary,
dedicated hardware
 software implementations of network functions
for
• firewall
• Intrusion detection system
•…
 Many implementations of NFV rely on Software
Defined Networking (SDN)
8-45
Summary

Firewall
 Type of firewalls
 pros and cons of each type
What is security intrusion?
 Intrusion detection

 Basic assumption and techniques
 Host-based and network-based IDS
Challenges of intrusion detection
 Unified threat management system

8-46
References
Chapter 8.9 “Operational Security: Firewall and
Intrusion Detection Systems” in Kurose &
Ross’s book
 “Insertion, Evasion and Denial of Service: Eluding
Network Intrusion Detection” by Ptacek and
Newman

8-47