Defence R&D Canada - Ottawa
Technical Memorandum
DRDC Ottawa TM 2003-139
November 2003
A Finite State Machine Model of TCP
Connections in the Transport Layer
J. Treurniet and J.H. Lefebvre
Defence R&D Canada - Ottawa
Mike Hsiao, 20080822
Outline


Summary and Introduction
Theory
–
–
–


Implementation of the Algorithm
Results and Discussion
–
–
–
–

2
Transmission Control Protocol
TCP Finite State Machine Model
Detection of Anomalies
Network Management
Security
Research
False Positives
Conclusion and Future Work
Summary
Background – RFC & implementation

The protocols employed on computer
networks are intended to be well-defined via
Request for Comments (RFC) documents.
–
3
In reality, there is enough freedom in the RFC
specifications of the protocols that they can vary
widely from one implementation to the next.
Additional information: http://lxr.linux.no/
Summary
Diagram of a “textbook” TCP connection
Background – TCP

TCP is a reliable protocol, and its behavior follows a
pattern that is somewhat predictable.
–
The “flags” convey the commands that carry a connection
through these stages.
SYN
SYN-ACK
Three-way handshake
ACK
…
Data exchange
FIN
FIN-ACK
Closing
4
ACK
Summary
Background – FSM

The authors model a TCP connection as an FSM,
–
–
5
defining the states of a FSM to reflect the stages of a
connection and
using the flags as the events that bring about transitions
among states.
The TCP FSM proposed by the authors
Summary
Background – Anomaly detection

To use the model for strict anomaly detection, a
failure state is introduced to indicate the occurrence
of a disallowed event or an attempted illegal
transition.
–
When the time-ordered flags of a TCP connection
is input to the FSM,


–
6
if the connection enters a failure state or
otherwise does not complete,
State X
the connection is flagged as anomalous.
Event Y
Failure State
Introduction
Background

The TCP FSM can be used to detect some network
management issues and network security events,
and it can also be used as a research tool to study
the behavior of TCP on the Internet.

”“Strict anomaly detection“
” defines a set of permitted
events and detects activity which represents
exceptions to those events.
–
7
TCP, FSM, anomalous
Introduction
A snapshot of TCP anomalies

Impart information to
–
Network management


–
Network security


8
Unresponsive hosts
Behaviors that lead to resource consumption
Scanning activity
The appeal of the model is in its simplicity
and it utility in a wild range of applications
Theory
Transmission Control Protocol
9
Note: Only certain flag combination are valid.
Theory
Transmission Control Protocol
In an ungraceful termination,
the connection is torn down by
transmission of a reset (R)
from either the client or the server.
Graceful
Termination
10
On receiving an acceptable reset,
if the receiving TCP is in
1) a non-synchronized state,
=> returns to LISTEN;
2) a synchronized states,
=> aborts the connection
=> informs its user.
Theory
In Practice

11
in most Berkeley-derived systems
In practice, stages of a connection are only allowed
to exist for a set amount of time, to free up resources
that aren't being used.
twice the max amount of time in RFC;
some implement is one minute.
Theory
TCP client and server
12

The traditional state transition diagram for TCP is a
graphical representation of two separate but related
finite state machines, one for the client and one for
the server.

The authors have simplified the traditional TCP state
transition diagram to remove the differentiation of
client and server.
Theory
Server
Client
Attempt to
close socket
Receive close
request
Simultaneous
close
13
They proposed a 7-state TCP FSM.
14
Theory
6 events in the proposed TCP FSM
(e.g., SYN+FIN)
15
Such packets are either “crafted” or appear as a result of corruption of the packet.
Theory
almost 10% of all SYN packets are retransmitted
16
Theory
Detection of Anomalies
17

If the final state of the sequence is not Closed
(C2), then the connection is flagged as
anomalous.

For streams that terminate in an allowed
state other than Closed, we denote the event
as “timeout“
” to signify that the stream has
ended but the connection has not been
closed.
Theory
Example of Anomalies

Denial of Service (DoS)
–
–

TCP half-open scan
–

18
many H2 | timeout events for one host
X | timeout or C1 | timeout
H1 | RAPU, H1 | timeout, H2 | RAPU or H2 | timeout
Scans for OS fingerprinting (e.g., QueOS)
Implementation of the Algorithm
Traffic Traces



19
Unfiltered traffic is collected at the external
interface of 3 class B networks using
tcpdump.
Only the first 68 bytes of each packet were
collected to obtain the full TCP header.
At peak times, an hourly file contained on the
order of 2 million packets.
Implementation of the Algorithm
Implementation


The TCP FSM algorithm was written in MATLAB.
In practice, each packet was read sequentially from
the raw traffic file.
–
–

20
Each socket pair was assigned a stream holding the
complete trace for that socket.
How to deal with the truncation of the traffic data? (i.e.,
streams that start or end in the middle of a connection)
August 21, 2000 data for 24 hours starting at
midnight.
Results and Discussion
Analysis Results (1/3)
21
They do not proceed the
normal synchronization
process.
Results and Discussion
Analysis Results (2/3)
22
They do not complete
the synchronization
process.
Results and Discussion
Analysis Results (3/3)
23
Results and Discussion
1) Approximately 37% of the Internet TCP traffic does not obey the protocol.
2) If we remove “SYN scan” and “bad flag scan”, ↓
24
Results and Discussion
Network Management

Unresponsive and non-existant hosts
–

Delays (i.e., packets do not arrive in an appropriate time)
–

H1 | FAPU
Backscatter effects
–
25
X | timeout and C1 | timeout
Malfuncioning TCP
–

L | {SA, APU, FAPU, RAPU}, H2 | RAPU and X | timeout
Abandoned connections
–

H1 | RAPU, H1 | Timeout, H2 | RAPU and H2 | timeout
L | SA or L | SA
Results and Discussion
Security (1/2)

TCP connect and half-open scans
–
–
–

Stealth scans and inverse mapping (firewall)
–
–
–
26
Open port: {S-SA-APU-RAPU} or {S-SA-APU-FAPU-APU-FAPU-APU}
Half open: H2 | RAPU
Closed port: H1 | RAPU, H1 | timeout
SYN-FIN (e.g., Firewalls will be configured to block incoming
packets with only the SYN flag set)
L | APU + RST or ICMP destination unreachable
L | SA, L | FAPU, bad flag, L | RAPU
Results and Discussion
Security (2/2)

Aggregating scanning events
–
L | {SA, APU, FAPU, RAPU}, H1 | {RAPU, timeout} and H2 | {FAPU,
RAPU} or bad flag
7
3

Role of the Firewall
–
–
27
H2 | timeout
H2 | RAPU
2
1
1m 10m 1h
1
5h 12h
1
1d 2d
5d >5d
Results and Discussion
False Positives

By definition, all anomalies found using strict
anomaly detection are anomalous and
therefore there is no false positive rate to
consider.
–
–
28
It is difficult to determine the intent of the user
who sends a SYN and receives a RST in
response (failure H1 | RAPU).
It has different meanings and possesses different
importance to different applications.
Conclusions and Future Work
Conclusions and Future Work (1/2)






29
Examining packet follows on a network that do not conform to
the protocol
Successfully detect TCP-based scanning activity, delays,
outages, misconfigurations and other unexpected TCP
behaviour
Correlation engines could be used to correlate the events found
through application of this model.
A successful full connect scan cannot be detected with this
method because the flag pattern obeys the protocol
specification.
Include an option for TCP traffic that is not officially allowed but
is nevertheless present on the Internet.
IDS
Conclusions and Future Work
Conclusions and Future Work (2/2)



30
Using the proportionalities of anomalies found by
implementing the TCP FSM, one can introduce
failures using a Markov chain approach.
As well as a mechanism for traffic generation,
Markov models may provide the capability of
assigning a probability that a connection is valid.
This may be the basis of a statistical anomaly
detection engine.
Comments


The usage of a protocol can be considered as a
distinctive signature to a program.
Our work also include service level, and an
application is more complicated than TCP.
–
–


31
More events (messages)
States are unclear.
Too much false positive for anomaly detection model.
Perfect attacks may exist.
Further Readings

TCP
–
–

Apps
–
–
–
32
Floyd, S. (2002). Inappropriate TCP Resets Considered
Harmful. (work in progress).
End-to-end Interest Maillist. Frequency of RST terminated
connections. USC Information Sciences Institute.
Sasha/Beetle (2000). A Strict Anomaly Detection Model
for IDS. Phrack, 10(56).
Das, K. (2002). Protocol Anomaly Detection for Networkbased Intrusion Detection. SANS Institute.
Lemonnier, E. (2001). Protocol Anomaly Detection in
Network-based IDSs.