Performance Analysis of the
ANGEL System for
Automated Control of Game
Traffic Prioritisation
Jason But, Thuy Nguyen, Lawrence
Stewart, Nigel Williams, Grenville
Armitage
[email protected]
Centre for Advanced Internet Architectures (CAIA)
Swinburne University of Technology
Outline
System Architecture
System Implementation
Classification Techniques
System Performance
Accuracy of Classification
Timeliness of Classification
Processing Capability
Demonstration Video
Conclusions
NetGames’07
http://www.caia.swin.edu.au
[email protected]
September 2007
2
ANGEL Architecture
Performance Issues
Processing Performance
Accuracy and Stability
Scalability
Timeliness
NetGames’07
http://www.caia.swin.edu.au
[email protected]
September 2007
3
[email protected]
September 2007
4
Scalability
Built into the design
Difficult to test without deployment
NetGames’07
http://www.caia.swin.edu.au
Classification Techniques
Possible Approaches
Port based
Stateful reconstruction
Machine Learning algorithms
ANGEL deliberately separates flow classification from
prioritisation
Extensible to support different traffic types
NetGames’07
http://www.caia.swin.edu.au
[email protected]
September 2007
5
Classification – ANGEL
Machine Learning based
Naive Bayes Algorithm
Classification based on probabalistic knowledge
Real-time classification
Must classify using a small portion of flow
Should continuously classify
We use a sliding window of 25 packets per unique flow
Classification Model
Constructed as stated here1
Game Traffic – Wolfenstein Enemy Territory (ET) traffic of a
month-long trace collected at a public server in Australia
Non Game Traffic – From a 24-hour trace collected by the
University of Twente, Germany, at an aggregated 1Gbps link
1
T. Nguyen and G. Armitage. "Training on multiple sub-flows to optimise the use of machine learning classifiers in
real-world IP networks". Proceedings of the IEEE 31st Conference on Local Computer Networks, Florida, USA, 2006
NetGames’07
http://www.caia.swin.edu.au
[email protected]
September 2007
6
ANGEL Testbed
Home
LAN 1
CPE 1
ISP POP 1
Router
Internet
ISP POP 1
ANGEL FM
ISP
Internet
Router
ISP Core
Switch
ISP POP 2
ANGEL FM
Home
LAN 2
CPE 2
NetGames’07
ISP
Database
Server
ISP
ANGEL FC
ISP POP 2
Router
http://www.caia.swin.edu.au
[email protected]
September 2007
7
September 2007
8
Performance – Accuracy
100
99
98
Recall (%)
97
96
95
94
93
92
Recall
91
Precision
2000
2001
2002
2003
2004
2005
2006
2007
2008
0
1
2
3
4
5
6
7
8
9
90
M (Packets)
Where M is the number of packets missed from the
beginning of a flow
NetGames’07
http://www.caia.swin.edu.au
[email protected]
Performance – Stability
Classification accuracy is relatively high
Repeated classification on a 25 packet sliding window leads
to fluctuating classifications for the duration of the flow
This leads to:
Extra processing load as the client needs to re-deploy
prioritisation rules
Extra network load as classification changes are
communicated to ANGEL devices
Poor performance as game traffic may lose prioritisation for
short periods of time
To improve classification statibility we developed the
"Confirmed Classification" algorithm
Essentially deploys a low-pass filter to the output of the
classifier
Classification changed when two consecutive,
non-overlapping windows of packets generate the new
classification
NetGames’07
http://www.caia.swin.edu.au
[email protected]
September 2007
9
Performance – Stability
Non-game (Kazaa) flows
0.4
No State−Confirmation
State−Confirmation
0.0
CDF (0−1)
0.6
0.4
0.2
No State−Confirmation
State−Confirmation
0.0
CDF (0−1)
0.8
0.8
1.0
Stability for ET flows
0
0
1
2
3
4
5
20
40
60
6
Number of Classification State Changes per Flow
Number of Classification State Changes per Flow
Flows exhibiting
classification changes
dropped from 15 to 1
This flow only changed
state once
NetGames’07
http://www.caia.swin.edu.au
Other traffic types tested
- similar results
Significant improvement
in both the number of
flows and the number of
classification changes
[email protected]
September 2007
10
Performance – Timeliness
ANGEL – Initial classification for a new flow is non-game
With the "Confirmed Classification" algorithm we need to
capture two windows (50 packets) of a flow before it can be
classified as game traffic
Classification timeliness is dependent on (bi-directional)
packet rate generated by the game
Observations for ET show classification typically occurs
between 0.5 and 1 second(s) after flow begins
NetGames’07
http://www.caia.swin.edu.au
[email protected]
September 2007
11
Processing Performance – Flow Meter
40
30
20
50000
45000
40000
Packet Rate (PPS)
35000
30000
25000
20000
15000
10000
5000
1000
0
10
Packet loss (%)
50
Captures packets and forwards statistics to Classifier
Need to capture and process traffic with negligible loss
Compares with performance of underlying capture library
Supported by memory (5MB) and CPU (30%) usage rates for
all input packet rates
NetGames’07
http://www.caia.swin.edu.au
[email protected]
September 2007
12
Processing Performance – Flow Classifier
Tested under a worst case scenario - Single process
classifying all flows
Generated trace file consisting of multiple flows by
duplicating and combining a source trace file
Replayed trace file to a Flow Meter and then onto Classifier
Packet rate limited to 25,000pps - Flow Meter limit
Equivalent of 500 concurrent flows
Classifier able to correctly classify all flows
Memory footprint (< 5MB)
CPU usage (< 0.2%)
Suggests the bottleneck is the Flow Meter rather than the
classifier
NetGames’07
http://www.caia.swin.edu.au
[email protected]
September 2007
13
http://www.caia.swin.edu.au
[email protected]
September 2007
14
Demonstration
NetGames’07
Conclusions
We have built a working ANGEL System
Separate modules
Scalable - multiple Metering points
Game traffic classified with >96% accuracy
"Confirmed Classification" technique improves classification
stability
System bottleneck is the Flow Meter - limited by
performance of underlying packet capture facility
Machine Learning approach can scale to large numbers of
flows
User-perceived performance
Game flows typically classified and prioritisation rules
established within 1 second
Successful classification when traffic captured after flow has
started
Modular system - can grow to support other traffic flow types
NetGames’07
http://www.caia.swin.edu.au
[email protected]
September 2007
15