Observing Slow Crustal Movement
in Residential User Traffic
Kenjiro Cho (IIJ), Kensuke Fukuda (NII),
Hiroshi Esaki (Univ. of Tokyo), Akira Kato (Keio Univ.),
Jun Murai (Keio Univ.)
August 16 2008
motivation
many media coverage on explosive traffic growth by video content
I
YouTube is just the beginning[Cisco2008b]
but technical sources report only modest traffic growth worldwide
I
MINTS: 50-60% in U.S. and worldwide
I
Cisco visual networking index: worldwide growth of 50% per
year over last few years
why is traffic growth important?
I
one of the key factors driving research, development and
investiment in technologies and infrastructures
I
with annual growth of 100%, it grows 1000-fold in 10 years
I
with annual growth of 50%, it grows 58-fold in 10 years
key question: what is the macro level impact of video and other
rich media content on traffic growth at the moment?
2 / 22
residential broadband subscribers in Japan
28.7 million broadband subscribers as of March 2008
I
DSL:12.7 million, FTTH:12.2 million, CATV:3.3 million
shift from DSL to FTTH: about to exceed DSL
100Mbps bi-directional fiber access costs 40USD/month
I
significant impact to backbones
Number of subscribers [million]
I
15
DSL
CATV
FTTH
10
5
0
2000 2001 2002 2003 2004 2005 2006 2007 2008
Year
3 / 22
traffic growth in backbone
rapidly growing residential broadband access
I
low-cost high-speed services, especially in Korea and Japan
I
Japan is the highest in Fiber-To-The-Home (FTTH)
traffic growth of the peak rate at major Japanese IXes
modest growth of about 40% per year since 2005
4
300
3
Annual growth rate
Aggregated IX traffic [Gbps]
I
200
2
100
Traffic volume
Growth rate
0
2000 2001 2002 2003 2004 2005 2006 2007 2008
Year
1
0
4 / 22
data collection across major ISPs
focus on traffic crossing ISP boundaries (customer and external)
I
tools were developed to aggregate MRTG/RRDtool traffic logs
only aggregated results published not to disclose individual ISP
share
challenges: mostly political or social, not technical
(B1)
external 6IXes
JPNAP/JPIX/NSPIXP
(B2)
external domestic
local IXes
private peering/transit
(B3)
external international
external provider edge
ISP
customer edge
(A1)
RBB customers
DSL/CATV/FTTH
(A2)
non-RBB customers
leased lines
data centers
dialup
5 traffic groups at ISP cusomer and external boundaries
5 / 22
methodology for aggregated traffic analysis
month-long traffic logs for the 5 traffic groups with 2-hour
resolution
I
each ISP creates log lists and makes aggreagated logs by
themselves without disclosing details
biggest workload for ISP
I creating lists by classifying large number of per-interface logs
I
I
some ISPs have more than 100,000 logs!
maintaining the lists
I
frequent planned and unplanned configuration changes
data sets
I 2-hour resolution interface counter logs
I
I
from Sep/Oct/Nov 2004, May/Nov 2005-2008
by re-aggregating logs provided by 7 ISPs
IN/OUT from ISPs’ view
6 / 22
traffic growth
22-68% increase in 2007
I
RBB: 22% increase for inbound, 29% increase for outbound
I
a sharp increase in international inbound due to popular video
services
400
150
A1(in)
A1(out)
A2(in)
A2(out)
Traffic (Gbps)
Traffic (Gbps)
300
200
100
0
2004/09 2005/05
2006/05
2007/05
2008/05
100
B1(in)
B1(out)
B2(in)
B2(out)
B3(in)
B3(out)
50
0
2004/09 2005/05
2006/05
2007/05
2008/05
7 / 22
changes in RBB weekly traffic
in 2004, inbound and outbound was almost equal
in 2008, outbound (downloading to users) became larger
both constatnt portion and daily fluctuations grew in 2008
I
implies a shift from p2p to video (e.g, YouTube)
8 / 22
analysis of per-customer traffic in one ISP
one ISP provided per-customer traffic data
I Sampled NetFlow data
I
I
from edge routers accommodating fiber/DSL RBB customers
week-long data from Apr 2004, Feb 2005, Jul 2007, Jun 2008
I
Feb 2005 and Jun 2008, before and after the advent of
YouTube
9 / 22
ratio of fiber/DSL active users and total traffic volumes
I
in 2008, 80% of active users are fiber users, consuming 90%
of traffic
2005
2008
fiber
DSL
fiber
DSL
active users (%)
46
54
79
21
total volume (%)
79
21
87
13
10 / 22
PDF of daily traffic per user
2 lognormal distributions: asymmetric, symmetric high-volume
I high-volume dist: not growing much
I total(left) fiber(middle) DSL(right) in 05(top),08(bottom)
I mode: 3.5MB,32MB/day(2005), 5MB,94MB/day(2008)
0.4
0.3
0.2
0.1
10
6
7
8
9
10
10
10
10
Daily traffic per user (bytes)
10
10
(d) Total (2008)
10
0.3
0.2
0 4
10
10
6
7
8
9
10
10
10
10
Daily traffic per user (bytes)
10
10
(e) Fiber (2008)
0.4
0.3
0.2
0.1
11
10
6
7
8
9
10
10
10
10
Daily traffic per user (bytes)
10
10
11
10
0 4
10
5
10
6
7
8
9
10
10
10
10
Daily traffic per user (bytes)
10
10
0.3
0.2
11
10
In
Out
0.5
0.4
0 4
10
0.3
0.2
(f) DSL (2008)
In
Out
0.1
5
10
0.4
0.1
5
0.5
Probability density
Probability density
11
In
Out
0.5
0 4
10
0.4
0.1
5
In
Out
0.5
Probability density
0 4
10
(c) DSL (2005)
In
Out
0.5
Probability density
Probability density
(b) Fiber (2005)
In
Out
Probability density
(a) Total (2005)
0.5
0.4
0.3
0.2
0.1
5
10
6
7
8
9
10
10
10
10
Daily traffic per user (bytes)
10
10
11
10
0 4
10
5
10
6
7
8
9
10
10
10
10
Daily traffic per user (bytes)
10
10
11
10
11 / 22
CCDF of daily traffic per user
only outbound (download for users) increased
0
0
10
10
In
Out
-2
10
-3
-2
-3
10
-4
10
-4
10
-5
-5
10
10
10
10
10
10
(a) 2005
-6
In
Out
-1
Cumulative distribution
Cumulative distribution
-1
10
4
10
5
10
6
7
8
9
10
10
10
10
10
Daily traffic per user (bytes)
10
11
10
10
12
10
(b) 2008
-6
4
10
5
10
6
7
8
9
10
10
10
10
10
10
Daily traffic per user (bytes)
11
10
12
10
12 / 22
CDF of traffic volume consumed by top heavy-hitters
graph: the top N% of heavy-hitters use X% of the total traffic
highly skewed distribution in traffic usage
no noticeable change from 2005 to 2008
probably because client-type users also have long-tailed
distributions
1
0.8
Cumulative traffic
I
In (2005)
Out (2005)
In (2008)
Out (2008)
0.6
0.4
0.2
0 -4
10
-3
10
-2
10
10
Cumulative heavy hitters
-1
0
10
13 / 22
correlation of inbound/outbound volumes per user
fiber (left) and DSL (right) in 2005 (top) and 2008 (bottom)
2 clusters: one below the unity line, another in high volume region
no clear boundary: heavy-hitters/others, client-type/peer-type
11
10
11
10
(a) Fiber (2005)
10
10
Daily inbound traffic (byte)
Daily inbound traffic (byte)
10
10
9
8
10
7
10
6
10
5
10
4
10 4
10
5
10
6
7
8
9
10
10
10
10
Daily outbound traffic (byte)
10
10
11
10
9
8
7
10
6
10
5
10
4
7
10
6
10
5
4
10 4
10
5
10
6
7
8
9
10
7
8
9
10
10
10
10
10
Daily outbound traffic (byte)
10
10
11
10
10
11
10
10 4
10
8
(d) DSL (2008)
10
10
10
9
10
10
(c) Fiber (2008)
Daily inbound traffic (byte)
Daily inbound traffic (byte)
10
10
10
11
10
(b) DSL (2005)
10
10
10
9
8
10
7
10
6
10
5
10
5
10
6
7
8
9
10
10
10
10
Daily outbound traffic (byte)
10
10
11
10
4
10 4
10
5
10
6
10
10
10
10
Daily outbound traffic (byte)
11
14 / 22
protocols/ports ranking
classify client-type/peer-type with threshold: 100MB/day upload
protocol
port
TCP
*
(< 1024)
80 (http)
554 (rtsp)
443 (https)
20 (ftp-data)
(>= 1024)
6346 (gnutella)
6699 (winmx)
7743 (winny)
1935 (rtmp)
6881 (bittorrent)
UDP
*
53 (dns)
others
ESP
GRE
ICMP
total
(%)
97.43
13.99
9.32
0.38
0.30
0.93
83.44
0.92
1.40
0.48
0.20
0.25
1.38
0.03
1.35
1.09
0.07
0.01
2005
client
type
94.93
58.93
50.78
2.44
1.45
1.25
36.00
0.84
1.14
0.15
0.81
0.06
3.41
0.14
3.27
1.35
0.12
0.05
peer
type
97.66
8.66
5.54
0.19
0.19
0.90
89.00
0.93
1.43
0.51
0.14
0.27
1.19
0.02
1.17
1.06
0.06
0.01
total
(%)
96.00
17.98
14.06
1.36
0.58
0.24
78.02
0.94
0.68
0.30
0.22
0.22
1.94
0.04
1.90
1.93
0.09
0.02
2008
client
type
95.51
76.16
64.96
8.21
1.63
0.17
19.35
0.67
0.24
0.04
0.73
0.02
2.50
0.12
2.38
1.85
0.08
0.05
peer
type
96.06
11.35
8.26
0.58
0.46
0.25
84.71
0.97
0.73
0.33
0.16
0.24
1.88
0.03
1.85
1.94
0.09
0.02
15 / 22
temporal behavior of TCP port usage
3 types: port 80, well-kown port but 80, dynamic ports
total users (top), client-type (middle), peer-type (bottom) in 2005
(left) and 2008 (right)
16 / 22
summary of per-customer traffic analysis
overall traffic is still dominated by heavy-hitters, mainly using p2p
I
but its traffic decreased in population share and volume share
current slow growth is due to stalled growth of dominant
aggressive p2p traffic
client-type traffic slowly moving towards high-volume
I
circumstantial evidence: driven by video and other rich media
17 / 22
growth model based on lognormal distributions
fitting client-type outbound volumes to lognormal dist.
p(x) =
1
√
xσ 2π
exp(
−(ln x − µ)2
)
2σ 2
E (x) = exp(µ + σ 2 /2)
I
I
by definition, mean grows much faster than mode
simplistic growth projections for outbound traffic per user
(MB/day) for client-type users
2004 Apr
2005 Feb
2007 Jul
2008 Jun
growth/yr
2009 Jun
2010 Jun
2011 Jun
mode
26.2
32.0
65.7
94.1
1.36
121
164
223
mean
110.6
162.7
483.2
862.6
1.62
1217
1966
3176
18 / 22
outbound traffic growth of client-type users
Traffic Volume per user [MB/day]
4000
Mean
Mode
3000
2000
1000
0
2004/04 2005/04 2006/04 2007/04 2008/04 2009/04 2010/04 2011/04
Year
19 / 22
conclusion
apparent slow growth attributed to decline of p2p traffic
I
but p2p willl not go away anytime soon
I
p2p could evolve for large scale distribution
crustal is slowly swelling with video content
I
similar to how web traffic was perceived in late 90es
network capacity also grows 50% per year (by various sources)
difficult to predict future traffic (lognormal!)
many challenges ahead
I
technical factors: content caching, CDN, QoS
I
economic factors: access cost, capacity/equipment costs
I
political/social factors: net-neutrality, content management
20 / 22
acknowledgments
I
IIJ, SoftBank Telecom, K-Opticom, KDDI, NTT
Communications, SoftBank BB for data collection support
I
ministry of internal affairs and communications for
coordination
21 / 22
references
[CFEK2006] K. Cho, K. Fukuda, H. Esaki, and A. Kato.
The impact and implications of the growth in residential user-to-user traffic.
In SIGCOMM2006, Pisa, Italy, Aug. 2006.
[Cisco2008b] Cisco.
Approaching the zettabyte era.
http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/
ns705/ns827/
white paper c11-481374 ns827 Networking Solutions White Paper.html,
June 2008.
[Cisco2008a] Cisco.
visual networking index – forecast and methodology, 2007-2012.
http://www.cisco.com/en/US/netsol/ns827/
networking solutions sub solution.html, June 2008.
[Odlyzko2008] A. M. Odlyzko.
Minnesota Internet traffic studies.
http://www.dtc.umn.edu/mints/home.html.
22 / 22