Evolution 0.9:
The Evolution of the U.S.
Peering Ecosystem
William B. Norton
Co-Founder & Chief Technical Liaison
Equinix, Inc.
Gigabit Peering Forum VII
Herndon, VA
September 9, 2003
Internet Researcher
• Four Years of Research
• Document Internet Operations Practices –
Interconnect/Peering/IXes/etc.
• Research Process – Focus with Community and
– Write White Paper version M.m
– Walk throughs & Feedback
– Fix/Update Paper
• 200 Walkthroughs later – a document reflecting
community Internet Operations knowledge
So far 6 White Papers
Internet Operations White Papers
“Interconnection Strategies for ISPs”
“Internet Service Providers and Peering”
“A Business Case for Peering”
“The Art of Peering: The Peering
Playbook”
5) “The Peering Simulation Game”
6) “Do ATM-based Internet Exchanges
Make Sense Anymore?”
1)
2)
3)
4)
Freely available. Send e-mail to [email protected]
New white Paper: The Evolution of the U.S. Peering Ecosystem, Agenda…
Agenda
The Evolution of the Internet Peering Ecosystem
– The Internet Peering Ecosystem Environment
– Each Player, their Motivations, their Behavior
• Four Major Events, led to …
• Three Major Evolutions in the Peering
Ecosystem
The Evolution of the U.S.
Peering Ecosystem
Act I: 1998-2000
The Internet Peering Ecosystem
• The Environment
• The Players
– Their role in the Peering Ecosystem
– Their behavior in the Peering Ecosystem
– Their relationship to others in the Peering
Ecosystem
• The Evolution of the Peering Ecosystem
The Internet Peering Ecosystem
• Many “Internet Regions” (usually defined
by geopolitical boundaries)
• Each with its own Peering Ecosystem
consisting of
– Tier 1 ISPs
– Tier 2 ISPs
– Content Players
and Enterprises
Tier 1 ISPs
Tier 2 ISPs
Content / Enterprise Companies
Internet Region
Interconnect Regions
• Large Internet Regions have multiple
geographical areas where ISPs peer
Internet Region
Interconnect Region
Interconnect Region
Interconnect Region
Interconnect Region
Interconnect Region
Player: Tier 1 ISPs
Def: A Tier 1 ISP is an ISP that has access to the entire Regional
Internet routing table solely through Peering Relationships
(I.e. doesn’t buy transit from anyone).
Peering
Interconnections
Tier 1 ISP
Backbone Backhaul
Transit (Downstream)
Connections
Transit Customers
Tier 1 Peering
• Full Mesh Peering among Tier 1 ISPs
within each Interconnect Region
Tier 1 ISP
Tier 1 Peering
Tier 1 ISP
Tier 1 ISP
Tier 1 ISP
Tier 1 ISP
Tier 1 ISP
Some Tier 1 ISPs in the U.S.
AT&T
UUNet
Sprint
C&W
Qwest
Level3
Tier 1 Peering Across the U.S.
AT&T
P
P
Sprint
P
P
P
P
P
P
UUNet
P
P
P
C&W
P
P
Qwest
Level3
AT&T
P
P
Sprint
P
P
P
AT&T
Sprint
P
P
P
P
P
P
P
P
P
P
Sprint
P
P
P
P
P
P
P
P
P
P
P
P
P
P
AT&T
UUNet
P
P
P
P
C&W
P
Level3
P
AT&T
P
Qwest
P
P
Sprint
P
P
P
Sprint
P
P
P
P
P
P
P
C&W
P
P
Level3
P
P
UUNet
P
P
P
C&W
P
P
Qwest
Level3
P
P
Qwest
8 Interconnect Regions
UUNet
P
UUNet
P
P
P
C&W
P
Level3
AT&T
P
P
P
P
Qwest
Level3
P
P
P
P
Sprint
P
P
Sprint
P
C&W
P
Qwest
C&W
P
Level3
P
P
P
AT&T
Level3
UUNet
P
P
P
C&W
P
UUNet
P
Qwest
UUNet
P
Qwest
AT&T
P
P
P
P
P
Tier 1 Peering Motivations
•
Generally not interested to peer with
anyone else
1) They already get the traffic for free
2) They may be foregoing revenue (they
could sell transit)
3) They don’t want to give away that which
differentiates them
Player: Tier 2 ISP
Def: A Tier 2 ISP is an ISP that purchases
(and therefore resells) transit within an
Internet Region.
Upstream
Transit
Connection(s)
Peering
Interconnections
(optional)
Tier 2 ISP
Backbone Backhaul
(optional)
Transit (Downstream)
Connections
Transit Customers
Tier 2 ISP Peering Motivations
1) Reduce transit costs
2) Improve Performance
3) Greater control over routing
Tier 2 Peering
• Sparse Mesh Peering in each Interconnect
Region
Tier 2 ISP
Tier 2 Peering
Tier 2 ISP
Tier 2 ISP
Tier 2 ISP
Tier 2 ISP
Tier 2 ISP
Tier 2 ISPs
• Widely varying characteristics and
behaviors
• Scope: Global, National, Regional, Local
• Underfunded to well funded
• Food fights to solid relationships within the
community
• Big Peoples Table and Little Peoples Table
perhaps a bad generalization
The Content Provider and
Enterprise Player
•
•
•
•
•
Always buy transitTransit Provider
Interconnections
Focus on Content (Upstream(s))
Focus on Service
Few Network Engineers
SLAs valuable
Content Provider
(and Enterprise)
The Internet Peering Ecosystem
Internet Region
Tier 1 ISPs
Tier 2 ISPs
Content / Enterprise Companies
Peering Inclinations and Peering
Policies
• Def: A Peering Inclination is a
predilection towards or against peering as
demonstrated by Peering behavior in a
Peering Ecosystem
• Def: A Peering Policy documents and
defines the prerequisites to peering.
Four types of Peering Inclination
•
•
•
•
Open Peering – Anyone, Anywhere
Selective Peering – Some Requirements
Restrictive Peering – No
No Peering – No, prefer buying transit
The Evolution of the U.S.
Peering Ecosystem
Act II: 2000-2003
Evolution of the U.S. Peering
Ecosystem
Four key events leading to drastic disruption in the
Peering Ecosystem:
1) The 1999/2000 Telecom Collapse
2) The growth of the Used Equipment Market
3) The Upstream Provider for the Cable Companies
(@Home) went bankrupt
4) Peer-to-Peer file sharing systems (like Kazaa)
grow in popularity, traffic grows exponentially
between Access Providers (1.5MB MP3 
700MB AVI files)
Leads to 3 Major Evolutions
Evolution #1 – Cable Companies
are Peering
•
•
40% of traffic Kazaa
3-5 Gbps of transitTransit Provider
traffic-> Gbps of Interconnections
(Upstream(s))
peering potential !
Significant because
1) Volume
Cable Company
2) Open Peering
3) Kazaa Effect
(Optional)
Backbone Backhaul
Peering
U.S. Cable Companies Eyeballs
Cable Modem Subscribers
MSO
Comcast
Time Warner
Cox
Charter
CableVision
Shaw (Big Pipe)
Rogers
Adelphia
Bright House
Mediacom
RCN
Insight
Cable One
Total
Country YE02
Change in 02
USA
3,620,300
1,199,100
USA
2,613,000
1,027,000
USA
1,407,900
524,400
USA
1,180,000
572,300
USA
770,100
263,500
Canada
750,000 ?
Canada
650,000 ?
USA
610,000
305,900
USA
490,000
159,000
USA
191,000
76,000
USA
160,400
49,800
USA
144,800
56,700
USA
78,100
45,200
12,665,600
4,278,900
Source: The Bridge & Leichtman Research
Source: I$P HO$TING ReportMay 2003, Volume VII, Num
Article "Dialup Bakeoff - Is it really so bad at EarthLink? O
Source: Bill Norton conversations with Rogers and Shaw 2
AOL
Cable Company
Cable Company
Cable Company
Cable Company
Cable Company
Cable Company
1998-2000
Tier 1 ISPs
$
Transit
Tier 2 ISPs
Thickness represents
In-Group Peering
Content
2000-2003 Evolution #1 – Cable
Companies Peering
Tier 1 ISPs
CableCos
Tier 2 ISPs
Content
Peering
Evolution #2 – Network Savvy
Large Scale Content Companies
get into Peering
• To reduce transit costs
• To improve the end user experience
• They need to move out of bankrupt colo anyway
Significant because
1) Volume of traffic is huge
2) Content Providers have Open Peering
3) Leading players pave the way
1998-2000
Tier 1 ISPs
$
Transit
Tier 2 ISPs
Thickness represents
In-Group Peering
Content
2000-2003 Evolution #2 – Large
Scale Network Savvy Content
Peers
Tier 1 ISPs
Tier 2 ISPs
LSNSC
Content
Other Broadband Players
• Significant, but…
• Not Open Peers so disincentive to peer
• So less volume and less disruptive
Evolution #3 – Cable Companies
Freely Peer with Content
Companies
• Content literally on the Cable Company
Network
• Lowest possible latency from content to
eyeballs
• Internet Gaming, Broadband Streaming, etc.
1998-2000
Tier 1 ISPs
$
Transit
Tier 2 ISPs
Thickness represents
In-Group Peering
Content
2000-2003 Evolution #1 – Cable
Companies Peering
Tier 1 ISPs
CableCos
Tier 2 ISPs
Content
2000-2003 Evolution #2 – Large
Scale Network Savvy Content
Peers
Tier 1 ISPs
Tier 2 ISPs
LSNSC
Content
2000-2003 Evolution #3 – Cable
Peers with Large Scale Network
Savvy Content
Tier 1 ISPs
CableCos
Tier 2 ISPs
LSNSC
Content
International Dynamics –
Separate White Paper
IIJ
NTT
DSL
Tier 1 ISPs
KDDI CableCos JT
Tier 2 ISPs
Content
JP
U.S.
U.S. Tier 1 ISPs are Tier 2 ISPs in Japan Internet Region
Japan Tier 1 ISPs are Tier 2 ISPs in the U.S. Internet Region
2003 Peering Ecosystem Evolving
Summary
NoPeering Open
Selective
Restrictive
Lg.Content
Tier2 ISPs
CableCos
RBOCs
Tier 1 ISPs
Foreign
ISPs
Excite@Home