BGP: Introduction and Issues
Michalis Faloutsos
(with the help of various
contributions of slides)
Advanced Networks
1
What Is BGP?
Border Gateway Protocol BGP-4
The de-facto interdomain routing protocol
BGP includes specifications:
• Which information gets advertised and how
BGP includes a routing protocol:
• Establishes and uses a routing table
Internal Gateway Protocol (I-BGP in the
book)
Advanced Networks
2
Why Is There Such Fuss about BGP?
BGP dictates routing at the AS level
• Absence of understanding: poor performance
BGP is complicated
• Designed to be flexible
• Involves multiple fields
Understanding BGP behavior is not intuitive
• Implementation and business policies
The routing of the Internet relies on BGP
Advanced Networks
3
Some Open Questions
How well does BGP work now?
How can I manage a BGP network?
How secure and robust is BGP?
• Cyber-terrorism
How would we re-design BGP now?
How well will BGP scale for our future
needs?
Advanced Networks
4
Roadmap
Introduction to BGP
Highlights of BGP issues
Goal: instigate interest in BGP
Advanced Networks
5
Some Basic Numbers
• 43,000 Autonomous Systems approx.
• Corporate Networks
• ISP Internal Networks
• National Service Providers
• Identified by ASN a 16 bit value
• Assigned by IANA
• Superlinear growth (Huston, Siganos et al.)
Advanced Networks
6
How A BGP graph Looks Like
AS 2
AS 5
AS 4
AS 3
Each AS has
designated BGP
routers
BGP routers of an
AS communicate
internally with
another protocol
(IGP)
AS 1
Advanced Networks
7
IP Addresses and Prefixes
IP addresses have 32 bits: 4 octets of bits
(IPv4)
A prefix is a group of IP addresses
128.32.101.5 is an IP address (32 bits)
128.32.0.0/16 is a prefix of the 16 first bits:
• 128.32.0.0 – 128.32.255.255
(2^16 addresses)
128.32.4.0/24 is a prefix of the 24 first bits longer
Advanced Networks
8
Routing is Based on Prefixes
A BGP Routing table has prefixes for entries
For a IP address of a packet, find longest match
Example: packet IP 128.32.101.1
128.1.1.4 matches the first 8 bits – no match!
128.32.0.0/16 match for 16 bits
128.32.101.0/24 is a longer match
Advanced Networks
9
Prefix Matching in More Detail
For a IP address of a packet, find longest match
Example: Compare
•
•
•
•
•
•
•
packet IP 128.32.101.1
With 128.32.0.0/16
IP
: 01000000. 001000000. 01100101 .00000001
Mask : 11111111. 111111111. 00000000 .00000000
AND : 01000000. 001000000. 00000000 .00000000
Prefix : 01000000. 001000000. 00000000. 00000000
Equal? Yes
Advanced Networks
10
Advertising Routing Information
Each AS advertises what it can reach from
each BGP router
Policies I: filter what you advertise
Policies II: filter from what you hear
advertised
Build up a BGP routing table
• Remember which prefix you hear from which link
Advanced Networks
11
What Does a Routing Table Look Like?
Prefix
Origin AS Path
128.32.0.0/16
123
14 56 123
123
34 101 203 123
15
50 15 15
128.32.101.0/24
Origin AS “owns” the address
Routing tables can have peculiarities
Advanced Networks
12
Route Advertising
Distance Vector style protocol
Hear advertisements: IP prefix, AS-path
Filter if desired (i.e. ignore)
Append yourself: IP prefix, myAS+AS-path
Forward to appropriate ASs
Advanced Networks
13
Basic AS relationships
Customer – Provider
• Customer pays Provider for service
• The Customer is always right
Peer to Peer: mutual cooperation
• Ex. MCI and AT&T
Sibling-Sibling
• Ex. AT&T research and AT&T wireless
Advanced Networks
14
The Internet as a Directed Graph
Every edge is
bidirectional reg. traffic!
Business relationships
are represented
Provider
Peer
Advanced Networks
Customer
Peer
15
The Initial Idea
Data flows between customers-providers
Top level providers are peers
• They exchange information to ensure connectivity
What can possibly go wrong?
Advanced Networks
16
And then came the rain…
Thousands of ASs
Complicated relationships
Multiple providers for one AS, and IP prefix!!
• Multihoming
Traffic engineering
• I want to use multiple paths and load balance
Advanced Networks
17
AS Relationships
200
100
10
11
1
2
12
3
Provider
Peer
Customer
Peer
13
4
Customer – Provider: customer pays and is always right
Peer to Peer: Exchange traffic only between their customers
Sibling-Sibling: Exchange traffic at will
Advanced Networks
18
The Rules of BGP Routing
Transit traffic: traffic that does not go to my
customers (or their customers)
A provider carries any traffic to, from
customer
Peers exchange traffic only if between their
customers
Advanced Networks
19
How BGP Policy Restricts Routing
Provider
100
200
Peer
Customer
Peer
10
11
1
2
12
13
3
4
Path Properties:
Routing rules:
• Provider accept everything
• Peer only if it is for its
customers
• Up then down
• No up-down-up, at most 1
peer-peer steps
Advanced Networks
20
What paths are allowed?
Provider
100
200
Peer
Customer
Peer
10
11
1
2
12
13
3
4
Advanced Networks
21
Implementing BGP Rules
What do you do with an advertisement:
Through customer link
• Advertise to all (customers, peers, providers)
Through sibling link
• Advertise to all
Through provider link
• Advertise to customer only (and possibly siblings)
Through peer link
• Advertise to customer only (and possibly siblings)
Advanced Networks
22
How Policies Affect Routing
Customer 1
ISP1
A Provider will get rid of
traffic as soon as
possible,
But a Provider will carry
the traffic for its
customer
Did anyone say traffic is
asymmetric?
ISP2
Customer 2
Advanced Networks
23
BGP Path-Length Asymmetry
Consider number of AS traversed by a path
Asymmetry: 46% of pairs differ by at least one AS hop
[Siganos 01]
Advanced Networks
24
Determining The Logical Graph
The business relationships are critical
How can I find the relationships?
1. Infer relationships from routing tables
2. IRR database: manually maintained – error prone
Advanced Networks
25
Two Inference Algorithms
Inference algorithm [Gao 00]
• Using 1 routing table
• Exploit the up-down path property
in a routing path, assume highest degree node as peak
Inference using multiple tables [Subramanian02]
• Use multiple points of observation to improve results
Advanced Networks
26
Things Become Interesting:
Traffic Engineering
AS 2
LP 100
LP 80
slow
How can I pick a route?
Local Preference: path
attribute
AS2 wants to prefer fast
thick link
Advertisement from right
router of AS2 has higher
Local Preference
Any BGP router in AS2 will
prefer the thick link
208.1.1.0/24
AS 1
Advanced Networks
27
Load Balancing - Appetizer
I want to share traffic between my two
providers
How can I do this?
Advanced Networks
28
Load Balancing: Long Prefix Match
Takes All!
138.39/16
ISP 3
138.39.1/24
ISP 2
138.39.1/24
ISP 1
138.39/16
138.39.1/24
Customer 138.39.1/24
Advanced Networks
29
So How Can I Balance the Load?
Ask my provider to not aggregate my prefix
• Will this work?
Split my prefix in two
• 138.39.1.0/24
• A: 138.39.1.31/28
• B: 138.39.1.32/28
Advertise only one part to ISP2
ISP2 traffic destined for prefixes in A
ISP1 traffic destined for prefixes in B
Advanced Networks
30
Summary Up to Now
BGP-4 is the de facto protocol for
interdomain routing
BGP was developed to achieve:
• Flexible policy implementation
• Scalability via route aggregation given CIDR
There are many open issues
• BGP is a hot research topic
Advanced Networks
31
The Growth of BGP Table
Advanced Networks
32
The BGP Growth: The Truth
Growth flattened
out
Why?
• Better
management
• Dot-com crash?
Advanced Networks
33
Routing Table Variation
100k
Larger ASes have significantly larger tables
Advanced Networks
34
Update Activity Per Prefix
Measure rate of announcements +
withdrawals + path updates
Compare relative update rate per prefix
length to the relative number of prefixes
of that length
>1 implies higher than average update rate
(less stable)
 <1 implies lower than average update
rate(more stable)

Advanced Networks
35
Measured Update Rate
Bursty!
Advanced Networks
36
BGP robustness
Measuring the BGP updates
Advanced Networks
37
Analyzing Messages By Content
Aggregated per 30 seconds
Advanced Networks
38
Initial Observations
Updates show daily and weekly periodicity
There is no evidence of BGP disturbance:
• The Baltimore tunnel train 18 July that destroyed
Internet lines
• Sept 11 attack
There are some spikes at:
• 19 July
• 18-22 September
Advanced Networks
39
BGP Updates Correlations
Advanced Networks
40
BGP Under Attack
Advanced Networks
41
Router CPU Activity Correlates…
Advanced Networks
42
The Attack of The Worm
Advanced Networks
43
Classification of Instabilities
Advanced Networks
44
Isolating Instability: 1 unstable peer
Advanced Networks
45
Detecting abnormal BGP activity
Advanced Networks
46
The Worm Activity
Advanced Networks
47
The Worm Correlates Again…
Advanced Networks
48
Advanced Networks
49
Advanced Networks
50
Advanced Networks
51
Advanced Networks
53
Advanced Networks
54
Summary of BGP Instability
Globally correlated BGP instability is not
uncommon
Some causes are well understood
(misconfiguration, bad path announcements)
Some others are less well understood, and
more worrisome:
• worms
Advanced Networks
55
BGP: Up Close and Personal
Establishing a connection
Messages
Path Attributes
Advanced Networks
56
Establishing A BGP Session
BGP uses TCP connections
• For reliability
A BGP session is between two routers
• Typically directly connected (Ethernet, FDDI)
Routers establish a BGP session
• Authentication and set-up
• Update and withdrawals
• If disconnected, all paths are invalidated
Advanced Networks
57
Messages
First open TCP connection
• Identification and authentication
BGP messages
•
•
•
•
OPEN: set-up, negotiate timer for keep-alive
UPDATE: routing changes
NOTIFICATION: termination, and error messages
KEEPALIVE: confirm that connection is active
Advanced Networks
58
UPDATE Message
Advertise reachability information
Withdraw paths to prefixes
Update information prefixes
Introduce new prefixes
Modify important path attributes for new
prefixes and the related paths
Advanced Networks
59
Standard Path Attributes
Attribute: Type, Length, Value
Origin: where did I hear this from
• BGP (external) or IGP (internal)
AS Path: sequence of ASs
Flexible handling of loops
• Recovering from disconnected ASs!
Next Hop: set explicitly who the next router
should be (possibly a non BGP speaker)
Advanced Networks
60
Multi-Exit Discriminator (MED)
MED 10
AS 2
AS 1
MED 50
•Indication to external peers of the
preferred path into the AS
•Lowest Med Preferred
Advanced Networks
61
Note for MED
Defines preference for incoming traffic
One AS sets the values
Another AS interprets and uses them
Thus:
• We need cooperative ASes
• Only between two ASes (1 hop scope)
MED is meaningless in the next hop
MED can be used only if both routes are
advertised from the same AS
Advanced Networks
62
Local Preference
AS 2
AS 1
L1
Choosing paths
internally
• Defining outgoing
traffic
Within an AS:
• Set Local Preference
to pick the path you
want to send data to
L2
The higher Local
AS3
Preference is
Here MED can not be used since we preferred
have different AS
Advanced Networks
63
Aggregation of updates
AS want to aggregate as much as possible
• Reduce routing state
• Reduce information that needs to be exchanged
Main idea: send one update instead of two
• All other attributes are the same (path, preferences etc)
• The prefixes have to be subsets, or adjacent
subset
adjacent
Advanced Networks
64
Atomic Aggregate
Sometimes we aggregate paths that are
different (ie different AS sequence)
Atomic aggregate shows that some of the
destinations in this prefix are not necessarily
following the same path
This does not allow other routers to deaggregate the path, thus creating entries that
should not appear
Advanced Networks
65
Atomic Aggregate
138.39/16 :
14
AS 3
138.39/16 :
314
AS 1
138.39/16
138.39.1/24
AS 4
138.39/16
AS 2
138.39.1/24
AS 5
AS 2 thinks that all 138.39/16 follow the 3 1
4 path
66
Advanced Networks
BGP Route Selection Process
1.
2.
3.
4.
5.
6.
Maximum prefix length match
Highest Local Priority
Shortest AS Path
Lowest MED (if routes through same AS)
Min Cost Next hop router (consulting IGP)
Prefer external to internal routes
1. Pick lowest BGP identifier among many E-BGP
2. Pick lowest BGP identifier among many I-BGP
Advanced Networks
67
No Valley Prefer Customer Routing
An abstraction of common sense policy
• No valley = don’t transit traffic for peer or provider
• Ie. If you don’t get paid, don’t do it
ASes use the following rule, when selecting a
path
• Prefer a path through a customer
• Prefer a path through a peer
This policy can be implemented using BGP
attributes for paths
Advanced Networks
68
Advanced Networks
69
Internal BGP (I-BGP)
Communication between routers of an AS
I-BGP very similar to E-BGP except:
• Different advertising rules
Do not re-advertise a path internally
• I-BGP 1 -> I-BGP 2 -X- I-BGP3
But readvertise (in -out, out -in):
• I-BGP 1 -> I-BGP 2 -> E-BGP1
• E-BGP 1 -> I-BGP 2 -> I-BGP1
Advanced Networks
70
I-BGP Re-Advertising
I-BGP
E-BGP
A
B
D
C
B will not re-ad. to C
what it hears from A
But it will re-ad to D
Why?
Paths are identified
by AS, and internally
you have the same
AS
• To avoid routing loops
Advanced Networks
71
I-BGP Mesh
I-BGP routers form a fully connected mesh
(clique)
• Scalability becomes an issue
The full mesh is independent of physical
connectivity
Advanced Networks
72
A Subtle Difference I-BGP vs E-BGP
E-BGP: routers communicate using the IP of
the physical interface (link)
• Link based reliability
I-BGP: routers have “virtual or loopback”
interface
• Even when link fails, routers may be reachable
• Node based reliability
Advanced Networks
73
Static vs Dynamic Configuration
In practice, many customers do not speak
BGP
Do not have an AS number
They are configured statically
Advanced Networks
74
One vs Many Providers
Single homed vs multihomed
Customers for reliability and performance
connect to many providers
Difference:
• Single homed: easy to manage
• Multihomed: tricky
 Route aggregation
 Load balancing
Advanced Networks
75
Multihoming
How can I use my multiple connections
efficiently?
Multihoming is quite widespread
Users take it up to themselves to find
reliability and good performance [Huston]
Consequence: non-aggregatable state
Advanced Networks
76
Two routers two links
ISP 1
B
C
A
Multiplex traffic at link or IP
layer
Use Next Hop to point to
virtual router IP
• Second look up: how can I reach
virtual IP?
• Pick one of the two links randomly
or statically
• Furthermore, when one link is
down the other one is chosen
Advanced Networks
77
Multihoming with One Routers 1 ISP
ISP 1
B
C
P1
A
P2
Customer can advertise
different prefixes on each
link
• Reliability?
Use Multi-Exit-Discriminator
• Cust. Sets MED, ISP uses it
Use Local Preference
• ISP sets LP and picks link
• Customer’s IBGP to pick router
Advanced Networks
78
Issues
The mechanisms are there, but they depend
on many factors
• How much traffic each prefix generates?
Balancing incoming and outgoing traffic
Dynamically adapting to changing conditions
Technical issues:
• Some ISPs do not accept very long prefixes
 Longer than they would be in classfull routing
 Longer than 19 for new prefixes
Advanced Networks
79
Multihoming with Two Routers 1 ISP
Internet
Similar case for Provider to
Customer
For Customer to Provider:
ISP 1
C
P1
B
ISP 1
Customers
A
P2
• A could alternate paths
 Reordering of packets
• ISP 1 could advertise different
addresses on each link
 I.e. 1: ISP customers

2: Default (everybody
else)
Advanced Networks
80
Multihoming With Two Providers
138.39/16
ISP 3
138.39.1/24
ISP 2
138.39.1/24
ISP 1
138.39/16
138.39.1/24
Customer 138.39.1/24
Advanced Networks
81
Two Providers Multihoming: Getting
Address Space
Given two ISP
Get space from one (we saw before)
Get space from both
• Advertise only related prefix to ISP
 Aggregation but not reliability
• Advertise both prefixes to ISPs
 No aggregation but reliability
Get space independently of both
• Max flexibility, not reliability
• If too narrow of prefix, may not propagate, no connectivity
Advanced Networks
82
I-BGP Scalability
Full mesh is not scalable: O(N^2) IBGP sessions
Approaches to scalable I-BGP
• Hierarchical structure: Route reflectors
• Divide and conquer: Confederations
Advanced Networks
83
Route Reflection
D
E
B
C
A
Explicitly allow some IBGP routers to
readvertise
Route reflectors:
represent other routers
Hierarchical structure
avoids loops and
problems
Advanced Networks
84
Confederations
D
E
B
C
A
Decompose an AS to
sub-AS
Externally: one AS
Internally: like E-BGP
(E-I-BGP)
Loop avoidance:
• AS-CONFED-SET
• AS-CONFED-SEQUENCE
Advanced Networks
85
Confederation BGP Rules
Differences of BGP between sub-ASes
Local-Preference is transitive
• Recall in BGP it is not
NEXT-HOP attribute is forwarded
Path within AS is monitored with sub-AS no.
• AS-CONFED-SEQ
• AS-CONFED-SEQUENCE
Advanced Networks
86
Handling the Dynamic Nature
Advanced Networks
87
To Refresh or not to Refresh?
BGP uses hard state:
BGP routers consider a path is “usable”
• until explicitly withdrawn
• the session fails
How do I detect if a connection failed?
• Keep-alive messages
Advanced Networks
88
Path Updates Frequency
Send updates of a path no sooner 30 sec
Why?
• Stability
• Overhead reduction
Side-effects
• Convergence can take longer
What is the right interval?
• Recent studies say that 30s is too long
Advanced Networks
89
Route Flapping and Dampening
Flapping: constant alternating updates
It can happen!
Route update dampening:
• Consider stability of path before using it
How store “penalty” value for each path
Issue: router needs to remember withdrawn
paths
Advanced Networks
90
Overview up to Now
BGP is Distance Vector
BGP uses TCP and hard-state
Routing updates are “delayed” and batched
Route dampening to alleviate instabilities
Advanced Networks
91
End
Advanced Networks
92
Current Research: the AS Topology
Characterize the topology using power-laws
(Faloutsos 99, Siganos et al 01)
Modeling the evolution of the topology
(Barabasi, Siganos01)
Advanced Networks
93
Ongoing Research: AS paths
107 Gb of AS paths over 3 years
Exponential growth, but distances remain the
same
Inflation due to policy:
• 20% of paths are larger than they could
Significant Routing Asymmetry:
• 40% paths by at least one hop
Advanced Networks
94
Near Future Research Plans
Mine the collected paths for patterns
How stable were the paths?
Can we identify “illegal” paths?
Identify pathologies (ie. loops)
Advanced Networks
95
Overview of Plans for Research
How well does BGP work now?
How secure and robust is BGP?
• Cyber-terrorism: how much damage can be done?
How would we design BGP now?
• People are asking this question
How well will BGP scale?
How can we manage BGP (avoid human errors)?
Approach:
• Analytical and simulations with SSFNET
Advanced Networks
96
Conclusions
BGP is an open and exciting topic
The community knows very little
Big ticket items:
•
•
•
•
Measurements and modeling
Robustness, security
Network Management: traffic engineering
Scalability
Advanced Networks
97
Multihoming With Two Providers
138.39/16
ISP 3
138.39.1/24
ISP 2
138.39.1/24
ISP 1
138.39/16
138.39.1/24
Customer 138.39.1/24
Advanced Networks
98
Policies on Transitivity
Isp 1
Isp 2
Nontransit
A customer should not
Be transit for its Providers
Not allowed
AS X
Transit
AS 1
AS 3
AS 4
AS 2
Advanced Networks
99
More BGP attributes
•Communities
•Confederations
•Route Reflectors
Advanced Networks
100
Convergence
Advanced Networks
101
Multi-Exit Discriminator(MED)
•Indication to external peers of the
preferred path into the AS
•Lowest Med Preferred
Advanced Networks
102
Note in MED
One AS sets the values
Another AS interpets and uses them
Thus:
• Cooperative ASes
• Only between two ASes (1 hop scope)
MED is meaningless in the next hop
MED can be used only if both routes are
advertised from the same AS
Advanced Networks
103
Local Preference
Choosing paths internally
Within an AS:
• Set Local Preference to pick the path you want to
send data to
The higher Local Preference is preferred
Advanced Networks
104
Atomic Aggregate
Sometimes we aggregate paths that are
different (ie different AS sequence)
Atomic aggregate shows that some of the
destinations in this prefix are not necessarily
following the same path
Advanced Networks
105
Aggregator
Notify that an aggregation took place
• Which AS
• Which router
For management and traceback purposes
Advanced Networks
106
Example: The Intended Use
Provider
Peer
Advanced Networks
Customer
Peer
107
BGP Graph and Routing Policies
200
100
10
11
1
2
12
3
13
4
Up then down: 1, 10, 100, 200, 13, 4
No valleys, no up-down-up, no more than 1 peer-peer
Advanced Networks
108