NL-T1 Expectations, findings, and
innovation
Bas Kreukniet, Sr Network Specialist at SURFSARA
Geneva Workshop 10 Februari 2014
Outline
1. Expectations from NL-T1 grid administrators
2. Findings while connecting to the LHCONE
3. Innovation: Ethernet OAM and NSI
Expectations from NL-T1 administrators
Advice from grid administrators NL-T1:
• Bulk data with simple applications (grid-FTP).
“Keep it simple”
• Network provisioning from application is
considered “complex”
• Don’t rush to merge LHCOPN with LHCONE
NL-T1 connected to LHCONE since 20 Jan 2014
BGP routing starts at the connected site.
BGP routing for T1’s:
Not only a “NREN thing” – it already starts at your
organisation!
Focus on some BGP topics for connecting sites to
LHC networks
LHCOPN connectivity
LHCONE connectivity
LHCONE.inet.0: 133 destinations, 133 routes
(133 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
NL-T1
AS1162
GÉANT
AS20965
LHCONE-RS
CERN
AS20641
128.142.0.0/16 *[BGP/170] 1w2d 04:07:47, MED 120, localpref 100
AS path: 20965 20641 513 I
> to 62.40.126.161 via xe-4/0/1.2012
CERN
AS513
Preferred route: LHCOPN, LHCONE or Internet
•
•
•
•
primary: direct T1–T1 over LHCOPN
secondary: T1–T1 over LHCOPN via another T1
tertiary: LHCONE
quaternary: Internet
Preferred route: route (a)symmetry
If everyone makes this choice, all connections are symmetric.
But: sites may have different preferences:
ISP-A
10 Gb/s
site 1
10 Gb/s
1 Gb/s
100 Gb/s
ISP-B
site 2
Tie-breaker between LHCOPN and LHCONE
CNAF routers at NL-T1
LHCOPN
131.154.128.0/17 *[BGP/170] 14:58:18, localpref 100
AS path: 34878 137 I
LHCONE
131.154.128.0/17 *[BGP/170] 1w2d 04:23:40, MED 120, localpref 100
AS path: 20965 137 I
LHCOPN / LHCONE route preference
BNL routes at NL-T1
LHCOPN
130.199.185.0/24 *[BGP/170] 2w5d 09:35:43, MED 10, localpref 100
AS path: 513 43 I
> to 192.16.166.73 via xe-1/1/0.0
[BGP/170] 3d 01:15:22, MED 51, localpref 100
AS path: 39590 513 43 I
> to 109.105.124.17 via xe-2/1/0.0
LHCONE
130.199.185.0/24 *[BGP/170] 1w2d 04:27:43, MED 120, localpref 100
AS path: 20965 293 43 I
> to 62.40.126.161 via xe-4/0/1.2012
Problems and concerns regarding BGP
• A site advertising his routes has no control who to
send a route to. At best they can give hints with BGP
communities.
• The site receiving a route decides which route to
accept and how to accept.
BGP in LHC networks
Only As is not enough. More info needed.
 community for origin of a route T1/T2?
Or even better:
-site connected to LHCOPN
-site connected to LHCONE
- Site connected to both:
Type A “prefer LHCONE for this route”
Type B “prefer LHCOPN for this route”
- Specials: dedicated link between two (T1) sites.
(“VPN”or “private link”)
Route Preference Solutions
• Idea: Tag routes with two types of communities:
• One for origin or source
• One for destinations
• See also: BGP hinting by Martin Sweeny (Indiana U)
• BGP Always-compare-MED always on. We sometimes
add metrics on incoming routes.
Operational Issues
• We received routes over LHC from a site,
but traffic we send is blackholed by that site
• The site was still reachable over the Internet.
• This happened to us twice recently: on LHCOPN (accidental route
redistribution) and LHCONE (incoming IP filter).
• Configuration errors will be made (we’re also just human)
• LHCOPN Link NL-T1 – TRIUMF link is still unstable
• 31 outages last 4 months
 Monitoring remains important!
LHCONE and LHCOPN layer 2 monitoring
Ethernet OAM monitoring or Layer 2 monitoring
NIKHEF, TRIUMF to participate as Measurement Points
Looking for T1’s and T2 to participate
Advantages:
• Layer2 keep-alive, ping and traceroute
• Interdomain, intervendor solution
• L2 devices can be made visible
• unidirectional fibercuts can be signalled
LHCONE Innovation
NL-T1 likes to partcipate in NSI for LHCONE
NSI experiences so far:
• Use-case: Life Science Grid (LSG) in Holland
makes use of NSI since autumn 2013.
• Freek contributed to standard
• Sander wrote NSI client and implemented it for
“Cloud Bypassing” in the Life Science Grid.
Cloud Bypassing on Life Science Grid (LSG)
Life Science grid
cluster @ TUD
Life Science grid
cluster @ AMC
d ef a
ult
AMC
campus
network
campus
network
u lt
defa
Regular
Internet
SARA data
center
network
fa
de
RUG
campus
network
ul t
Life Science grid
cluster @ SURFsara
de
fa
ult
Life Science grid
cluster @ RUG
Compute clusters at ±10 locations in the Netherlands
Cloud Bypassing on Life Science Grid (LSG)
Life Science grid
cluster @ TUD
Life Science grid
cluster @ AMC
d ef a
ult
AMC
campus
network
campus
network
M
or
(B
Regular
Internet
u lt
ific
ec
p
s
e
P)
or
G
M
(B
e
sp
e
GP cific
)
RUG
campus
network
SARA data
center
network
lt
fa u
de
More specific
(BGP)
Life Science grid
cluster @ SURFsara
defa
de
fa
SURFnet
dynamic
lightpaths
ult
More specific
(BGP)
Life Science grid
cluster @ RUG
Offloading campus networks (some only have 1 Gb/s Internet)
LHCONE Innovation: Offloading is cheaper
Internet full routing (incl backup): € 8k – €10k per month for 10 Gb/s
LHCONE or dynamic lightpath: € 2k – 3k per month for 10 Gb/s
Questions?
Erik
Ruiter
Bas
Kreukniet
Diederik
Vandevenne
Sander
Boele
Farhad
Davani
Freek
Dijkstra