When simplicity becomes complex
On the road to a scalable and dynamic SURFnet7 network
SURFnet7 indepth
Wouter Huisman
What do we want from a network?
• 
• 
• 
• 
Scalable
Flexible
Cost efficient
Endusers service requirements
•  Bottomline is keep it simple
SURFnet6 network
Hamburg
Groningen1
Hamburg Middenmeer1
Harlingen
Winschoten1
Leeuwarden
Assen1
Den Helder
Emmeloord
Beilen1
Dwingeloo1
Emmen1
Hoogeveen1
Subnetwork 4:
Blue Azur
Lelystad2
Meppel1
NLR
NLR
DLO
Alkmaar1
BT
ROADM
Amsterdam1
Haarlem1
Amsterdam2
Almere1
Lelystad1
Zwolle1
DAS3
Enschede1
Apeldoorn1
Leiden1
Münster SWOV
NBD
Hilversum1
Schiphol
Oegstgeest
Lisse
Subnetwork 3:
Red
Muenster1
3XLSOP
Breukelen1
Arnhem
Schiphol-Rijk
Subnetwork 1:
Green
TNO
Soesterberg
Zutphen1
Zoetermeer
LSOP Rtd
Wageningen1
ROADM
DenHaag
Rotterdam4
Amersfoort1
NNI
InHolland
Rotterdam1
Dordrecht1
Middelburg
Zierikzee
Bergen-op-Zoom
Nijmegen1
Ede
Nieuwegein1
Subnetwork 2:
Dark blue
Breda1
Den Bosch1
Venlo1
Eindhoven1
Vlissingen
ROADM
Krabbendijke
Venlo2
Roermond1
Subnetwork 5:
Grey
Sittard1
Tilburg1
Maasbracht1
Heerlen1
Heerlen2
Brussel
Maastricht1
Maastricht7
Aaken
Aachen Paris
3
Geneva (CERN) Geneva
Collapsed IP core
Border Routers
Located in Ams1
and Ams2
SURFnet
AS1103
Core Routers
Customer SURFnet – SNE masters BGP in de praktijk
Transport layer
Juniper T-series: 4
Ciena OM5200: 20
Nortel MERS 8600: 20
Ciena OME6500: 100
Ciena CPL: 100
Nortel OME1010/1060: 350
What to optimize of SURFnet6?
And make SURFnet7 simple… SURFnet7:
Scalable network
Institute
Applica<on Applica<on Applica<on Routed IP
(Layer 3)
✔2008 Next Generation
Ethernet
Service
Institute
Applica<on Applica<on Applica<on Service
(Layer 2)
Bandwidth
13
✔2010 Bandwidth
Optisch Transport
(Layer 0/1)
Concepts in the SURFnet7 tender
• 
• 
• 
• 
Single technology end-2-end transport
Remove as many OEO as possible
Introduce 40G and 100G
Improve the BoD solution
•  Selected Ciena CES solution
–  5410, 3930, 5150
–  Based on PBB-TE
Ciena CESD solution
Ciena 5410
•  Core switch, 2 per main POP
•  Modular switch with 400Gb/s per slot
– 
– 
– 
– 
– 
32-port 1G
4-port 10G
10-port10G
40-port 10G/48-port 1G
4-port 100G/2-port 40G
•  Pluggable optics
– 
– 
– 
– 
1G SX, LX, CWDM, DWDM
10G SR, LR, CWDM, DWDM
40G LR4 CFP
100G LR4 CFP
Ciena 5150
• 
• 
• 
• 
• 
Pizza box with extension slots
2U height
48x 1G SFP
4x 10G XFP
Dual AC/DC power
Ciena 3930
• 
• 
• 
• 
• 
Pizza box
1U height
8x 1G SFP
2x 1G/10G SFP+
Dual AC/DC power
Tiered Network Architecture
Tiered Network Architecture
Tiered Network Architecture
Tiered Network Architecture
Tiered Network Architecture
Tunable XFP & OTN XFP
5410 -­‐ CES CPL – DWDM OADM Transponder • 
• 
• 
• 
No transponder shelves required Tunable DWDM XFP used Save on power, footprint, OAM ~25% of the links require G.709 Scalability issue
•  Uptake lightpath services
was significant
•  Technology limitation
•  Bandwidth claimed for
100%, even though not
all used.
•  Single service per port
!
!
Statistical Multiplexing
EIR
a
CIR
a b a,b,c
b
c c
26
a
b
c
QoS
•  2 Color marking (Green and Yellow)
•  8 levels with .1p priority
–  Management mapped to 7
–  Lightpath mapped to 5
–  IP services mapped to 3
•  Strict queueing, which guarantees that
Lightpaths suffer last from congestion
PBB-TE Encapsulation
Payload Payload Payload Payload SA VID C-­‐VID C-­‐VID DA SA S-­‐VID S-­‐VID DA SA SA DA DA 802.1 802.1q 802.1ad I-­‐SID B-­‐VID B-­‐SA B-­‐DA 802.1ah 28
PBB-TE Protection
• 
Connection Oriented Ethernet
– 
– 
Primary & Backup Tunnels Monitored via 802.1ag CCMs
CCMs configurable for 10 msec intervals
MEP A
MEP C
Ingress PBB-TE
Edge Bridge
MEP B
Primary Tunnel
Protected Node
MEP D
Egress PBB-TE
Edge Bridge
29 •  Node config of a tunnel, transits, services
•  Concept VS
Creating a tunnel
A Startnode C B Transit node Transit node Z Endnode virtual-switch create vs TRANSIT-1334!
pbt transit create pbt-transit TRANSIT-1334_503 parent-port 5/3!
pbt transit add pbt-transit TRANSIT-1334_503 class-element 1 bvid 1334!
vlaninterface
create vlan
1337pbt-transit
name TRANSIT-1337!
virtual-switch
attach
TRANSIT-1334_503 vs
vlan add vlan 1337 port 3.1!
TRANSIT-1334!
addpbt-transit
vlan 1337 port
to_Asd001A_5410_01T_l1!
pbt transitvlan
create
TRANSIT-1334_101
parent-port 1/1!
pbt transit add pbt-transit TRANSIT-1334_101 class-element 1 bvid 1334!
virtual-switch interface attach pbt-transit TRANSIT-1334_101 vs
TRANSIT-1334!
Creating a tunnel
A Startnode C B Transit node Transit node Z Endnode pbt tunnel-group create group DH5150_5410s tunnel-sync on!
pbt encap-tunnel create static-encap P_DH5150_5410s tunnel-group
DH5150_5410s pair-index 4 port to_Asd001A_5150_02T_l1 bvid 1333 destbridge-name Asd001A_5150_02 weight 6!
pbt decap-tunnel create static-decap P_DH5150_5410s tunnel-group
DH5150_5410s pair-index 4 port to_Asd001A_5150_02T_l1 bvid 1333 srcbridge-name Asd001A_5150_02!
cfm service create static-encap P_DH5150_5410s name PBT-1333 next-mepid
1333 level 2!
cfm service set service PBT-1333 ccm-interval 100ms alarm-time 0!
cfm service enable service PBT-1333!
Creating a service
A Startnode C Z B Endnode virtual-circuit pbt create static-vc 3669IP tunnel 1Hedr1_Asd_D egress-isid 73860 ingress-isid 73860!
virtual-circuit pbt set static-vc 3669IP retain-stag yes!
virtual-switch add reserved-vlan 150!
virtual-switch ethernet create vs 3669IP vc 3669IP reserved-vlan 150!
virtual-switch ethernet set vs 3669IP description "3669IP"!
virtual-switch ethernet set vs 3669IP encap-cos-policy fixed encap-fixed-dot1dpri 3!
! MTU size increased to 9190 for CN 5150 Hedr001A_5150_01 port 1.1!
port set port 1.1 max-frame-size 9190!
port set port 1.1 vs-ingress-filter off!
virtual-switch ethernet add vs 3669IP port 1.1!
port set port 1.1 acceptable-frame-type all untagged-data-vs 3669IP!
aggregation set port 1.1 agg-mode manual!
lldp set port 1.1 mode rx-only notification off!
virtual-switch ethernet set port 1.1 vs 3669IP encap-cos-policy fixed encap-fixed-dot1dpri 3!
traffic-profiling set port 1.1 mode standard-dot1dpri!
traffic-profiling standard-profile create port 1.1 profile 1 name 3669IP01 cir 0 cbs 16 pir 1000000 ebs 64!
traffic-profiling set port 1.1 nonconform-standard-profile 3669IP01!
traffic-profiling enable port 1.1!
cfm service create vs 3669IP name 3669IP next 1 level 4!
cfm service set service 3669IP alarm-priority 3!
cfm service set service 3669IP alarm-time 10000!
cfm service set service 3669IP reset-time 3000!
cfm service set service 3669IP ccm-interval 1s!
cfm service enable service 3669IP!
! CN 5150 Hedr001A_5150_01 port 1.1 forced to Enabled!
port enable port 1.1!
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
virtual-circuit pbt create static-vc 2013LP tunnel 0809prot egress-isid 1073041 ingress-isid 1073041
virtual-circuit pbt set static-vc 2013LP retain-stag yes
virtual-switch add reserved-vlan 151
virtual-switch ethernet create vs 2013LP vc 2013LP reserved-vlan 151
virtual-switch ethernet set vs 2013LP description "2013LP"
virtual-switch ethernet set vs 2013LP encap-cos-policy fixed encap-fixed-dot1dpri 5
virtual-switch l2-cft set vs 2013LP tunnel-method transparent
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol 802.1x disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol lacp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol lacp-marker disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol lldp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol oam disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol rstp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-cdp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-dtp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-pagp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-pvst disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-stp-uplink-fast disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-udld disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-vtp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol gvrp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol vlan-bridge disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol all-bridges-block disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol bridge-block disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol garp-block disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol gmrp disposition forward
virtual-switch l2-cft enable vs 2013LP
port set port 1 vs-ingress-filter off
virtual-switch ethernet add vs 2013LP port 1
port set port 1 acceptable-frame-type all untagged-data-vs 2013LP
rstp disable port 1
mstp disable port 1
aggregation set port 1 agg-mode manual
lldp set port 1 mode disable notification off
port set port 1 untagged-ctrl-vs 2013LP
virtual-switch ethernet set port 1 vs 2013LP encap-cos-policy fixed encap-fixed-dot1dpri 5
traffic-profiling set port 1 mode standard-dot1dpri
traffic-profiling standard-profile create port 1 profile 1 name 2013LP01 cir 0 cbs 16 pir 1000 ebs 64
traffic-profiling set port 1 nonconform-standard-profile 2013LP01
traffic-profiling enable port 1
cfm service create vs 2013LP name 2013LP next 2 level 4
cfm service set service 2013LP alarm-priority 3
cfm service set service 2013LP alarm-time 10000
cfm service set service 2013LP reset-time 3000
cfm service set service 2013LP ccm-interval 1s
cfm service set service 2013LP dmm-interval 1s
cfm delay send service 2013LP local-mepid 2 mepid 1 repeat 1
cfm service set service 2013LP lmm-interval 1s
cfm frame-loss send service 2013LP local-mepid 2 mepid 1 repeat 1
Service CFM
MEP A Startnode MEP C B Z Endnode Service CFM is added to each service -­‐  Run from client sub port to client sub port -­‐  Takes the same path as customer data -­‐  Set at 1s interval, to allow hitless tunnel protec<on switches à ideal measurement point for availability calcula<ons Monthly Reporting
•  Availability is based on Service CFM alarms
–  Time between raise and clear of the alarm is the down time of
the service
•  Volume reporting is based on the standard MIB entries
–  Per port and per service (subservice of a physical port) is
possible
•  Considering reporting of latency and jitter
Can we manage elephant flows?
•  Unpredictability of the dynamic nature of BoD
•  BoD tend to generate large flows (> 1Gb/s)
•  Can we rely on the NMS?
BW enforcement for tunnels and services
•  Layered structure of tunnels and services
•  Each tunnel has a bandwidth profile
•  Services are added to tunnels as long as tunnel
can accommodate
Link capacity Tunnel profile Service BW Tunnel CIR: 2 node network
A Future lightpaths Exis<ng lightpaths B Consider: •  10G Link between A and B •  Filled with single tunnel and consuming 50% of CIR bandwidth •  5G SLP is to be reserved for new install •  Tunnel profile is changed to claim 100% CIR bandwidth •  In mean <me a DLP is reserved claim 1G bandwidth on the exis<ng tunnel, and consumes the bandwidth of the SLP service •  End result: SLP can not claim its bandwidth on this link SLP: sta<c lightpath DLP: dynamic lightpath Solution
•  DLP can only be transported across DLP tunnels
•  Claim Bandwidth in advance
–  Create dummy tunnels with required CIR for new
services
–  Or increase the CIR bandwidth of existing SLP
tunnels
2 elevator dilemma
•  Assume 2 elevators, both
with capacity for exact 10
people
2 elevator dilemma
•  Assume 2 elevators, both
with capacity for exact 10
people
Drop at least 1 person
LAG problem
•  Consider Lightpath flows of 6 Gb/s and of 5 Gb/s
–  A CES will autonomous decide, which LAG member to use
–  A LAG of 2 x 10G should be sufficient
–  But you have 50% chance on packet drop
•  Problem with LAGs with flows > 10% physical link speed
Solutions to the LAG issue
•  Use higher line rate instead of LAGs
–  Use 100G
•  Use LAGs, but be conscious not to put high bandwidth
flows on a LAG as you are loosing control
NMS implements Strict & Loose CAC flag
Strict CAC LAG
•  LAG with 1 member
•  Can accommodate any services
Loose CAC LAG
•  LAG with possibly more link members
•  Accommodates Lightpaths upto 1Gb/s And IP
wouter.huisman[at]surfnet.nl
W
www.surfnet.nl
+31 30 2 305 305
Creative Commons “Attribution” license:
http://creativecommons.org/licenses/by/3.0/