GÉANT IP Layer
Mian Usman
IP Network Architect, GÉANT
TNC15, Porto
17th June 2015
Networks ∙ Services ∙ People
www.geant.org
Contents
• Service Architecture
• Challenges
• SDN Approach
• White Box Switches
• Services and IP Trunks over Infinera Packet TIM
Networks ∙ Services ∙ People
www.geant.org
2
Services
Network
• IP, L2VPN, L3VPN, MDVPN,
Multicast, etc
• IP/MPLS Network
• DWDM Network
Infrastructure
Networks ∙ Services ∙ People
www.geant.org
Operations
Service Architecture
• Dark Fibre
3
Challenges
• Diverse Service Portfolio
• Support of Multicast and
MDVPN type services
• Drive for early adoption of
JUNOS version
• Demanding Big Science Users
• High performing and stable
yet innovative network
• Deliver services at competitive
cost
• Committed to long term
contracts and relationship
with supplier
• High switching cost (Dark
fibre)
Networks ∙ Services ∙ People
www.geant.org
4
Innovation in GÉANT Project
• Synectics model of “Cycling Worlds”
Service Activity
World
Networks ∙ Services ∙ People
Joint Research
Activity World
www.geant.org
5
SA1 and JRA2 Work
• SA1 will carry out two studies and deploy a pilot
implementation based on the results:
• Layer 1 infrastructure study
• Multi-domain NREN Transit network
• Run technology trials to assess new
technologies and new network solutions to
enhance the GÉANT network and improve cost
efficiency.
• JRA2 will look at how the existing service and
platform could be evolved and what new SDN
based connectivity service be offered to NREN
community:
• SDN based connectivity services (BoD)
• POC of SDX and SDN-IP using white label
switches
• SDN at Optical Layer using Infinera Packet TIM
• INaaS (Infrastructure and Network as a Service)
• NetIC (Network in Campus)
• Create a GÉANT network evolution plan
Networks ∙ Services ∙ People
www.geant.org
6
GÉANT Network Topology
GÉANT IP Topology
20150604 rgh
UBUNTUNET
AS36944
TEIN3
AS23911
NORDUnet
AS2603
BELNET
Backup
AS2611
EEnet
AS3221
BELnet
AS2611
BRU
BE
100 GE
SigmaNet
AS5538
SURFnet
AS1103
VIA
OLE
UBUNTUNET
AS36944
Cynet
AS3268
TWAREN
AS7539
JANET
AS786
40 GE
LON
UK
LINX
AS
30 GE
5G over 10G
SURFnet
Backup
AS1103
20 GE
RedCLARA
AS27750
STM-64; OC-192
10 GE
ESNET
AS293
Shared
AMS
NL
KAUST
AS50999
Internet2
AS11537
LITnet
AS2847
KAU
LT x2
DFN
Backup
AS680
AMS-IX
AS
CANARIE
AS6509
CANARIE
Backup
AS6509
STM-16; OC-48
2.4 Gbps
STM-4; OC-12
622 Mbps
HAM
DE
ASGC
AS24167
HEAnet
Backup
AS1213
STM-1; OC-3
155 Mbps
Cloud colours:
Blue=NREN
Green=R&E Peer
Red=IX
Orange=Upstream
Cogent
AS 174
DUB
IE x2
PSNC
AS8501
DE-CIX
AS15169
CESnet
AS2852
ESNET
AS293
FCCN
Backup
AS1930
LUX
LU x2
FCCN
AS1930
RESTENA
Backup
AS2602
IP only PoP
FRA
DE
CAREN
AS197118
RESTENA
AS2602
LIS
PT x2
IP and Transmission
PoP
POZ
PL
LEVEL 3
AS3356
NISN
AS297
HEAnet
AS1213
BASNET
AS21274
URAN
AS12687
SINET
AS2907
300 Mbps
GÉANT
RIG
LV
ENSTINet
AS6879
ESNET
AS293
IUCC
AS378
GigE / 1 Gbps
TAL
EE x2
NORDUnet
Backup
AS2603
60 GE
DFN
AS680
Cynet
Backup
AS3268
IUCC
Backup
AS378
SINET
AS2907
AS20965
PRA
CZ
Ulakbim
Backup
AS8517
ACOnet
AS1853
Internet2
AS11537
RENATER
AP1
AS2200
RENATER
AP2
AS2200
TIFR
AS2697
JANET
Backup
AS786
LJU
SI
VIE
AT
Cogent
AS 174
RedIRIS
Backup
AS766
AMRES
AS13092
ARNES
Backup
AS2107
VIX
AS1121
GARR
Backup
AS137
SWITCH/CERN
Backup
AS559 / AS513
BRA
SK
LEVEL 3
AS3356
GEN
CH
PAR
FR
CARIBNET
AS54308
SWITCH
AS559
CERN
AS513
MAR
FR
MAD
ES
LEVEL 3
AS3356
ARNES
AS2107
BUD
HU
ZAG
HR
MREN
AS40981
CARnet
Backup
AS2108
MIL
IT
ACOnet
Backup
AS1853
GARR
AS137
Ulakbim
AS8517
RoEduNet
AS2614
HUNGARnet
AS1955
BUC
RO
CAT
IT
SOF
BG
BREN
AS6802
MARnet
AS5379
RedIRIS
AS766
Networks ∙ Services ∙ People
www.geant.org
Cogent
AS 174
Malta
AS12046
CARnet
AS2108
TEIN3
AS24490
CESnet
Backup
AS2852
HUNGARnet
Backup
AS1955
SAnet
AS2607
SILK
AS132467
ARN
AS3208
GRnet
Backup
AS5408
ATH
GR
x2
GRnet
AS5408
GÉANT Network Topology
• GÉANT leveraging on the existing NREN
infrastructure to build GÉANT IP
network
• Layer 1 study will look at which fibre
routes could be potentially replaced
with CBFs or Alien Waves
• Multi-domain transit network study will
look at how NRENs can use the existing
NREN and GÉANT optical infrastructure
to peer with and offer transit to each
other
Networks ∙ Services ∙ People
www.geant.org
SDN Approach
• Separate elements of control plane into individual controllers
• Reduce reliance on the vendor OS
• Reduce service interruptions due to OS issues
• Reduce the time it takes to test the OS version prior to upgrades
• Multicast service managed by a separate controller
• No Multicast on Juniper Routers
Networks ∙ Services ∙ People
www.geant.org
9
GÉANT Network Topology
GÉANT IP Topology
20150604 rgh
UBUNTUNET
AS36944
TEIN3
AS23911
NORDUnet
AS2603
BELNET
Backup
AS2611
EEnet
AS3221
BELnet
AS2611
BRU
BE
100 GE
SigmaNet
AS5538
SURFnet
AS1103
VIA
OLE
UBUNTUNET
AS36944
Cynet
AS3268
TWAREN
AS7539
JANET
AS786
40 GE
LON
UK
LINX
AS
30 GE
5G over 10G
SURFnet
Backup
AS1103
20 GE
RedCLARA
AS27750
STM-64; OC-192
10 GE
ESNET
AS293
Shared
AMS
NL
KAUST
AS50999
Internet2
AS11537
LITnet
AS2847
KAU
LT x2
DFN
Backup
AS680
AMS-IX
AS
CANARIE
AS6509
CANARIE
Backup
AS6509
STM-16; OC-48
2.4 Gbps
STM-4; OC-12
622 Mbps
HAM
DE
ASGC
AS24167
HEAnet
Backup
AS1213
STM-1; OC-3
155 Mbps
Cloud colours:
Blue=NREN
Green=R&E Peer
Red=IX
Orange=Upstream
Cogent
AS 174
DUB
IE x2
PSNC
AS8501
DE-CIX
AS15169
CESnet
AS2852
ESNET
AS293
FCCN
Backup
AS1930
LUX
LU x2
FCCN
AS1930
RESTENA
Backup
AS2602
IP only PoP
FRA
DE
CAREN
AS197118
RESTENA
AS2602
LIS
PT x2
IP and Transmission
PoP
POZ
PL
LEVEL 3
AS3356
NISN
AS297
HEAnet
AS1213
BASNET
AS21274
URAN
AS12687
SINET
AS2907
300 Mbps
GÉANT
RIG
LV
ENSTINet
AS6879
ESNET
AS293
IUCC
AS378
GigE / 1 Gbps
TAL
EE x2
NORDUnet
Backup
AS2603
60 GE
DFN
AS680
Cynet
Backup
AS3268
IUCC
Backup
AS378
SINET
AS2907
AS20965
PRA
CZ
Ulakbim
Backup
AS8517
ACOnet
AS1853
Internet2
AS11537
RENATER
AP1
AS2200
RENATER
AP2
AS2200
TIFR
AS2697
JANET
Backup
AS786
LJU
SI
VIE
AT
Cogent
AS 174
RedIRIS
Backup
AS766
AMRES
AS13092
ARNES
Backup
AS2107
VIX
AS1121
GARR
Backup
AS137
SWITCH/CERN
Backup
AS559 / AS513
BRA
SK
LEVEL 3
AS3356
GEN
CH
PAR
FR
CARIBNET
AS54308
SWITCH
AS559
CERN
AS513
MAR
FR
MAD
ES
LEVEL 3
AS3356
ARNES
AS2107
BUD
HU
ZAG
HR
MREN
AS40981
CARnet
Backup
AS2108
MIL
IT
ACOnet
Backup
AS1853
GARR
AS137
Ulakbim
AS8517
RoEduNet
AS2614
HUNGARnet
AS1955
BUC
RO
CAT
IT
SOF
BG
BREN
AS6802
MARnet
AS5379
RedIRIS
AS766
Networks ∙ Services ∙ People
www.geant.org
Cogent
AS 174
Malta
AS12046
CARnet
AS2108
TEIN3
AS24490
CESnet
Backup
AS2852
HUNGARnet
Backup
AS1955
SAnet
AS2607
SILK
AS132467
ARN
AS3208
GRnet
Backup
AS5408
ATH
GR
x2
GRnet
AS5408
White Box Switches
• Juniper Support Contract up for renewal in Aug-16
• Opportunity to replace Juniper MX routers in smaller PoPs with White Box Switches
• Reduce the maintenance and support cost
• Offer additional functionality
• Innovative solutions in GÉANT production network
Networks ∙ Services ∙ People
www.geant.org
11
100G Aggregation Switches
• GÉANT running out of slots on Juniper MX in bigger PoPs e.g. Frankfurt, Amsterdam and Geneva
To Paris
To Frankfurt
GÉANT
GENEVA
MX-960
To Madrid
To Milan
100G
CERN
GARR
100G
100G
Aggregation Switch
ESNET
Networks ∙ Services ∙ People
www.geant.org
100G
N x10G
RENATER
12
GÉANT Network Topology
GÉANT IP Topology
20150604 rgh
UBUNTUNET
AS36944
TEIN3
AS23911
NORDUnet
AS2603
BELNET
Backup
AS2611
EEnet
AS3221
BELnet
AS2611
BRU
BE
100 GE
SigmaNet
AS5538
SURFnet
AS1103
VIA
OLE
UBUNTUNET
AS36944
Cynet
AS3268
TWAREN
AS7539
JANET
AS786
40 GE
LON
UK
LINX
AS
30 GE
5G over 10G
SURFnet
Backup
AS1103
20 GE
RedCLARA
AS27750
STM-64; OC-192
10 GE
ESNET
AS293
Shared
AMS
NL
KAUST
AS50999
Internet2
AS11537
LITnet
AS2847
KAU
LT x2
DFN
Backup
AS680
AMS-IX
AS
CANARIE
AS6509
CANARIE
Backup
AS6509
STM-16; OC-48
2.4 Gbps
STM-4; OC-12
622 Mbps
HAM
DE
ASGC
AS24167
HEAnet
Backup
AS1213
STM-1; OC-3
155 Mbps
Cloud colours:
Blue=NREN
Green=R&E Peer
Red=IX
Orange=Upstream
Cogent
AS 174
DUB
IE x2
PSNC
AS8501
DE-CIX
AS15169
CESnet
AS2852
ESNET
AS293
FCCN
Backup
AS1930
LUX
LU x2
FCCN
AS1930
RESTENA
Backup
AS2602
IP only PoP
FRA
DE
CAREN
AS197118
RESTENA
AS2602
LIS
PT x2
IP and Transmission
PoP
POZ
PL
LEVEL 3
AS3356
NISN
AS297
HEAnet
AS1213
BASNET
AS21274
URAN
AS12687
SINET
AS2907
300 Mbps
GÉANT
RIG
LV
ENSTINet
AS6879
ESNET
AS293
IUCC
AS378
GigE / 1 Gbps
TAL
EE x2
NORDUnet
Backup
AS2603
60 GE
DFN
AS680
Cynet
Backup
AS3268
IUCC
Backup
AS378
SINET
AS2907
AS20965
PRA
CZ
Ulakbim
Backup
AS8517
ACOnet
AS1853
Internet2
AS11537
RENATER
AP1
AS2200
RENATER
AP2
AS2200
TIFR
AS2697
JANET
Backup
AS786
LJU
SI
VIE
AT
Cogent
AS 174
RedIRIS
Backup
AS766
AMRES
AS13092
ARNES
Backup
AS2107
VIX
AS1121
GARR
Backup
AS137
SWITCH/CERN
Backup
AS559 / AS513
BRA
SK
LEVEL 3
AS3356
GEN
CH
PAR
FR
CARIBNET
AS54308
SWITCH
AS559
CERN
AS513
MAR
FR
MAD
ES
LEVEL 3
AS3356
ARNES
AS2107
BUD
HU
ZAG
HR
MREN
AS40981
CARnet
Backup
AS2108
MIL
IT
ACOnet
Backup
AS1853
GARR
AS137
Ulakbim
AS8517
RoEduNet
AS2614
HUNGARnet
AS1955
BUC
RO
CAT
IT
SOF
BG
BREN
AS6802
MARnet
AS5379
RedIRIS
AS766
Networks ∙ Services ∙ People
www.geant.org
Cogent
AS 174
Malta
AS12046
CARnet
AS2108
TEIN3
AS24490
CESnet
Backup
AS2852
HUNGARnet
Backup
AS1955
SAnet
AS2607
SILK
AS132467
ARN
AS3208
GRnet
Backup
AS5408
ATH
GR
x2
GRnet
AS5408
PXM
GÉANT IP Trunk Over Packet Tim Use Case
CZ
100G
www.geant.org
PL
40
G
20
G
DE
100G
CH
100G
100G
IT
AT
40G
SK
20
G
GR
Networks ∙ Services ∙ People
30G
100G
G
20
• Current GÉANT Eastern Ring Topology
• IP Trunks follow the physical fibre path
• Started the trunk link optimization based
on traffic flows
• Majority of the traffic flows from
Western Ring to Eastern Ring
• Majority of the traffic goes through AT or
HU router
• Majority of the traffic on CZ and SK
router is a pass through traffic
SI
HR
BG
RO
HU
PXM
GÉANT IP Trunk Over Packet Tim Use Case
3x10G
100G
CH
3x10
G
DE
0G
1
x
2
3 x1
0G
100G
100G
0G
4 x1
CZ
AT
www.geant.org
40G
HU
2x
10
G
GR
Networks ∙ Services ∙ People
4x10G
IT
0G
2x1
• Trunk Link Optimization using pass
through links
• No through traffic via smaller PoPs
e.g. GR, SI, SK, HR
100G
SK
SI
HR
BG
RO
PL
PXM
GÉANT IP Trunks Over Packet Tim
Networks ∙ Services ∙ People
www.geant.org
PXM
GÉANT IP Trunk Over Packet Tim Use Case
50G
30G
0G
2 x1
G
40
www.geant.org
3x10
G
DE
40G
CH
3 x1
0G
CZ
30G
30G
IT
AT
40G
HU
2x
10
G
GR
Networks ∙ Services ∙ People
3x10G
100G
0G
2x1
• Reduce the number of 10GE
interfaces
• Reduce the flow limitations
based on 10GE interfaces
• Makes capacity planning
easier
• Reduce the number of slots
needed on the Juniper MX
router
• Packet TIM can be used to
upgrade Western Ring hence
reducing the number of 100G
required on Juniper
SK
SI
HR
BG
RO
PL
GÉANT Network Topology
GÉANT IP Topology
20150604 rgh
UBUNTUNET
AS36944
TEIN3
AS23911
NORDUnet
AS2603
BELNET
Backup
AS2611
EEnet
AS3221
BELnet
AS2611
BRU
BE
100 GE
SigmaNet
AS5538
SURFnet
AS1103
VIA
OLE
UBUNTUNET
AS36944
Cynet
AS3268
TWAREN
AS7539
JANET
AS786
40 GE
LON
UK
LINX
AS
30 GE
5G over 10G
SURFnet
Backup
AS1103
20 GE
RedCLARA
AS27750
STM-64; OC-192
10 GE
ESNET
AS293
Shared
AMS
NL
KAUST
AS50999
Internet2
AS11537
LITnet
AS2847
KAU
LT x2
DFN
Backup
AS680
AMS-IX
AS
CANARIE
AS6509
CANARIE
Backup
AS6509
STM-16; OC-48
2.4 Gbps
STM-4; OC-12
622 Mbps
HAM
DE
ASGC
AS24167
HEAnet
Backup
AS1213
STM-1; OC-3
155 Mbps
Cloud colours:
Blue=NREN
Green=R&E Peer
Red=IX
Orange=Upstream
Cogent
AS 174
DUB
IE x2
PSNC
AS8501
DE-CIX
AS15169
CESnet
AS2852
ESNET
AS293
FCCN
Backup
AS1930
LUX
LU x2
FCCN
AS1930
RESTENA
Backup
AS2602
IP only PoP
FRA
DE
CAREN
AS197118
RESTENA
AS2602
LIS
PT x2
IP and Transmission
PoP
POZ
PL
LEVEL 3
AS3356
NISN
AS297
HEAnet
AS1213
BASNET
AS21274
URAN
AS12687
SINET
AS2907
300 Mbps
GÉANT
RIG
LV
ENSTINet
AS6879
ESNET
AS293
IUCC
AS378
GigE / 1 Gbps
TAL
EE x2
NORDUnet
Backup
AS2603
60 GE
DFN
AS680
Cynet
Backup
AS3268
IUCC
Backup
AS378
SINET
AS2907
AS20965
PRA
CZ
Ulakbim
Backup
AS8517
ACOnet
AS1853
Internet2
AS11537
RENATER
AP1
AS2200
RENATER
AP2
AS2200
TIFR
AS2697
JANET
Backup
AS786
LJU
SI
VIE
AT
Cogent
AS 174
RedIRIS
Backup
AS766
AMRES
AS13092
ARNES
Backup
AS2107
VIX
AS1121
GARR
Backup
AS137
SWITCH/CERN
Backup
AS559 / AS513
BRA
SK
LEVEL 3
AS3356
GEN
CH
PAR
FR
CARIBNET
AS54308
SWITCH
AS559
CERN
AS513
MAR
FR
MAD
ES
LEVEL 3
AS3356
ARNES
AS2107
BUD
HU
ZAG
HR
MREN
AS40981
CARnet
Backup
AS2108
MIL
IT
ACOnet
Backup
AS1853
GARR
AS137
Ulakbim
AS8517
RoEduNet
AS2614
HUNGARnet
AS1955
BUC
RO
CAT
IT
SOF
BG
BREN
AS6802
MARnet
AS5379
RedIRIS
AS766
Networks ∙ Services ∙ People
www.geant.org
Cogent
AS 174
Malta
AS12046
CARnet
AS2108
TEIN3
AS24490
CESnet
Backup
AS2852
HUNGARnet
Backup
AS1955
SAnet
AS2607
SILK
AS132467
ARN
AS3208
GRnet
Backup
AS5408
ATH
GR
x2
GRnet
AS5408
PXM
GÉANT Services Over Packet Tim
• The Packet TIM can also be to used deliver deterministic network services to the NRENs and end users
• Bandwidth on-demand and GÉANT Plus circuits can also be delivered over optical layer
• The big science users overlay networks like LHCONE and LHCOPN can also benefit from using packet
TIM
• E-LAN service which connects NRENs at Optical Layer and enable them to peer directly with each
other or using a route server
Networks ∙ Services ∙ People
www.geant.org
PXM
OTS enabled
• Open Transport Switch (OTS) provides an Open Flow interface to the
DTN-X
• OTS adds transport extensions to the Open Flow interface
• Initially REST based interface
• First availability expected in Q4 2015 however, early versions available
now for evaluation.
OTS would make a useful API to allow big science users to request EVP-LINE
and EVP-LAN services via PXM
Networks ∙ Services ∙ People
www.geant.org
PXM
Evaluation and trial
• Field trial in 2015.
Evaluation and service development
• Field trial integration of PXM and OTS.
•
Evaluate OTS and PXM together.
Networks ∙ Services ∙ People
www.geant.org
Cambridge Lab
• Cambridge Lab
•
•
•
•
•
•
3 x Juniper MX-480
1 x Juniper MX-960
1 x Juniper M120
2 x PICA8 Switches
2 x Dell Switches (Cumulus Linux)
Servers for VMs
• Working with vendors to test new products (Infinera Packet TIM)
Networks ∙ Services ∙ People
www.geant.org
22
Summary
• Optimise the Layer 3 network by using direct pass through trunks and
removing routers from some PoPs
• Move services and traffic down a layer to reduce the cost of the network
• Make use of SDN and advance path computation to build intelligence in
the network
Networks ∙ Services ∙ People
www.geant.org
23
[email protected]
Thank you and any questions
Networks ∙ Services ∙ People
www.geant.org
Networks ∙ Services ∙ People
www.geant.org
24
PXM
Use case – LHCOPN/LHCONE
LHCOPN: High capacity TDM, but not flexible
LHCONE: flexibility of L3 VPN, but no reserved bandwidth
Can we give users the best of both solutions?
Networks ∙ Services ∙ People
www.geant.org
PXM
EPL & EVPL Services over Layer 1 VPN
Tier 1 site
NREN
Tier 2 Campus
NREN
GÉANT 500G fiber cloud
OF/REST
User service
requests
• MEF type EVP-LAN or EVP-Line
• pool of provisionable OTN B/W
• OTS REST/OF API to allow experimenter’s
applications to manage connectivity
Networks ∙ Services ∙ People
www.geant.org
CERN
Wigner Data Center
LHCOPN
• NRENs are providing LHC with point-to-point Layer2 circuits
• LHC centers built a virtual routed network out of the circuits
• LHC centers are providing Network Services to each other:
• CERN is providing un-restricted transit
• Some centers are providing limited transit
• Some LHC centers are peering
• NRENs support individual link operations & management
• LHC Sites are responsible for network management (layer 3 configurations)
including operations, monitoring, troubleshooting, capacity planning, security
management, AUP enforcement, etc.
Networks ∙ Services ∙ People
www.geant.org
LHCONE – as it exists at the moment
• LHCONE is currently setup as an overlay VRF on existing NREN
infrastructure which are interconnected via regional networks and Open
Exchanges
• NRENs provide the network including core links and routers as a
virtual overlay on their regular infrastructure
• NRENs have a peering or transit relationship with each other
• LHC centers are strictly users of the services
• Restrictions apply to the advertised IP Space
• LHCONE infrastructure includes dedicated access, Trans-Atlantic and some
backbone links
Networks ∙ Services ∙ People
www.geant.org
LHC Workshop Recap
• Rather than maintaining distinct networks, the LHC community should aim to unify its network
infrastructure
• Traffic aggregation on few links
• Concerns
• If the T1s and T2s upgrade to 100G, then the global infrastructure needs to follow
• LHCONE Evolution
• Currently LHCONE exists side-by-side with general R&E infrastructure
• Traffic is segregated but what's the real benefit?
Networks ∙ Services ∙ People
www.geant.org