GÉANT Transport Network Evolution
Mian Usman
IP Network Architect, GÉANT
TNC16
13th June 2016
Networks ∙ Services ∙ People
www.geant.org
Agenda
• GÉANT Network
• Network Architecture Evaluation
• Transport SDN
Networks ∙ Services ∙ People
www.geant.org
2
GÉANT Network
• Two main networks
• Infinera based DWDM
• Juniper based IP/MPLS
• 3 main overlays on the IP/MPLS network
• Global R&E IP
• LHCONE
• IAS – GÉANT internet service GWS+Peerings
• A number of other services with much smaller data rate
•
•
•
•
GN+
MDVPN
BoD
Etc..
Networks ∙ Services ∙ People
www.geant.org
3
The DWDM network
• Elements providing
large capacity
Point-to-point
across defined
fibre routes
• 10G and 100G
Lambda services
are provided
directly on this
infrastructure
Networks ∙ Services ∙ People
www.geant.org
4
The IP/MPLS network
• Juniper MX routers
• IP traffic any to any
.. with policy
• All services expect
from Lambdas are
provided here
• Use the DWDM
network for some
of its core links,
leased capacity for
others
• Provides Overlays
Networks ∙ Services ∙ People
www.geant.org
5
GÉANT IP Topology
20160524 JB
UBUNTUNET
AS36944
TEIN3
AS23911
Cynet AP1
AS3268
BELnet AP1
AS2611
LINX
Internet
Exchange
60 Gbps
JANET AP1
AS786
LON
UK
INEX
Internet
Exchange
20 Gbps
STM-64; OC-192
10 Gbps
Shared
SURFnet AP2
AS1103
RedCLARA
AS27750
ESNET
AS293
IUCC AP1
AS378
STM-1; OC-3
155 Mbps
AMS
NL
KAUST
AS50999
DUB
IE x2
FCCN AP2
AS1930
LITnet AP2
AS2847
KAU
LT x2
CANARIE
AS6509
CANARIE
Backup
AS6509
FRA
DE
GARR AP2
AS137
SILK
AS132467
ARNES AP2
AS2107
BRA
SK
MAR
FR
TEIN3
AS24490
CARnet AP1
AS2108
RedIRIS AP2
AS766
Networks ∙ Services ∙ People
www.geant.org
ARN AP1
AS3208
MIL
IT
Ulakbim AP1
AS8517
CARnet AP2
AS2108
Cogent
AS 174
GARR AP1
AS137
GRnet AP2
AS5408
Cogent
AS 174
MREN AP1
AS40981
GRnet AP1
AS5408
BUC
RO
RoEduNet AP1
AS2614
BREN AP2
AS6802
Cogent
AS 174
ATH
GR x2
HUNGARnet AP1
AS1955
ZAG
HR
ACOnet AP2
AS1853
MAD
ES
LEVEL 3
AS3356
BUD
HU
LJU
SL
UoM / Malta AP1
AS12046
MIX-IT
Internet
Exchange
Sanet AP1
AS2607
ARNES AP1
AS2107
VIE
AT
Cogent
AS 174
SWITCH AP1
AS559
AMRES AP1
AS13092
HUNGARnet AP2
AS1955
VIX
Internet
Exchange
CERN AP1
AS513
RedIRIS AP1
AS766
CESnet AP2
AS2852
GRENA AP
AS20545
ACOnet AP1
AS1853
LEVEL 3
AS3356
SWITCH/CERN AP2
AS559 / AS513
JANET AP2
AS786
Cynet AP2
AS3268
Ulakbim AP2
AS8517
Internet2
AS11537
TIFR
AS2697
GEN
CH
CESnet AP1
AS2852
NASRA AP
AS47623
IUCC AP2
AS378
SINET
AS2907
RENATER
AP2
AS2200
PAR
FR
PRA
CZ
DFN AP1
AS680
AS20965/21320
CARIBNET
AS54308
PSNC AP2
AS8501
ESNET
AS293
RESTENA AP2
AS2602
RENATER AP1
AS2200
PSNC AP1
AS8501
POZ
PL
DE-CIX
Internet
Exchange
CAREN
AS197118
CIXP
Internet
Exchange
Upstream
BASNET AP2
AS21274
LEVEL 3
AS3356
NISN AP1
AS297
HEAnet AP2
AS1213
Internet
Exchange
DFN AP2
AS680
URAN AP1
AS12687
Cogent
AS 174
LUX
LU x2
FCCN AP1
AS1930
HAM
DE
ASGC
AS24167
RESTENA AP1
AS2602
LIS
PT x2
R&E Peer
LITnet AP1
AS2847
Internet2
AS11537
HEAnet AP1
AS1213
NREN
SigmaNet AP2
AS5538
RIG
LV
SINET
AS2907
IP only PoP
TAL
EE x2
NORDUnet AP2
AS2603
ENSTINet AP1
AS6879
AMS-IX
Internet
Exchange
STM-16; OC-48
2.4 Gbps
STM-4; OC-12
622 Mbps
IP and
Transmission PoP
SigmaNet AP1
AS5538
UBUNTUNET
AS36944
ESNET
AS293
GigE / 1 Gbps
GÉANT
SURFnet AP1
AS1103
TWAREN
AS7539
40 Gbps
30 Gbps
BRU
BE
BELNET AP2
AS2611
VIA
OLE
100 Gbps
Eenet AP2
AS3221
Eenet AP1
AS3221
NORDUnet AP1
AS2603
MARnet AP1
AS5379
BREN AP1
AS6802
SOF
BG
MARnet AP2
AS5379
6
Traffic by Overlay
IAS = Internet service
None = GÉANT Global R&E IP
LHCONE = LHCONE
Less than 10% of traffic falls outside of these 3 main overlays.
Networks ∙ Services ∙ People
www.geant.org
7
Traffic Growth
LHCONE Monthly average Gbps
70
60
50
40
30
20
10
Traffic Q on Q growth
ROUTING AGGREGATES
400
0
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEPT
OCT
NOV
DEC
JAN
FEB
MAR
APR
MAY
.. peaks are smoothed so actually higher!
LAMBDAS
Year on year growth
300
Q1 2015 -> Q1 2016
Growth ROUTING only
200
100
0
Q1 2015
Networks ∙ Services ∙ People
Q2 2015
www.geant.org
Q3 2015
Q4 2015
Q1 2016
47.70%
Q1 2015 -> Q1 2016
Growth Including
Lambdas
26.40%
8
Network Utilisation
On a reroute event peak was over 70Gbps!
50% Of the Infinera superchannels have
less than 200Gbps of available capacity
.. and the other leg out of Geneva has daily peaks of 50Gbps.
Link
FRA-PRA
BRA-PRA
PAR-LON
GEN-MIL
AMS-FRA
BRA-VIE
GEN-PAR
BRU-LON
BRA-BUD
FIN-VIE
AMS-BRU
Available capacity
80
90
130
170
180
190
190
190
190
200
200
..lambda services are not protected but
restorable, so additional capacity is
required on alternate routes for
rerouting.
Networks ∙ Services ∙ People
www.geant.org
Core Network Architecture
• Current GÉANT Core network is utilizing fully functional routers
•
•
•
•
•
For IP services, peering, packet forwarding, MPLS services, etc
Deployed in sparse mesh (minimal router off-load)
Backbone trunks follow fibre path
Many packets transit through intermediate router hops
Each transit packet consumes portion of the optical interface and forwarding capabilities
• Does this add any value?
• Transit through a backbone router incurs unnecessary costs
• Increases latency
• Increases utilisation of interfaces at IP layer
Networks ∙ Services ∙ People
www.geant.org
10
Evaluation of Network Architectures
• IPoWDM (Thin Core)
• Router manages optical link
• Moves the cost of transponder/optics from optical layer equipment to the router
• Still a sparse mesh topology – unnecessary transit packets
• Hollow Core
• Mesh routers through OTN core
• GÉANT uses single routers in a PoP which performs functionality of both core and edge
router
• In a 100G backbone full mesh would result in both increased complexity and costs
Networks ∙ Services ∙ People
www.geant.org
11
Evaluation of Network Architectures
• IP over OTN
•
•
•
•
•
•
Interconnect router with appropriate capacity trunk link
Increase meshing between core routers
Trunk links can be multiplexed by OTN layer into 500G super channel
Transit traffic is kept at OTN layer instead of transiting through multiple routers
Maximizes slot capacity at both IP and Optical layer
Enables GÉANT to scale and converge the two networks
• Testing of Infinera PXM (OTN) Cards completed
• Will be rolled out in Eastern part of the network
Networks ∙ Services ∙ People
www.geant.org
12
Current and Proposed Architecture
• Current Architecture
•
•
•
•
Follows the physical fibre
Significant amount of transit traffic
Requires two 100G interfaces per trunk
Over provisioned
• Proposed Architecture
•
•
•
•
IP Trunks based on traffic requirements between the two PoPs
Significantly reduces the amount of transit traffic through routers
Multiple trunk links can be setup on a single 100GE interface
Reduces the amount of 100GE interfaces required on both IP and DWDM layer
Networks ∙ Services ∙ People
www.geant.org
13
Transport SDN App
• Working closely with Infinera to develop a solution for managing OTN (PXM) cards
using an OpenFlow controller
• Paves the way to multi-layer interaction
• It uses standard open APIs
• Enables us to bring more intelligence to the core network
• Reduce overprovisioning
• Reduce number of transit packets going through intermediate routers
• In future dynamically modify capacity on OTN layer to cater for peaky traffic on Layer 2
and 3
Networks ∙ Services ∙ People
www.geant.org
14
Transport SDN
Current network architecture
GÉANT
Router
NREN
GÉANT
Router
GÉANT
WDM/OTN
NREN
GÉANT
Router
ODU4/2e links
GÉANT
Router
Networks ∙ Services ∙ People
www.geant.org
Transport SDN
Revised architecture with PXM cards
GÉANT
Router
Legacy ODU4/2e
links to routers via
TIM ports
NREN
100GbE
100GbE
GÉANT
WDM/OTN
GÉANT
Router
Networks ∙ Services ∙ People
www.geant.org
100GbE
100GbE
GÉANT
Router
IP trunks. ODUflex used to manage
bandwidth between PXM cards.
These are static (not OTS controlled)
100GbE
Trunks follow traffic rather than fibre
PXM cards support
termination of EVPL
circuits on vlans
GÉANT
Router
NREN
SDN Phase 2
B/W management with transport SDN
SDN controller
OTSv
GÉANT
Router
OTSc
L2 SDN switch
100GbE
100GbE
N x 10GbE
N x 10GbE
NREN
L2 SDN switch
GÉANT
WDM/OTN
100GbE
GÉANT
Router
GÉANT
Router
N x 10GbE
Networks ∙ Services ∙ People
www.geant.org
NREN
ODUflex and OTSv
used to dynamically
insatiate these links.
GÉANT
Router
Trunks now controlled by OTS
SDN Vision
Integrated L2 and L3 services
•
•
WAN optimised switches operate as full layer 3 device
and replaces the GÉANT router
Layer 2 service as developed in earlier phases continues
to be supported.
NREN
100GbE
100GbE
GÉANT
Router
GÉANT
WDM/OTN
SDN Switch
•
•
Switches connected to the WDM/OTN cloud via a
100GE port on a PXM card
Both IP trunks and p2p L2 circuits all are
provisioned in the same way – ODU 4/2e or
ODUflex circuits terminating on PXM or TIM
All circuits may be provisioned statically or
dynamically via the OTSv
Networks ∙ Services ∙ People
www.geant.org
100GbE
NREN
SDN Switch
100GbE
•
100GbE
GÉANT
Router
Multi-layer
SDN controller
Summary
SDN enables R&E to differentiate from Commercial ISPs
• Exponential Bandwidth Increase
• Highly unpredictable traffic
• Brings application awareness in the network
• Integration with other e-Infrastrcture
• Improve network visibility and reduce operational costs
• Programmability, offers ability to use API to link applications to the network
• Support for virtual networks
• Remove dependence on vendor software ability to outsource/crowdsource service
development - greater agility and speed of service development
• Niche service development feasible – R&E no longer dependent on the wants and needs
of the big service providers that vendors pander to – ATT, Verizon, BT, Telefonica et al
Networks ∙ Services ∙ People
www.geant.org
19
[email protected]
Thank you and any questions
Networks ∙ Services ∙ People
www.geant.org
Networks ∙ Services ∙ People
© GEANT Limited on behalf of the GN4 Phase 1 project.
The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 691567 (GN4-1).
www.geant.org
20
Unpredictable Traffic
• LHCONE Traffic increase over last 12 months
Networks ∙ Services ∙ People
www.geant.org
21