Software Defined Networking for
big-data science
Eric Pouyoul
Chin Guok
Inder Monga (presenting)
SRS presentation
November 15th, Supercomputing 2012
Acknowledgements
Many folks at ESnet who helped with the deployment and planning
•  Sanjay Parab (CMU), Brian Tierney, John Christman, Mark
Redman, Patrick Dorn among other ESnet NESG/OCS folks
Ciena Collaborators:
•  Rodney Wilson, Marc Lyonnais, Joshua Foster, Bill Webb
SRS Team
•  Andrew Lee, Srini Seetharaman
DOE ASCR research funding that has made this work possible
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
ESnet:
World’s Leading Science Network
US R&E
(DREN/Internet2/NLR)
ASIA-PACIFIC
(ASGC/Kreonet2/
TWAREN)
CANADA
(CANARIE)
RUSSIA
AND CHINA
(GLORIAD)
CANADA
(CANARIE)
LHCONE
FRANCE
(OpenTransit)
CERN
(USLHCNet)
RUSSIA
AND CHINA
(GLORIAD)
ASIA-PACIFIC
(KAREN/KREONET2/
NUS-GP/ODN/
REANNZ/SINET/
TRANSPAC/TWAREN)
SEATTLE
PNNL
ASIA-PACIFIC
(BNP/HEPNET)
AUSTRALIA
(AARnet)
BOSTON
BOISE
LATIN AMERICA
CLARA/CUDI
US R&E
(DREN/Internet2/
NISN/NLR)
SACRAMENTO
CERN
CHICAGO
BNL
NEW YORK
SUNNYVALE
ASIA-PACIFIC
(ASCC/KAREN/
KREONET2/NUS-GP/
ODN/REANNZ/
SINET/TRANSPAC)
LBNL
SLAC
US R&E
(DREN/Internet2/
NASA)
FNAL
AMES
CANADA
(CANARIE)
PPPL
ANL
WASHINGTON DC
US R&E
(NASA/NISN/
USDOI)
KANSAS CITY
DENVER
US R&E
(Internet2/
NLR)
JLAB
EUROPE
(GÉANT/
NORDUNET)
ASIA-PACIFIC
(SINET)
ORNL
NASHVILLE
ALBUQUERQUE
AUSTRALIA
(AARnet)
EUROPE
(GÉANT)
ATLANTA
LATIN AMERICA
(CLARA/CUDI)
El PASO
US R&E
(DREN/Internet2/
NISN)
100G IP Hubs
4x10G IP Hub
LATIN AMERICA
(AMPATH/CLARA)
HOUSTON
Major R&E
and International
peering connections
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Opportunities for innovation (1)
Elephant Flows: ‘big-data’ movement for Science, end-to-end
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Opportunity (2): Global Multi-Domain Collaborations like LHC
detector
1 PB/s
O(1-10) meter
CERN →T1
mile
s
Level 1 and 2 triggers
O(10-100) meters
kms
France
350
565
Italy
570
920
UK
625
1000
Netherlands
625
1000
Germany
700
1185
Spain
850
1400
Nordic
1300
2100
USA – New York
3900
6300
USA - Chicago
4400
7100
Canada – BC
5200
8400
Taiwan
6100
9850
The LHC Open
Network
Environment
(LHCONE)
Level 3 trigger
O(1) km
LHC Tier 0
Deep archive and send
data to Tier 1 centers
CERN Computer Center
~50 Gb/s (25Gb/s ATLAS, 25Gb/s CMS)
500-10,000 km
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Source: Bill
Johnston
The LHC Optical
Private Network
(LHCOPN)
LHC Tier 1 Data
Centers
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
Universities/
physics
groups
LHC Tier 2 Analysis
Centers
Software-Defined Networking
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
What is Software-Defined Networking?
(as defined by Scott Shenker, October 2011)
http://opennetsummit.org/talks/shenker-tue.pdf
“The ability to master complexity is not the same as the
ability to extract simplicity”
“Abstractions key to extracting simplicity”
“SDN is defined precisely by these three abstractions
•  Distribution, forwarding, configuration “
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Fundamental Network Abstraction:
a end-to-end circuit
Wavelength, PPP, MPLS, L2TP, GRE, NSI-CS…
A
Z
Switching points, store and forward, transformation …
Simple, Point-to-point, Provisonable
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
New Network Abstraction:
“WAN Virtual Switch”
WAN Virtual Switch
Simple, Multipoint, Programmable
Configuration abstraction:
•  Expresses desired behavior
•  Hides implementation on physical infrastructure
It is not only about the concept, but implementation
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Simple Example: One Virtual Switch per
Collaboration
ALCF
WAN Virtual Switch
OLCF
NERSC
US R&E
(DREN/Internet2/NLR)
ASIA-PACIFIC
(ASGC/Kreonet2/
TWAREN)
CANADA
(CANARIE)
RUSSIA
AND CHINA
(GLORIAD)
CANADA
(CANARIE)
LHCONE
FRANCE
(OpenTransit)
CERN
(USLHCNet)
RUSSIA
AND CHINA
(GLORIAD)
ASIA-PACIFIC
(KAREN/KREONET2/
NUS-GP/ODN/
REANNZ/SINET/
TRANSPAC/TWAREN)
SEATTLE
PNNL
ASIA-PACIFIC
(BNP/HEPNET)
AUSTRALIA
(AARnet)
BOSTON
BOISE
LATIN AMERICA
CLARA/CUDI
US R&E
(DREN/Internet2/
NISN/NLR)
SACRAMENTO
CERN
CHICAGO
BNL
NEW YORK
SUNNYVALE
ASIA-PACIFIC
(ASCC/KAREN/
KREONET2/NUS-GP/
ODN/REANNZ/
SINET/TRANSPAC)
LBNL
SLAC
US R&E
(DREN/Internet2/
NASA)
FNAL
AMES
CANADA
(CANARIE)
PPPL
ANL
WASHINGTON DC
US R&E
(NASA/NISN/
USDOI)
KANSAS CITY
DENVER
US R&E
(Internet2/
NLR)
JLAB
EUROPE
(GÉANT/
NORDUNET)
ASIA-PACIFIC
(SINET)
ORNL
NASHVILLE
ALBUQUERQUE
AUSTRALIA
(AARnet)
EUROPE
(GÉANT)
ATLANTA
LATIN AMERICA
(CLARA/CUDI)
El PASO
LATIN AMERICA
(AMPATH/CLARA)
US R&E
(DREN/Internet2/
NISN)
HOUSTON
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Programmability
Site Domain
OpenFlow Controller
OF protocol
WAN
Domain
WAN Virtual Switch
Expose ‘flow’ programming interface leveraging standard OF protocol
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
“Programmable” by end-sites
Program flows:
Science Flow1:
Science Flow2:
Science Flow3:
WAN Virtual Switch
App
1
App
1
App
2
OF Ctrl.
App
2
OF Ctrl.
Multi-Domain
Wide Area Network
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Many collaborations, Many Virtual Switches
WAN
Virtual
Switch
WAN
Virtual
Switch
WAN
Virtual
Switch
WAN
Virtual
WAN VirtualSwitch
Switch
Multi-Domain
Wide Area Network
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
SRS Demonstration
Physical Topology
DTNs
Ciena 5410
@Ciena booth
@ANL
@BNL
SRS
Brocade
@SCinet
OSCARS virtual circuits
NEC IP8800
@ LBL
DTNs: Data Transfer Nodes
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Virtual Switch Implementation:
Mapping abstract model to the physical
SRS Virtual Switch
A
Virtual
B
C
D
Create Virtual switch:
•  Specify edge OF ports
•  Specify backplane topology and
bandwidth
•  Policy constraints like flowspace
•  Store the switch into a topology
service
Physical
B
OF Switch
C
OF Switch
A
OF Switch
OF Switch
D
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
WAN Virtual Switch:
Deploying it as a service
App 1
Policy/Isolation of
customer OF
control
OF
Customer Flowvisor
App 2
End-site
OF controller
OF
OpenFlow API
Virtual Switch
Software stack
Virtual Switch Application
Infrastructure
Software, Slicing
and provisioning
OF
Virtual SW
Controller
Network Flowvisor
OSCARS
Client
OSCARS API
OSCARS
OF
OF Switch
Wide Area Network
OF Switch
OF Switch
Lawrence Berkeley National Laboratory
OF Switch
U.S. Department of Energy | Office of Science
Example of ping across WAN virtual switch
3. Need to ARP
6. virtual-mac-addresses learned and mapped to real flow
SRS Virtual Switch
4. ARP
2. Packet_in
4. ARP
H1
1. Ping H2
5. ARP response
H2
OF Switch
7. flow_mod
8. Ping H2
OF Switch
8. Ping H2
Lawrence Berkeley National Laboratory
OF Switch
U.S. Department of Energy | Office of Science
What does this mean for networking?
User
SDN
Controller
OpenFlow
Customer/User Control Plane
Policy and Isolation
Network
Service
Interface
WAN Virtual Switch
Programmable service provisioning plane
Network
Service
Interface
Multi-domain
Legacy and OpenFlow control plane
End-to-End Dataplane
•  Creation of a programmable network provisioning layer
•  Sits on top of the “network OS”
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Summary
• 
Powerful network abstraction
•  Files / Storage
• 
Benefits
•  Simplicity for the end-site
•  Works with off-the-shelf, open-source controller
•  Topology simplification
•  Generic code for the network provider
•  Virtual switch can be layered over optical, routed or switched network
elements
•  OpenFlow support needed on edge devices only, core stays same
•  Programmability for applications
•  Allows end-sites to innovate and use the WAN effectively
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Future Work
Harden the architecture and software implementation
•  Move from experiment to test service
Verify scaling of the model
•  Using virtual machines, other emulation environments
Automation and Intelligent provisioning
•  Work over multi-domain
•  Wizards for provisioning
•  Dynamic switch backplane
Create recurring abstractions
•  Virtual switch in campus
•  How do we deal with a “network” of virtual switches
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Questions – please contact imonga at es.net
www.es.net
Thank you!
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science
Computer virtualization
Lawrence Berkeley National Laboratory
U.S. Department of Energy | Office of Science