INF-SEC2031
Been There Done
That: YHMAN’s
Private Cloud
Implementation and
Lessons Learnt
Kevin Barrass
YHMAN Network Development
and Support
Jonathan Gohstand
VMware, Inc.
Ed Carter
YHMAN Business Manager
#vmworldinf
Disclaimer
 This session may contain product features that are
currently under development.
 This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
 Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
 Technical feasibility and market demand will affect final delivery.
 Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
2
YHMAN Presentation to VMWorld Europe, 10th October 2012
BEEN THERE, DONE THAT:
YHMAN Private Cloud Implementation & Lessons Learnt
Ed Carter - YHMAN Business Manager
Kevin Barrass - YHMAN Network Support & Development Officer
Leadership in the Public Sector
The Practical Cloud: YHMAN
Best Practice: YHMAN
YHMAN Ltd ®
3
YHMAN Presentation to VMWorld Europe, 10th October 2012
Presentation Content
1. YHMAN Shared Virtual Data Centre (SVDC)
a Private Community Cloud
2. Stretched Cluster Data Centre Topology
with Secure Tenancy & Network Access
3. Lessons Learnt & The Way Forward
‘Right Here, Right Now’
- vCloud Networking and Security, vCNS 5.1
(including Live Demo vCNS Edge Generic Firewall/NAT)
- vCloud Director with vCNI or VXLAN
- VXLAN
4. Q & A
YHMAN Ltd ®
4
Background
YHMAN is a joint venture company of 8 universities in Yorkshire UK
est. 1998
The business drivers - ‘do more for less, better’
•
Funding changes require UK universities to
deliver even more within tightening budgets
•
Institutions must meet carbon reduction commitments
•
Opportunities to exploit economies of scale & balance asset
utilisation across shared service partners
•
Increasing pressure to deliver measurable cost efficiencies
•
To enable growth and enhanced service standards
•
Stringent security requirements to adhere to
YHMAN Ltd ®
5
Unique resilient ‘stretched’ 80Km Data Centre Network (DCN), currently based
on 3 nodes, provides performance, business continuity & disaster recovery
JANET/YHMAN Core Network Connection Points
(Points-of-Presence, PoP)
80km
Scalable Optical Network Infrastructure:
• Support for 4Gbps, 10, 40 & 100Gbps wavelengths over
wide area distances using C/DWDM
• Support for Ethernet & Fibre Channel Protocols
Overlay Virtual Data Centre Network:
• Low Latency allowing synchronous 2- or 3-way data
storage mirroring
• Providing the Data Centre (DC) interconnects,
currently 3 DCs but more can be provisioned
as demand
grows, optimised for access performance
YHMAN Ltd ®
6
• Multiple Spanning Tree, 802.1s
• 802.1q and 802.3ad DC Interconnects
• VRRP/HSRP
YHMAN Ltd ®
7
YHMAN Ltd ®
8
YHMAN Ltd ®
9
YHMAN Ltd ®
10
• SVDC currently uses vDS version 4.1.0
• Simplifies Management
• Maintain Portgroup consistency across all hosts in cluster
• Ingress traffic shaping as well as Egress
• Shape traffic going in/out of vShield Edge external interface to
control tenancy access to internet.
YHMAN Ltd ®
11
YHMAN Ltd ®
12
1
2
3
4
YHMAN Ltd ®
13
SVDC Tenants VM managers provided with
login access to SVDC vCenter Server to
manage VMs assigned to tenant.
Permissions provided to tenant to perform:
Create VMs
Power on/off VMs
Configure VMs
Console to VM
Install VMware Tools
Upload/Download from Datastore
Create Snapshots/Templates/Clones on VMs
VM Deployment options:
1.Tenant creates VMs/vAPP’s
2.Tenant deploys VM/vAPP’s
from Templates.
YHMAN Ltd ®
14
1. VMware vSphere and vShield is providing a stable and
scalable solution
2. Relying on vCenter client to provide a cloud-like interface
for our customers is not ideal
• Complex vCenter permissions, easy to make mistakes
• Opening large number of infrastructure IP’s to clients
3. Using VLAN’s adds additional admin overhead, change
complexity and makes solution less flexible
• VLAN’s need to be created by Network Team
• Systems Team add VLANs to Blade Chassis uplinks
(in our case, HP chassis Flex-10 cards)
• Systems Team create VLAN backed Portgroup
• Clients cannot self-provision networks on demand
YHMAN Ltd ®
15
MOVING FORWARD
1. vCloud Network & Security deployment
a. Involved with VMware Beta Testing of vCNS 5.1 primarily Edge
b. Improve on existing vShield service we offer
• Advanced Encryption Standard (AES-NI) support for secure VPN
• HTTPS and TCP support on load balancer
2. vCloud Director
a. Proof of concept with vCloud Director planned
b. Offer true Cloud portal for SVDC clients with
Software Defined Networking based on either:
• vCloud Director Networking Infrastructure (vCDNI)
MAC-in-MAC encapsulation or
• Virtual eXtensible Local Area Network (VXLAN)
MAC-in-IP encapsulation
3. VXLAN
a. Ongoing discussions with VMware
b. Internal testing along with “pie in the sky” thoughts
YHMAN Ltd ®
16
vCNS BETA TESTING
Features Tested
• Edge
• Firewall
• NAT/Routed
• IPSec VPN
• SSL VPN
• Edge HA
• Load Balancer
• Basic Testing of App
• Two Beta Builds tested
with all results regularly
fed back to
VMware
YHMAN Ltd ®
17
vCNS BETA RESULTS
• Edge HA provides fast stateful failover
• SSL VPN provides greater agility for our users
• Ability to connect into tenancy from anywhere securely not just
from site with IPSec VPN
• Improved Edge Command Line Interface (CLI)
• View Flow table Information
• View Firewall rules with matching flow info
• View statistics for a firewall rule using VSM User interface
• More flexible firewall rule format
• Object based
• Rule Direction
• Pre-NAT/Post-NAT inspection
• Rules based on source and destination interface
• Enhanced NAT rules with ability to add comments
• Multi Interface support
• AES-NI for improved VPN performance
YHMAN Ltd ®
18
vCNS MOVING FORWARD
• Completed vCNS Beta testing with VMware 3Q2012
• Re-ran beta tests on GA release of vCNS 5.1 3Q2012
• Starting internal testing of upgrade from vShield 5 to vCNS 5.1 4Q12
• Plan to deploy vCNS 5.1 4Q2012/1Q2013
• Utilise Edge HA for all tenants
• Make use of new SSL VPN for VM management
• Make use of new Load Balancer features
• HTTPS support
• TCP support for applications such as SMTP
• Deploy App firewall 2Q2013
YHMAN Ltd ®
19
vCD MOVING FORWARD
• Completed small virtual lab of vCloud Director 1.5.1 using
vCloud Director Network Isolation (vCDNI) 3Q2012
• Progress the Proof of Concept (POC) on vCloud Director
1.5.1 & 5.1 using real hardware 4Q2012
• If POC is on both vCloud Director 1.5.1 & 5.1 compare
vCDNI (MAC-in-MAC encapsulation) & VXLAN (MAC-in-IP)
4Q2012/1Q2013
YHMAN Ltd ®
20
VXLAN MOVING FORWARD
• Initially use VXLANs with external VLANs spanning our 3 DC’s
• External VLAN(s) handle North/South traffic from any of 3 DC’s
DC Network
Access Network
YHMAN Ltd ®
21
VXLAN MOVING FORWARD
VXLAN and ‘Pie in the sky’ thoughts
• Can VXLAN be used to eliminate the need for any VLAN’s spanning
our DC Interconnects between physical DC’s and support
Equal Cost Multipath [ECMP]?
• This is something YHMAN want to achieve and are keeping a close
eye on VMware and VXLAN.
Access Network
YHMAN Ltd ®
22
LIVE DEMO
YHMAN
vCNS Edge Demo
•Using similar virtual lab as used for beta
testing
•Create firewall & DNAT rule to publish SSH
service
•Access SSH service and show vShield
Manager User Interface and Edge CLI tools
to trace traffic through the Edge virtual
appliance
•Failover vCNS Edge showing HA
Tenant A
Legend :
192.168.0.1
Tenant A VM running SSH
Service- 192.168.0.2
(Inside_VM01)
192.168.142.100
Active
192.168.0.2
SSH
TCP22
Inside
vCNS Edge VMs
inside 192.168.0.1/24
outside 192.168.142.100/24
Standby
Outside
VMware vSphere + vCNS
YHMAN Ltd ®
Outside
External Access
Portgroup
Inside
Inside Portgroup
23
BEEN THERE, DONE THAT:
YHMAN Private Cloud Implementation & Lessons Learnt
Thank you - Q&A
Ed Carter - YHMAN Business Manager
Kevin Barrass - YHMAN Network Support & Development Officer
http://www.yhman.net.uk/projects/index.htm
YHMAN Ltd ®
24
FILL OUT
A SURVEY
AT WWW.VMWORLD.COM/MOBILE
COMPLETE THE SURVEY
WITHIN ONE HOUR AFTER
EACH SESSION AND YOU WILL
BE ENTERED INTO A DRAW
FOR A GIFT FROM THE
VMWARE COMPANY STORE
INF-SEC2031
Been There Done
That: YHMAN’s
Private Cloud
Implementation and
Lessons Learnt
Kevin Barrass
YHMAN Network Development
and Support
Jonathan Gohstand
VMware, Inc.
Ed Carter
YHMAN Business Manager
#vmworldinf
BEEN THERE, DONE THAT:
YHMAN Private Cloud Implementation & Lessons Learnt
Screen Dumps – Generic Firewall / NAT
http://www.yhman.net.uk/projects/index.htm
YHMAN Ltd ®
27
Select Edge to manage under the Datacenter>Network Virtualization>Edges
Create firewall rule to allow SSH from Laptop to Inside_VM01 based on Objects
YHMAN Ltd ®
28
Add DNAT and apply to outside interface
YHMAN Ltd ®
29
SSH to Inside VM01 and analyse traffic flow and perform
stateful failover
• View any current flow statistics for firewall rule using VSM interface
• Check User created DNAT rule for hits using Edge CLI
• View flow statistics for specified flow spec and flow matching
firewall rule using Edge CLI
• Verify flow table is being replicated to Standby Edge
• Debug traffic flow on “outside” interface
• Debug traffic flow on “inside” interface
• Show which Edge is active, standby
• Power off Active Edge
• Show SSH session is still active, also run ping to show
any lost packets
YHMAN Ltd ®
30
View any current flow statistics for firewall rule using VSM interface
Check User created DNAT rule for hits using Edge CLI
YHMAN Ltd ®
31
View flow statistics for specified flow spec TCP with Destination port of 22
View flow matching firewall rule using Edge CLI
YHMAN Ltd ®
32
Debug traffic flow on “outside” interface
Debug traffic flow on “inside” interface
YHMAN Ltd ®
33
Show which Edge is
active, standby
YHMAN Ltd ®
34
Failover Edge with SSH and ICMP session through active Edge
• New DNAT and Firewall rule created to allow ICMP Ping through Edge to Inside_VM01
• Show flows on active Edge. Flow = “192.168.142.1:1168--192.168.142.100:22”
• Show flows on standby edge. Flow = “192.168.142.1:1168--192.168.142.100:22”
• Power off Active Edge
• Show dropped pings
• Show active flows on now active Edge
YHMAN Ltd ®
35
Failover Edge with SSH and ICMP session through active Edge
• Standby Edge takes over, Failed Edge would be restarted by HA and become Standby Edge
• Active Edge has same active Flow = “192.168.142.1:1168--192.168.142.100:22”
• SSH session still active due to stateful failover
• Only dropped 4 pings
YHMAN Ltd ®
36
BEEN THERE, DONE THAT:
YHMAN Private Cloud Implementation & Lessons Learnt
Screen Dumps – SSL VPN
http://www.yhman.net.uk/projects/index.htm
YHMAN Ltd ®
37
Edge SSL VPN
• Configure Edge SSL VPN
• Ping and SSH over Edge SSL VPN with TCP Optimization Enabled
• Run Edge CLI commands to debug Edge SSL VPN
• SSH over Edge SSL VPN with TCP Optimization Disabled
• Show different flow characteristics and firewall requirements
when TCP Optimization is disabled.
YHMAN Ltd ®
38
Configure Server Settings
• Specify interface for Edge SSL VPN to bind to (192.168.142.100)
• Configure listening port (443)
• Configure Cipher (AES256-SHA)
• Select Server Certificate or use default Certificate (default)
YHMAN Ltd ®
39
Add IP Pool
• Configure IP Range and Gateway
• Add description
YHMAN Ltd ®
40
Add Private Network
• Configure Private Network Subnet
• Add description
• Enable TCP Optimization to prevent TCP over TCP meltdown
YHMAN Ltd ®
41
Add Authentication Server
• Configure Local Authentication
YHMAN Ltd ®
42
Add PHAT Installation package
• Add Windows “default” installation package
• Configure Edge Gateway for SSL VPN
YHMAN Ltd ®
43
Add Users to Local Authentication
• Add single test user
YHMAN Ltd ®
44
Enable Edge SSL VPN Service
• Go to Dashboard and click Enable button
YHMAN Ltd ®
45
Download and Install Full SSL VPN client PHAT
• Browse to SSL Service IP
• Log into Edge secure webpage
• Download and install full access client (PHAT Client)
YHMAN Ltd ®
46
Log into SSL VPN
• Run SSL VPN Client “VMwareTray Icon”
• Click “Login” then enter username and password
YHMAN Ltd ®
47
Debug Edge SSL VPN
• Create firewall rule to allow SSH and Ping to Inside_VM01
• SSH into Inside_VM01 and run constant ping to Inside_VM0
• Show Flow for SSH and Ping Sessions with TCP Optimization enabled
• Show Flow for SSH and Ping Sessions with TCP Optimization disabled
Rule-id
133127 is user
created rule
Rule-id
131074 is
Internal
generated
rule with Edge
as source
YHMAN Ltd ®
48
Debug Edge SSL VPN
• Show Flow for SSH and Ping Sessions with TCP Optimization disabled (not default)
Rule-id
133127 is user
created rule
Rule-id
131074 is
Internal
generated
rule with Edge
as source
YHMAN Ltd ®
49
View Edge SSL VPN Statistics
• View Edge SSL VPN stats from VSM User interface
YHMAN Ltd ®
50
View Edge SSL VPN Statistics
• View Edge SSL VPN stats from Edge CLI
YHMAN Ltd ®
51
BEEN THERE, DONE THAT:
YHMAN Private Cloud Implementation & Lessons Learnt
Thank you - Q&A
Ed Carter - YHMAN Business Manager
Kevin Barrass - YHMAN Network Support & Development Officer
http://www.yhman.net.uk/projects/index.htm
YHMAN Ltd ®
52