White Paper
Cisco Application Virtual Switch and VMware
vSphere Distributed Switch Failover Convergence
in Cisco Application Centric Infrastructure:
A Cisco IT Case Study
Cisco’s Business Need for Migration
Cisco® IT is a large global IT organization with multiple data centers distributed throughout the world. The
infrastructure for each data center is huge: for example, Cisco’s Allen, Texas, data center includes more than 5000
applications, 8000 virtual machines, and 1700 Cisco Unified Computing System™ (Cisco UCS®) blades. As Cisco’s
data centers grow, quick and agile application deployment becomes increasingly challenging. Cisco IT realized the
solution to this challenge is to make the infrastructure application aware. So Cisco turned to Cisco Application
Centric Infrastructure (Cisco ACI™).
Cisco ACI enables Cisco IT to use a common application-aware policy-based operating model across their physical
and virtual environments. This approach simplifies and accelerates the entire application deployment lifecycle.
Cisco IT hence is actively investigating mapping existing network elements and applications to the Cisco ACI
model and migrating its infrastructure entirely to Cisco ACI.
Cisco ACI Migration Options
The Cisco Application Policy Infrastructure Controller (APIC) integrates with a VMware-based virtual environment
by creating a distributed virtual switch mapped to the Cisco ACI environment in VMware vSphere vCenter. Cisco
ACI customers need to choose the distributed virtual switch that they want to use with Cisco ACI. Two options are
available: Cisco Application Virtual Switch (AVS) and VMware vSphere Distributed Switch (VDS).
AVS is a purpose-built virtual network-edge switch for Cisco ACI. It is based on the highly successful Cisco Nexus®
1000V Switch platform, which is the industry’s first and leading multihypervisor virtual switch. The Cisco Nexus
1000V has more than 10,000 customers and has been deployed by several large-scale service providers and large
enterprises, some with more than 20,000 virtual machines in production. AVS brings the Cisco ACI policy model to
virtual infrastructure, thus providing policy consistency across physical and virtual workloads. AVS tightly integrates
into Cisco ACI under APIC control, bringing networking features widely used in the physical world to the hypervisor
environment with better performance, greater scalability, and faster convergence than VDS.
Cisco IT tested both Cisco AVS and VMware VDS and choose Cisco AVS for its simplified management, greater
scalability, better performance, flexible deployment options, and faster convergence. Although AVS provides a
number of advantages, such as flexible deployment options using VLAN and Virtual Extensible LAN (VXLAN)
modes, detailed policy control for virtual workloads using microsegmentation, and greater security using the
stateful distributed firewall, this case study focuses its advantages for failover convergence.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 1 of 8
Cisco IT Requirements
AVS can use either VLAN or VXLAN encapsulation to forward traffic between the leaf switch and the VMware ESXi
host. Connectivity to the Cisco ACI leaf through VDS is only VLAN-based in the absence of VMware vShield
Manager. If VLAN encapsulation is used, each endpoint group (EPG) maps to a port group in the distributed virtual
switch and receives a user-friendly name and a VLAN ID. The Cisco ACI fabric is configured to translate VLAN
tags at the VMware port-group level into EPGs for policy application. If VXLAN encapsulation is used, AVS acts as
a VXLAN tunnel endpoint (VTEP) on the hypervisor, providing the capability to perform VXLAN tunneling over the
infrastructure VLAN between the leaf switch and the host. In VXLAN mode, only the infrastructure VLAN needs to
be defined on the network infrastructure (if any) between the host and the leaf, resulting in simpler configuration.
This approach is particularly useful in environments with blade servers in which blade switches lie between blade
servers and leaf switches. AVS in VXLAN mode thus offered Cisco IT more flexibility in deployment compared to
AVS in VLAN mode.
Although deployment flexibility is an important consideration, Cisco IT also has very strict criteria for failover
convergence in its infrastructure. For instance, network, computing, and storage IP downtime in various network
failover scenarios (using high availability) must be less than 5 seconds now, and less than 2 seconds within a year.
Hence, Cisco IT evaluated the two distributed virtual switch options, Cisco AVS in VXLAN mode and VMware VDS,
to see which would offer the best failover time compliant with requirements.
Test Topology
Cisco IT used the Cisco ACI computing and storage topologies shown in Figure 1 to measure failover times. Four
spine switches and six leaf switches were used in the Cisco ACI fabric. Three Cisco UCS domains were used, with
three tenants spread across them. Each Cisco UCS domain had two fabric interconnects. VDS and AVS both had
200 virtual machines at the network edge (DMZ) and 800 virtual machines in internal Virtual Routing and
Forwarding (VRFs) instances. Storage for the virtual machines was on separate network-attached storage (NAS)
servers and not on the Cisco UCS blades themselves. The topology was made highly available through port
channels and virtual port channels (vPCs) wherever applicable to facilitate measurement of failover time in various
scenarios.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 2 of 8
Figure 1.
Test Topology
Test Methodology
The main objective of Cisco IT was to measure how long the network took to converge through high availability
when certain links in the Cisco ACI fabric were shut down and then brought up again. Continuous ping tests were
run on all the applicable virtual machines and logged during each scenario to accurately measure the packets lost
during each failover scenario. Packet loss was measured during both link shutdown and link startup. Secure Copy
(SCP) tests on the virtual machines were conducted in parallel to mimic storage read-and-write operations during
each failover scenario. Testing occurred separately for AVS and VDS hosted virtual machines. The following
failover scenarios were tested to measure the convergence.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 3 of 8
●
Failure of a single port of a port channel to the fabric interconnect
●
Failure of a single vPC to the fabric interconnect
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 4 of 8
●
Failure of vPC to the computing switch (fabric interconnect)
●
Failure of a single switch to the computing switch (fabric interconnect): one on each vPC leaf pair, in pods 1
through 3 separately
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 5 of 8
●
Failure of a single Cisco UCS fabric interconnect (A or B): one on each Cisco UCS instance, in pods 1
through 3 separately
●
Failure of a single border leaf
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 6 of 8
●
Failure of a single spine switch
Test Results
Table 1 shows the test results. A test case was marked pass only if it resulted in virtual machine downtime of less
than 2 seconds. These results clearly show the superiority of AVS compared to VDS in the following failover
scenarios:
●
Failure of vPC to computing switch (fabric interconnect)
●
Failure of a single Cisco UCS fabric interconnect (A or B): one on each Cisco UCS instance, in pods 1
through 3 separately
Table 1.
Test Results
Test Case
Result (AVS)
Result (VDS)
Fail single port of a port-channel to fabric interconnect (FI)
PASS
PASS
Fail single port-channel of VPC to FI
PASS
PASS
Fail VPC to Compute (FI)
PASS
FAIL (> 60 seconds)
Fail single switch to Compute (FI) - one on each VPC leaf pair, pods 1-3 separately
PASS
PASS
Fail single UCS FI (A or B) - one on each UCS, pods 1-3 separately
PASS
FAIL (> 60 seconds)
Fail single border leaf
PASS
PASS
Fail single spine switch
PASS
PASS
Cisco AVS Benefits
A number of multicast, Internet Group Management Protocol (IGMP), and Address Resolution Protocol (ARP)
enhancements in AVS, particularly during failover scenarios, substantially reduced virtual machine downtime.
Some of these enhancements are listed here:
●
AVS proactively sends IGMP join messages during failover rather than just replying to IGMP query
messages, improving convergence time.
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 7 of 8
●
AVS sends a gratuitous ARP (GARP) message during VXLAN load balancing failover, accelerating IP
learning and decreasing convergence time.
●
AVS uses IGMPv3 instead of IGMPv2 for greater efficiency and reduced demands on the network.
●
AVS allows 30 seconds for stabilization before adding links to an uplink port channel to help ensure reliable
behavior during failover recovery.
These enhancements helped ensure that Cisco IT not only met the current downtime criterion of 5 seconds but
also the future one of 2 seconds. Cisco IT decided to proceed with AVS in VXLAN mode instead of VDS in VLAN
mode. Cisco IT has completed migration of its Research Triangle Park (RTP), North Carolina, data center to AVS
and is on track with migrating others.
Conclusion
The testing discussed here of the two virtual switches clearly shows that Cisco AVS offers better failover
convergence than VMware VDS in at least two scenarios. AVS also offers the advantages of policy consistency
across physical and virtual workloads, tighter integration into Cisco ACI, incorporation of networking features widely
used in the physical world into the hypervisor environment, flexible deployment options using VLAN and VXLAN,
detailed policy control of virtual workloads using microsegmentation, greater security using the distributed firewall,
better performance, and greater scalability. AVS thus is an excellent choice for Cisco ACI customers as the virtual
switch for a Cisco ACI deployment.
For More Information
http://www.cisco.com/c/en/us/products/switches/application-virtual-switch/index.html
Printed in USA
© 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
C11-736554-00
01/16
Page 8 of 8