Cisco AVS Solution Guide
Cisco Application Virtual Switch
Solution Guide
Sep 3rd, 2014
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Cisco AVS Solution Guide
1
Purpose ............................................................................................................................................................ 4
1.1
2
AVS Introduction .......................................................................................................................................... 5
2.1
AVS for Application Centric Infrastructure ............................................................................... 5
2.2
Cisco ACI Fabric Overview ............................................................................................................... 5
2.2.1
3
2.2.2
3.1.2
OpFlex Protocol.............................................................................................................................................. 6
No Local Switching Mode ........................................................................................................................... 7
Local Switching .............................................................................................................................................. 8
Switch Failover and Link Aggregation.................................................................................................. 9
4.1
Port-Channel Technology ................................................................................................................ 9
4.1.1
LACP.................................................................................................................................................................... 9
4.1.3
Virtual Port Channel..................................................................................................................................... 9
4.1.2
4.1.4
5
End Point Groups (EPGs) Concept ......................................................................................................... 6
AVS Switching Modes .................................................................................................................................. 7
3.1.1
4
Pre-Requisite ....................................................................................................................................... 4
Standard Port Channel ................................................................................................................................ 9
Static Port-Channel ....................................................................................................................................... 9
4.2
MAC Pinning ...................................................................................................................................... 10
4.3
Virtual Port Channel (vPC) ........................................................................................................... 11
AVS Recommended Topologies ........................................................................................................... 12
5.1
Topology #1 AVS Host Directly Connected to Leaf .............................................................. 14
5.2
Topology#2 AVS Host Connected to Leaf via FEX................................................................. 15
5.3
Topology#3 AVS Host Connected to Leaf via UCS FI ............................................................ 17
5.4
Topology#4 AVS Host Connected to Leaf via Switch ........................................................... 18
5.4.1
5.4.2
5.5
Double-Sided VPC with Nexus 5000 and AVS with MAC Pinning ........................................... 18
Double-Sided VPC with Nexus 5000 and AVS with VPC ............................................................. 19
Topology#5 AVS Host Connected to Leaf via Switch-FEX .................................................. 20
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 2
Cisco AVS Solution Guide
5.6
Topology#6 AVS Host Connected to Leaf via Multiple Switches ..................................... 21
5.7
Topology#7 AVS Host Connected to Leaf via UCS FI and Switch ..................................... 22
5.7.1
Single-Side VPC with Nexus 5000/UCS FI and AVS with MAC Pinning ................................ 22
6
AVS Implementation Best Practices ................................................................................................... 23
7
FAQ ................................................................................................................................................................. 24
7.1
8
Support Table ................................................................................................................................... 24
References ................................................................................................................................................... 25
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 3
Cisco AVS Solution Guide
1 Purpose
This document is intended as a solution-level reference for technical professionals responsible
for preparing, planning, and implementing the Cisco Application Virtual Switch (AVS) for Data
Center customer.
This document provides AVS planning considerations and topology recommendations, but does
not discuss all the foundational technologies, procedures and best practices for deploying the
routing, switching and data center setup required by the solution. Instead, it refers to detailed
documents that discuss those technologies and implementation methods, while focusing on
specific configuration and topologies for deploying AVS within the ACI (Application Centric
Infrastructure) Solution.
1.1 Pre-Requisite
This document assumes that readers have thorough understanding of the Cisco ACI (Application Centric
Infrastructure) and other Cisco Data Center technologies. Please refer to following links to understand
these concepts.
http://www.cisco.com/go/aci
http://www.cisco.com/go/datacenter
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 4
Cisco AVS Solution Guide
2 AVS Introduction
Cisco Application Virtual Switch (AVS) is a hypervisor-resident distributed virtual switch that is
specifically designed for the Cisco Application Centric Infrastructure (ACI) and managed by Cisco
APIC (Application Policy Infrastructure Controller).
2.1 AVS for Application Centric Infrastructure
The Cisco AVS is integrated with the Cisco Application Centric Infrastructure (ACI). Unlike
Nexus1000V where management is done by a dedicated Virtual Supervisor Module, Cisco AVS is
managed by the Cisco APIC. Cisco AVS implements the OpFlex protocol for control plane
communication.
Before we dive into the specifics of AVS, it is important to understand the basic concepts about
Cisco Application Centric Infrastructure fabric.
Note:
Cisco AVS is the Cisco vSwitch for ACI mode. If you are running Nexus 9000 in standalone mode,
then you can use Nexus 1000V as vSwitch but otherwise you will use Cisco AVS.
2.2 Cisco ACI Fabric Overview
The Cisco Application Centric Infrastructure Fabric (ACI) fabric includes Cisco Nexus 9000 Series
switches with the APIC (Application Policy Infrastructure Controller) to run in the leaf/spine ACI
fabric mode. In a recommended minimum configuration
•
•
•
Three Cisco Nexus 9K (9500 series or 9336PQ) switches deployed as spines.
Only Cisco Nexus 9K switches (9300 Series) can connect to the spine switches as leaf
switches or nodes (All other devices, appliances and switches connect to the leaf nodes)
The APIC is an appliance running on Cisco UCS server and connects to one of the leaf
nodes. It manages the ACI fabric. The recommended minimum configuration for the
APIC is a cluster of three replicated hosts. One can connect APIC to two different leaf
switches for redundancy
The APIC fabric management functions do not operate in the data path of the fabric. The
management is done via out of the band management network.
The following figure illustrates an overview of the leaf/spine ACI fabric.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 5
Cisco AVS Solution Guide
The ACI fabric provides low-latency forwarding across high-bandwidth links (40 Gbps, with a
100-Gbps future capability). Traffic with the source and destination on the same leaf
switch/node is handled locally. All other traffic traveling from the ingress leaf to the egress leaf
goes through a spine switch. Although this architecture appears as two hops from a physical
perspective, it is actually a single Layer 3 hop because the fabric operates as a single Layer 3
switch.
2.2.1
End Point Groups (EPGs) Concept
The central concept is to group endpoints (EPs) with identical semantics into endpoint groups (EPGs)
and then write policies that regulate how such groups can interact with each other.
2.2.2
OpFlex Protocol
OpFlex, the southbound API, is an open and extensible policy protocol used to transfer abstract policy in
XML or JavaScript Object Notation (JSON) between Cisco APIC and AVS switch.
http://tools.ietf.org/html/draft-smith-opflex-00
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 6
Cisco AVS Solution Guide
3 AVS Switching Modes
Cisco AVS supports two modes of traffic forwarding:
1- No Local Switching Mode
2- Local Switch Mode
The forwarding mode is selected during Cisco AVS installation when the VMware vCenter
Domain is created. VMware vCenter Domain creation is the step where APIC will communicate
with vCenter and will dynamically create Cisco AVS. Following picture shows the different
options on APIC controller
3.1.1
No Local Switching Mode
“No Local Switching” mode was formerly known as FEX enable mode. In “No Local Switching” mode, all
traffic (intra-EPG and/or inter-EPG) is forwarded by the physical leaf. In this mode, VXLAN is the only
allowed encapsulation type.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 7
Cisco AVS Solution Guide
3.1.2
Local Switching
“Local Switching” was formerly known as FEX Disable mode. In this mode all intra-EPG traffic is
locally forwarded by the Cisco AVS, without the involvement of the physical leaf, if the traffic
bound for the same host. All inter-EPG traffic is forwarded via the physical leaf.
In this mode, the Cisco AVS can use either use VLAN or VXLAN encapsulation for forwarding
traffic to the leaf and back. The encapsulation type is selected during Cisco AVS installation.
•
•
If VLAN encapsulation mode is used, a range of VLANs must be available for use by the
Cisco AVS. These VLANs have local scope in that they have significance only within the
Layer 2 network between the Cisco AVS and the leaf.
If VXLAN encapsulation mode is used, only the infra-VLAN needs to be available
between the Cisco AVS and the VXLAN. This results in a simplified configuration and is
the recommended encapsulation mode if there are one or more switches between the
Cisco AVS and the leaf.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 8
Cisco AVS Solution Guide
4 Switch Failover and Link Aggregation
Network architects can use different approaches for protection against switch or link failover and link
aggregation. The most common design approaches with Cisco AVS are virtual PortChannel (vPC) and
MAC pinning. Both design approaches provide protection against single-link and physical-switch failures,
but they differ in the way that the virtual and physical switches are coupled and the way that the
VMware ESX or ESXi server traffic is distributed over the 10 Gigabit Ethernet links. The essence of all
these approaches is Port-Channel technology.
4.1 Port-Channel Technology
A Port-Channel (also referred to as Ether-Channel) on the Cisco AVS implements the standards-based
IEEE 802.3ad or 802.1AX link aggregation protocol that incorporates the Link Aggregation Control
Protocol (LACP) for automatic negotiation.
4.1.1
LACP
LACP dynamically bundle several physical ports together to form a single port channel. LACP enables a
node to negotiate an automatic bundling of links by sending LACP packets to the peer node. LACP is
simply a way to dynamically build Port-Channel. Essentially, the “active” end of the LACP group sends
out special frames advertising the ability and desire to form a Port-Channel.
4.1.2
Standard Port Channel
Standard Port-Channel requires that all uplinks from one ESXi host in the Port-Channel group must be
connected to single and same upstream physical switch.
4.1.3
Virtual Port Channel
When ESXi host uplinks are spread across more than one upstream physical switch, the upstream
switches are clustered using Virtual Port-Channel (vPC).
4.1.4
Static Port-Channel
When LACP protocol is not running on the links, it is called Static Port-Channel mode. In Cisco OS and
Cisco Nexus OS, Static Port-Channel mode is also called “Mode ON”. Following command displays the
Static Port-Channel configuration on Nexus 5000 switch.
interface ethernet 1/1-2
channel-group 1 mode on
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 9
Cisco AVS Solution Guide
On the contrary, in APIC controller, LACP mode “Off” represents Static Port-Channel configuration.
A maximum of 16 links can be configured to form a Port-Channel group.
You can configure one of several types of port channel policies on the Cisco AVS: Link Aggregation
Control Policy (LACP) in active or passive mode, MAC pinning, or static. You can configure port channel
policies through the Cisco APIC GUI, the REST API, or the CLI.
MAC Pinning—MAC Pinning
Active—LACP active
Passive—LACP passive
Off— LACP Off (i.e Static Port-Channel)
4.2 MAC Pinning
In a MAC Pinning mode, the Gigabit Ethernet uplinks from the Cisco AVS are treated as stand-alone links.
In a two Gigabit Ethernet uplinks scenario, each Gigabit Ethernet interface is connected to a separate
physical switch with Layer 2 continuity on all IEEE 802.1Q trunked VLANs between the two switches.
Virtual Ethernet ports supporting virtual machines and vmkernel ports are allocated in a round-robin
fashion over the available Gigabit Ethernet uplinks. Each MAC address is pinned to one of the uplinks
until a failover event occurs. MAC pinning does not rely on any protocol to distinguish the different
upstream switches, making the deployment independent of any hardware or design. This independence
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 10
Cisco AVS Solution Guide
enables consistent and easy deployment of the Cisco AVS, and it is the preferred method for deploying
the Cisco AVS when the upstream switches cannot be clustered using Cisco vPC.
4.3 Virtual Port Channel (vPC)
vPC is required on the upstream physical switches to enable the Port-Channel to span both upstream
physical switches and still maintain availability for the VMware ESXi host should one switch fail or lose
connectivity. This vPC clustering is transparent to the Cisco AVS. From AVS Host point of view, it sees the
vPC cluster as one single switch.
For vPC to work, the Cisco Application Virtual Switch should be configured with LACP Port-Channel
(configuration is done via APIC) with the two Gigabit Ethernet uplinks defined by one port profile.
The two upstream physical switches should be configured with vPC. The upstream switch (for example a
Cisco Nexus 5000 or 7000 series switch) will appear as a single logical switch distributed over two
physical chassis.
Differences Between vPC and MAC Pinning
Design
Uplinks
Physical-Switch Requirements
vPC
Single logical PortChannel
Clustered physical switches using a multichassis Ether-Channel
(MEC) implementation such as Cisco vPC, virtual switching system
(VSS), or virtual blade switch (VBS) technologies
MAC
Pinning
All teamed uplinks in
same Layer 2 domain
No special configuration other than Layer 2 continuity between
both switches on all VLANs trunked to the VMware ESX or ESXi
server
vPC is the recommended approach when vPC or clustered physical switches are available at the physical
access layer. MAC pinning should be chosen when these options are not available.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 11
Cisco AVS Solution Guide
5 AVS Recommended Topologies
On a very high level there are seven commonly deployed topologies supported by AVS. For ease
of understanding these are divided into two groups
1- Standard Topologies
• Topology#1 AVS host directly connected to N9K leaf switch
• Topology#2 AVS host connected to N9K leaf switch via FEX
• Topology#3 AVS host connected to N9K leaf switch via UCS FI
Leaf Switch
Leaf Switch
1
Leaf Switch
FEX
2
UCS FI
3
2- Extended Topologies
• Topology#4 AVS host connected to N9K leaf via a single physical switch
i. Double-Sided VPC with Nexus 5000 and AVS with MAC Pinning
ii. Double-Sided VPC with Nexus 5000 and AVS with VPC
• Topology#5 AVS host connected to N9K leaf via a switch-FEX
• Topology#6 AVS host connected to N9K leaf via multiple switches
• Topology#7 AVS host connected to N9K leaf via UCS FI and a Switch
i. Single-Side VPC with Nexus 5000/UCS FI and AVS with MAC Pinning
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 12
Cisco AVS Solution Guide
Leaf Switch
Leaf Switch
Nexus 5K/6K/7K
Nexus 5K/6K/7K
Nexus 2000
4
Leaf Switch
Nexus 5K/6K/7K
Nexus 5K/6K/7K
5
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
6
Leaf Switch
Nexus 5K/6K/7k
UCS FI
7
Page 13
Cisco AVS Solution Guide
5.1 Topology #1 AVS Host Directly Connected to Leaf
In this topology ESXi host is directly connected to ACI Leaf Switch. This is typical scenario where
a rack mount server (For example a Cisco UCS C-Series Server) is running ESXi hypervisor and
AVS is running as a distributed virtual switch on it. It is recommended to use VLAN in this
topology.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 14
Cisco AVS Solution Guide
5.2 Topology#2 AVS Host Connected to Leaf via FEX
In this topology ESXi host (For example a rack mount server like Cisco UCS C-Series Server) is connected
to the FEX. FEX is then directly connected to APIC Leaf switch. VLAN local switching mode, VXLAN local
switching mode, and VxLAN non-switching mode are supported with this topology. It is recommended
to use VLAN with this topology.
There are some limitations with this topology that you should be aware of
•
•
vPC is not supported between FEX and Leaf or for hosts directly connected to FEX
Only a single physical link is supported between the FEX and an ESX host connected to that FEX.
This means that LACP or MAC Pinning is not supported for ESXi hosts connected directly to an
FEX that is connected directly to a leaf.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 15
Cisco AVS Solution Guide
•
For the single host connected between FEX and an ESXI host, when you choose an LACP mode
for a Cisco AVS, you should choose either MAC Pinning or Off on the APIC controller as shown in
the following diagram
•
These limitations does not apply to an extender that is connected to a Nexus 5000 or 7000
switch that is connected to a leaf (as shown in Topology#5)
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 16
Cisco AVS Solution Guide
5.3 Topology#3 AVS Host Connected to Leaf via UCS FI
In topology#3, ESXi host is running on a Cisco UCS B-Series blade server. The B-Series server or
the chassis is connected to UCS Fabric Interconnect. FI is then directly connected to ACI Leaf
switch. This topology connects the ESX hypervisor to the Cisco APIC using via fabric
interconnect, VPCs, LACP, and MAC pinning. In topology#3 one can use either VLAN or VXLAN.
The main concept is to use VXLAN when there is more than one hop between ESXi host and Leaf
Switch. Following picture shows the logical diagram of topology#3.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 17
Cisco AVS Solution Guide
5.4 Topology#4 AVS Host Connected to Leaf via Switch
This topology is a very common use case in scenario where for example customer has already deployed
Cisco Nexus 5000, 6000 or 7000 and they want to have Cisco ACI fabric inserted in their current
architecture. This topology connects the ESXi hypervisor to a Cisco APIC through the Cisco Nexus 5000
switch, virtual port channels, and MAC pinning. VXLAN will be used in this topology.
5.4.1
Double-Sided VPC with Nexus 5000 and AVS with MAC Pinning
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 18
Cisco AVS Solution Guide
5.4.2
Double-Sided VPC with Nexus 5000 and AVS with VPC
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 19
Cisco AVS Solution Guide
5.5 Topology#5 AVS Host Connected to Leaf via Switch-FEX
This topology is not very different than topology#4. The only difference is that the L2 switch in the
middle of AVS and Leaf switch has a Cisco Nexus 2000 Fabric Extender (FEX). And the ESXi host or AVS is
connected to FEX. This is the most common scenario where FEX is deployed as a Top of the Rack (TOR)
switch. VXLAN will be used in this topology. Topology#4 and 5 works almost the same in terms of
multicast.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 20
Cisco AVS Solution Guide
5.6 Topology#6 AVS Host Connected to Leaf via Multiple Switches
This topology represents a customer running data center with core and aggregation architecture with
Cisco Nexus 5000 or 7000 series switches. The customer wants to migrate to ACI based architecture in
phases. The leaf switch could be connected to Nexus 5000 or 7000 switch at the aggregation layer.
VXLAN is recommended in this topology.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 21
Cisco AVS Solution Guide
5.7 Topology#7 AVS Host Connected to Leaf via UCS FI and Switch
It is highly recommended to use VXLAN here because there are more than one hop between
AVS and Leaf switch.
5.7.1
Single-Side VPC with Nexus 5000/UCS FI and AVS with MAC Pinning
This topology connects the AVS ESXi host to the leaf switches using MAC pinning, directly or via
Cisco Nexus 5000 switches and Cisco UCS 62xx Series Fabric Interconnects.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 22
Cisco AVS Solution Guide
6 AVS Implementation Best Practices
Following configuration are recommended when deploying Cisco AVS with the Cisco APIC
•
•
•
•
•
•
•
Infra VLAN must be configured in the layer 2 network to establish connection between N9K leaf
and AVS for OpeFlex.
Infra VLAN must be configured on the leaf side port and AVS side ports of Cisco Fabric
Interconnect
DHCP relay policy must be configured for AVS so that the APIC can assign the IP address for the
AVS vtep vmk in ESXi host
Cisco Fabric Interconnect doesn’t support LACP on their southbound ports so it is recommended
not to configure AVS with the LACP policy
Configure vMotion on a separate VMKernel NIC with a dedicated EPG. Do not configure vMotion
on the VMKernel NIC created for OpFlex channel
One must not delete or change any parameters for the VMkernel NIC created for the OpFlex
channel
If VMkernel NIC created for the OpFlex channel is deleted by mistake, recreate it with the attach
port-group vtep, and configure it with a dynamic IP address. One should never configure a static
IP address for an OpFlex vmk NIC.
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Page 23
Cisco AVS Solution Guide
7 FAQ
7.1 Support Table
UCS port channel configuration is statically set to Link Aggregation Control Protocol (LACP) mode active.
This configuration cannot be modified; therefore, all upstream port-channel configurations must adhere
to LACP mode active as well. Alternatively, you can configure the upstream switch ports for LACP mode
passive.
Following table list different switches and their supported port-channeling modes
LACP
MAC Pinning
AVS
Yes
Yes
UCS Fabric
Interconnect
Yes (Northbound Ports)
Yes
Nexus 9000 Leaf
Nexus 5000
Nexus 7000
Yes
No (Southbound Ports)
Yes
Yes
Yes
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
Static Port
Channel
VPC
Yes
Yes
Yes
Not Applicable
No
No
Yes
Yes
Page 24
Cisco AVS Solution Guide
8 References
Please refer to following supporting documents for more detailed information.
Cisco Application Virtual Switch
http://www.cisco.com/c/en/us/products/switches/application-virtual-switch/index.html
Cisco Application Centric Infrastructure Fundamentals
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/acifundamentals/b_ACI-Fundamentals.html
Printed in USA
© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.
09/2014
Page 25