Design Guide
Data Center Routed Access
Design with Cisco Nexus 5500
Platform and 2000 Series Fabric
Extenders
(Updated to Cisco NX-OS Software Release
5.0(3)N1(1))
Design Guide
June, 2011
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 1 of 46
Contents
Introduction .............................................................................................................................................................. 4
Preliminary Design Considerations ....................................................................................................................... 5
Layer 3 Capabilities of the Cisco Nexus 5500 Platform ........................................................................................ 5
Topology Choice ................................................................................................................................................... 6
Layer 3 Access Layer ....................................................................................................................................... 7
Layer 3 and vPC Access Layer ........................................................................................................................ 8
Scalability Factors ................................................................................................................................................. 9
Layer 2 Best Practices for Layer 3 Access ............................................................................................................ 9
Layer 3 Access Design .......................................................................................................................................... 10
Routing at the Access Layer ............................................................................................................................... 10
Autostate ........................................................................................................................................................ 10
Layer 3 Peering at the Access Layer.............................................................................................................. 11
HSRP and VRRP ........................................................................................................................................... 11
Aggressive Timers.......................................................................................................................................... 12
Connectivity Between Aggregation and Access Layers ...................................................................................... 13
ECMP ............................................................................................................................................................. 13
Subnets for Layer 3 Links ............................................................................................................................... 14
Link Cost Tuning ............................................................................................................................................ 14
Reducing Polarization .................................................................................................................................... 15
Summarization ............................................................................................................................................... 16
Design with OSPF ............................................................................................................................................... 17
OSPF Timers.................................................................................................................................................. 18
Design with EIGRP ............................................................................................................................................. 19
Multicast Considerations ...................................................................................................................................... 20
Layer 3 Access Design with vPC .......................................................................................................................... 22
Preliminary considerations .................................................................................................................................. 22
Cisco Nexus 5000 Series vPC Baseline Configuration ....................................................................................... 22
Peer Link ........................................................................................................................................................ 23
Behavior of vPC with Routing ............................................................................................................................. 24
Interaction of vPC with Gateway Redundancy Protocols ............................................................................... 25
Peer Gateway................................................................................................................................................. 25
Interactions of vPC with SVIs ......................................................................................................................... 26
What Is a Layer 3 vPC? ................................................................................................................................. 27
ECMP Between Aggregation and Access Layers........................................................................................... 28
Establishing IGP Peering over the Peer Link ................................................................................................. 29
Multicast .............................................................................................................................................................. 30
PIM Dual-DR Implementation for Directly Connected Sources and Receivers .............................................. 31
PIM-DR and Source Registration ................................................................................................................... 31
PIM Prebuilt SPT ............................................................................................................................................ 31
Sizing the Peer Link for Multicast Traffic ........................................................................................................ 32
Example Configurations........................................................................................................................................ 32
Layer 3 Access Design Without vPC .................................................................................................................. 33
Cisco Nexus 5000 Series Switch 1 (N5k1) ..................................................................................................... 33
Cisco Nexus 5000 Series Switch 2 (N5k2) ..................................................................................................... 35
N5k1 with Multicast ........................................................................................................................................ 37
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 2 of 46
N5k2 with Multicast ........................................................................................................................................ 39
Layer 3 Access Design with vPC ........................................................................................................................ 41
N5k1 ............................................................................................................................................................... 41
N5k2 ............................................................................................................................................................... 43
For More Information ............................................................................................................................................. 46
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 3 of 46
Introduction
This chapter includes design recommendations for the use of the Cisco Nexus® 5500 switch platform as a routed
access layer.
At the time of this writing, the Cisco Nexus 5000 Series Switches include the following products:
●
Cisco Nexus 5548P and 5548UPSwitches: This one-rack-unit (1RU) 10 Gigabit Ethernet and Fibre Channel
over Ethernet (FCoE) switch offers up to 960-Gbps throughput. It has up to 48 ports: 32 fixed 1/10-Gbps
Enhanced Small Form- Factor Pluggable (SFP+) Ethernet and FCoE ports and one expansion slot.
Expansion modules include 16 ports of 10-Gbps SFP+ Ethernet and FCoE, and 8 ports of 10-Gbps SFP+
Ethernet and FCoE plus 8 ports of 1/2/4/8-Gbps native Fibre Channel. The Cisco Nexus 5548 switches can
be configured with a Layer 3 daughter card to provide Layer 3 switching functions.
●
Cisco Nexus 5596UP Switch: This line-rate switch offers 96 ports of 10 Gigabit Ethernet and FCoE. It has
48 fixed 1/10-Gbps SFP+ Ethernet and FCoE ports and three expansion slots. Expansion modules include
16 ports of 10-Gbps SFP+ Ethernet and FCoE, 8 ports of 10-Gbps SFP+ Ethernet and FCoE plus 8 ports of
1/2/4/8-Gbps native Fibre Channel, and a Layer 3 module.
●
Cisco Nexus 5020 Switch: This 2RU 10 Gigabit Ethernet and FCoE switch offers throughput of 1.04 terabits
per second (Tbps). It has up to 56 ports, with 40 fixed 10 Gigabit Ethernet ports with SFP+ connectors and
two expansion slots.
●
Cisco Nexus 5010 Switch: This 1RU 10 Gigabit Ethernet and FCoE switch offers up to 28 ports: 20 fixed 10
Gigabit Ethernet ports with SFP+ connectors and one expansion slot.
The design best practices provided in this document apply to designs that use the Cisco Nexus 5548 and 5596
products.
At the time of this writing, the Cisco Nexus 2000 Series Fabric Extenders includes the following products:
●
Cisco Nexus 2148T Fabric Extender: This product has 48 1-Gbps RJ-45 copper ports and four 10 Gigabit
Ethernet SFP+ uplink ports.
●
Cisco Nexus 2224TP GE Fabric Extender: This product provides 24 Fast Ethernet and Gigabit Ethernet
(100/1000BASE-T) server ports and two 10 Gigabit Ethernet uplink ports in a compact 1RU form factor.
●
Cisco Nexus 2248TP GE Fabric Extender: This product has 48 100-Mbps and 1-Gbps copper ports and
four 10 Gigabit Ethernet SFP+ uplink ports (requires Cisco® NX-OS Software Release 4.2(1)N1(1) or later).
●
Cisco Nexus 2232PP 10GE Fabric Extender: This product has 1/10-Gbps SFP and SFP+ Ethernet ports
and eight 10 Gigabit Ethernet SFP+ uplink ports (requires Cisco NX-OS Release 4.2(1)N1(1) or later). The
Cisco Nexus 2232PP is also suitable for carrying FCoE traffic. Servers can attach to the Cisco Nexus
2232PP with Twinax cables or optical connectivity in the same way as to a Cisco Nexus 5000 Series
Switch.
Note:
All ports of the Cisco Nexus 5500 switch platform have the capability to operate as 1 or 10 Gigabit
Ethernet ports. With Cisco NX-OS 5.0(3)N1(1) and later, the Cisco Nexus 5500 platform ports can be configured
for 1 and 10 Gigabit Ethernet speeds.
The exact number of fabric extenders that can be connected to a single Cisco Nexus 5000 Series Switch is
hardware and software dependent. For the latest information, please check the Cisco Documentation page.
With Cisco NX-OS 5.0(2)N1(1), the Cisco Nexus 5000 Series supports 12 fabric extenders, with the Cisco Nexus
5500 platform supporting up to 16 fabric extenders.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 4 of 46
With Cisco NX-OS 5.0(3)N1(1) and later, the Cisco Nexus 5500 platform supports up to 24 fabric extenders when
exclusively the Layer 2 functions are used, and 8 fabric extenders when the Cisco Nexus 5500 platform is used in
Layer 3 mode.
This chapter assumes that the reader is familiar with the way that virtual PortChannel (vPC) technology works. For
additional information, refer to the document “Cisco NX-OS Software Virtual PortChannel Fundamental Concepts”
at http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/design_guide_c07-625857.pdf for a more
in-depth discussion of vPC design choices, and to “Spanning-Tree Design Guidelines for Cisco NX-OS Software
and Virtual PortChannels” at http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-57283400_STDG_NX-OS_vPC_DG.pdf for more information about spanning tree.
This chapter does not explain the Layer 2 design best practices. These are covered in the document “Data Center
Access Design with Cisco Nexus 5000 Platform and 2000 Series Fabric Extenders and Virtual PortChannels.” at
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-57282901_Design_N5K_N2K_vPC_DG.pdf.
This document is an extension of the document “Data Center Design IP Network Infrastructure,” at
http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DC_3_0/DC-3_0_IPInfra.html.
This document also complements the document “Data Center Secure Multi-Tenancy,” at
http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/VMDC/vmdcDdg11.pdf.
Preliminary Design Considerations
This document divides the design and configuration of the Layer 3 access layer into these categories:
●
Classic Layer 3 access design with spanning tree running on the access switches and Layer 3 Interior
Gateway Protocol (IGP) running between the access and aggregation layers: The design consists of pairs
of Cisco Nexus 5500 switches, which provides redundant Layer 2 connectivity to support network interface
card (NIC) teaming, virtual machine mobility within a pod (intra-pod mobility), and high-availability clustering.
●
Deployment with vPC on the Cisco Nexus 5500 platform to support vPC-connected servers (that is, servers
connected with the PortChannel NIC teaming option) and dual-connected fabric extenders: The Cisco
Nexus 5500 switches are deployed in pairs and connected to the aggregation layer with Layer 3 links.
Layer 3 Capabilities of the Cisco Nexus 5500 Platform
The Layer 3 engine card on the Cisco Nexus 5500 platform provides the following functions:
●
Layer 3 interfaces: Routed ports on Cisco Nexus 5500 platform ports, switch virtual interfaces (SVIs) that
can route traffic coming from Cisco Nexus 5500 platform ports or from fabric extender ports; PortChannels
can be configured as Layer 3 interfaces as well.
●
Routing protocol support: Static, Routing Information Protocol Versions 1 and 2 (RIPv1 and v2), Open
Shortest Path First Version 2 (OSPFv2), Enhanced Interior Gateway Routing Protocol (EIGRP), and Border
Gateway Protocol (BGP).
●
Support for gateway redundancy protocols: Hot Standby Router Protocol (HSRP) and Virtual Router
Redundancy Protocol (VRRP).
●
Multicast support: Protocol-Independent Multicast (PIM), Any-Source Multicast (ASM), Source-Specific
Multicast (SSM), Anycast Rendezvous Point (RP), Auto-RP, Multicast Source Discovery Protocol (MSDP),
and Internet Group Management Protocol Versions 1, 2, and 3 (IGMPv1, v2, and v3).
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 5 of 46
●
Virtual Route Forwarding lite (VRF-lite) support: Includes multicast routing per VRF instance.
●
Functions such as Dynamic Host Configuration Protocol (DHCP) relay and IP helper.
Licensing controls how many fabric extenders can be used with a Layer 3 configuration and the number of routes
that can be used:
●
With the Base license, you can configure EIGRP stub or use OSPF for up to a maximum of 256 nondirectly
connected routes.
●
With the Enterprise license, you can configure full EIGRP, OSPF with up to 8000 routes, BGP, and VRF-lite.
Without a Base License installed, it is possible to configure SVIs but traffic won’t be routed.
With the Layer 3 license installed, the configuration supports up to 8 fabric extenders.
The Layer 3 card can forward up to 160 Gbps of traffic, or 240 million packets per second (mpps).
The First In, First Out (FIFO) port-to-port latency for the Cisco Nexus 5548P is approximately 2 microseconds, and
for the Cisco Nexus 5548UP and 5596UP it is approximately 1.8 microseconds. With the Layer 3 card, the FIFO
port-to-port latency for routed traffic is approximately 4.7 microseconds.
Topology Choice
In campus designs, a Layer 3 access layer would consist of individual Layer 3 switches dual-homed to the
aggregation layer in Layer 3 mode, where VLANs would not be shared among access switches. For instance, in
Figure 1, in topology (a), you can see that subnet 10.50.1.x is not shared between the two access switches.
The (a) design would not be suitable for a data center in which servers connect in a redundant fashion with NIC
teaming or are configured for high-availability clustering, and in which virtual machine mobility is used. These data
center technologies require Layer 2 adjacency between the access switches.
Topology (b) illustrates a Layer 3 access design for the data center. Access-layer devices are deployed in pairs.
The Layer 2 domain exists exclusively within the pair of access switches. The scope of the Layer 2 domain can be
configured by adding fabric extenders, which allows ports to be added to the same pair of access-layer devices
without adding any spanning-tree load to the topology.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 6 of 46
Figure 1.
Comparing Layer 3 Access Designs in the Campus (a) and in the Data Center (b)
Layer 3 Access Layer
A Layer 3 access-layer design consists of a pair of Cisco Nexus 5500 switches, with or without fabric extenders,
connected with Layer 3 links to an aggregation layer of Cisco Nexus 7000 Series Switches. For example, a pair of
Cisco Nexus 5500 switches with fabric extenders may provide connectivity for a pod consisting of four to eight
racks with two fabric extenders per rack (numbers vary based on the rack density, level of redundancy required,
desired oversubscription ratio, density of MAC addresses, etc.). These Cisco Nexus 5500 switches would then be
aggregated by, for instance, a pair of Cisco Nexus 7000 Series Switches, providing aggregation of multiple pods or
even data center rows. Figure 2 illustrates the building block for this type of design.
The Cisco Nexus 5500 switches at the access layer may also provide connectivity for blade server enclosures. In
some cases, blade server enclosures may contain a switch such as a Cisco blade switch or virtual blade switch
(VBS), and this switch would run spanning tree with the Cisco Nexus 5500 switch as shown in the figure. In the
figure, the fabric extender provides a Layer 2 domain for intra-pod mobility, NIC teaming, and high-availability
clusters.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 7 of 46
Figure 2.
Layer 3 Access Design Without vPC
Layer 3 and vPC Access Layer
In addition to using the topology previously described, the Cisco Nexus5500 platform can be configured for vPC
and connected to an aggregation layer in Layer 3 mode, as shown in Figure 3.
With this design, the Layer 2 domain can be built with fabric extenders dual-connected (active-active fabric
extenders). It can have servers connected in PortChannel mode, and it can accommodate connectivity for blade
switching elements in vPC mode instead of regular spanning-tree mode.
Figure 3.
Layer 3 Access Design with vPC and Active-Active Fabric Extenders
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 8 of 46
Scalability Factors
The following list summarizes important scalability factors for the Cisco Nexus 5500 platform used in Layer 3
mode:
●
The Layer 3 engine on the Cisco Nexus 5500 platform supports up to 8000 adjacencies; this means that in
the Layer 2 domain pod, you cannot have more than 8000 individual MAC addresses. With a maximum of 2
x 384 ports (maximum of 8 fabric extenders with Layer 3 mode) per pod, this limit does not constitute a
constraint. With virtualized servers in a high-density server farm, the MAC address utilization is higher than
the raw count of ports (768 ports). For example, with 10 virtual machines per physical server, assuming that
each server is using a fabric extender port and that each server has routed connectivity, the number of
required adjacencies would be 7680.
●
The Layer 2 forwarding table can accommodate up to 24,000 MAC addresses (this limit already takes into
account the hash collision space).
●
The routing table can accommodate a maximum of 8000 longest-prefix match entries. Most data center
designs consist of an OSPF stub or a totally stubby area and an EIGRP stub area, so the routing table
would normally be far below the limit of 8000 routes.
●
The solution supports a maximum of 2000 Layer 3 multicast groups in non-vPC mode and a maximum 1000
multicast groups in vPC mode. This calculation already accounts for the presence of a (*,G) and a (S,G)
entry for each group.
●
The solution supports a maximum of 4000 Layer 2 multicast groups. The Layer 2 table uses only one entry
for each group. The same group in a different VLAN would count as an additional entry.
●
For Cisco NX-OS Release 5.0(2)N1(1) and later, the Cisco Nexus 5500 platform hardware supports 4013
concurrent VLANs. The total number of available VLANs depends on the use of VSANs (each VSAN
consumes a VLAN) and on the number of Layer 3 interfaces (a Layer 3 interface consumes an internal
VLAN). SVIs conversely do not consume any additional VLAN. You can control VLAN utilization with the
command show resource vlan.
Layer 2 Best Practices for Layer 3 Access
The Layer 3 design recommendations included in this document build on top of the existing best practices for Cisco
Nexus 5500 platform connectivity to servers, fabric extenders and blade switches.
The document “Data Center Access Design with Cisco Nexus 5000 Platform and 2000 Series Fabric Extenders
and Virtual PortChannels” at http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-57282901_Design_N5K_N2K_vPC_DG.pdf describes the Layer 2 design best practices for designs in which the Cisco
Nexus 5500 platform at the access layer connects to the Cisco Nexus 7000 Series Switches at the aggregation
layer in Layer 2 mode. The recommendations that refer to the connectivity from the Cisco Nexus 5500 platform to
servers, fabric extenders, and blade switches are equally applicable to a Layer 3 design.
The following list provides a summary:
●
Choose the spanning-tree algorithm. Keep in mind that Multiple Spanning Tree (MST) Protocol scales
better; Rapid Per-VLAN Spanning Tree Plus (PVST+) Protocol may be easier to deploy, but deploying it
with vPC brings no real additional benefit over deploying one single Spanning-Tree for all VLAN. In a routed
access design, spanning tree runs exclusively between two access switches and helps mostly in the case of
blade switch connectivity and prevents misconfiguration.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 9 of 46
●
With a Layer 3 access design, you may want to make one Cisco Nexus 5500 switch the root and the peer
Cisco Nexus 5500 switch the secondary root. In topologies in which only the fabric extender connects to the
Cisco Nexus 5500 platform, this approach does not provide any particular benefit given that the Layer 2
topology is limited to two switches. In topologies in which blade switches connect to the Cisco Nexus 5500
platform, it is useful to have a root and a secondary root to control the topology. In either case, assigning a
root and a secondary root helps ensure a deterministic configuration.
●
If you are using vPC in the Layer 3 access layer, you should match spanning-tree root priorities with vPC
priorities.
●
Enable pathcost method long to account for the use of 10 Gigabit Ethernet links. vPC links have a
predefined cost of 200. This cost is hard-coded and does not depend on the number of links in the
PortChannel. If you need to modify the spanning-tree cost of links in the topology to obtain a certain
forwarding topology, you can use the command spanning-tree cost <cost>.
●
Host ports should be configured for PortFast (spanning-tree port type edge) or TrunkFast (spanning-tree
port type edge trunk) to remove the delay in the transition from blocking to forwarding on server ports.
●
The Cisco Nexus 5548 and 5596 both support as many PortChannels as the number of ports with or without
a fabric extender. A single PortChannel can have up to 16 active ports. A vPC PortChannel can have up to
32 active ports (16 active ports for each vPC peer).
●
In most deployments, you should use the Link Aggregation Control Protocol (LACP) to negotiate the
creation of PortChannels for both vPC topologies and non-vPC topologies.
●
Bridge Assurance should not be enabled on vPC ports. Bridge Assurance does not add much benefit in a
PortChannel-based configuration, and it may intervene in certain vPC failure scenarios in which it is actually
less disruptive not to error-disable any port. Also, if you want to take advantage of In Service Software
Upgrade (ISSU) on the Cisco Nexus 5000 Series, you should not enable Bridge Assurance on any link
except the peer link (referred to as the multichassis EtherChannel trunk [MCT] in the command-line
interface [CLI]), where it is automatically enabled.
Layer 3 Access Design
Most validated best practices that Cisco has published for Layer 3 designs apply equally to routed access designs
with the Cisco Nexus 5500 platform. This section discusses these best practices and provides information about
aspects that are specific to the Layer 3 engine of the Cisco Nexus 5500 platform.
Routing at the Access Layer
Autostate
Autostate refers to the capability of a Layer 3 SVI to sense the presence of active forwarding ports on the VLAN for
which it provides routing. If a given VLAN does not have any forwarding ports, the Layer 3 SVI goes down. An SVI
interface goes up if the associated VLAN has at least one forwarding port. This mechanism prevents traffic from
begin black-holed by attracting traffic to a given subnet only if that subnet has active ports.
The autostate mechanism on the Cisco Nexus 5500 platform extends to the fabric extender ports. The SVI status
reflects the presence or absence of active forwarding ports on fabric extenders. Figure 4 illustrates this concept.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 10 of 46
Figure 4.
Autostate with Fabric Extenders
Layer 3 Peering at the Access Layer
The Layer 3 access layer runs OSPF or EIGRP. Adding an SVI to the OSPF or EIGRP process both enables the
announcement of the reachability of this subnet and allows the Cisco Nexus 5500 switches to peer over this SVI.
A single Layer 3 access device may have hundreds of SVIs, which would make the Cisco Nexus 5500 platform
peers on hundreds of Layer 3 links without any added benefit. One Layer 3 link established over an SVI between
the access-layer Cisco Nexus 5500 platform is sufficient to provide a Layer 3 escape route should the links to the
core fail. In addition, having hundreds of peering links and VLANs consumes CPU resources unnecessarily.
To correct this situation, you need to make all SVIs passive, with the exception of the Layer 3 VLAN SVI, which is
carried only on the PortChannel that links the two Cisco Nexus 5500 switches.
The configuration for OSPF is as follows:
(config)#interface vlan <number>
(config-if)# ip ospf passive-interface
The configuration for EIGRP is as follows:
(config)#interface vlan
<number>
(config-if)#ip passive-interface eigrp
HSRP and VRRP
For Cisco NX-OS 5.0(3)N1(1) and later, the Cisco Nexus 5500 platform with the Layer 3 engine can forward traffic
for up to 4000 HSRP groups and 255 VRRP groups. Given that each additional HSRP group requires control-plane
resources, the number of groups that you can effectively use depends on timer settings and on the way that the
control plane is used. At the time of this writing, up to 255 HSRP groups have been tested with aggressive timers.
Note:
As a general recommendation, if you plan to use aggressive timers you should keep the total number of
groups within 100 or so. If you observe HSRP groups flapping, this is clear indication that the control plane is
oversubscribed, and you should use default timers. Also, if you are using vPC, there is no need to tune the HSRP
timers because both vPC peers forward traffic.
In the Layer 3 access design, the HSRP gateway is configured on the Cisco Nexus 5500 switch in the access
layer. The access layer for each pod is built with two Cisco Nexus 5500 switch, of which one is the HSRP primary
switch and the peer is the HSRP secondary switch.
According to established best practices, the HSRP priority should match the spanning-tree root and secondary root
placement.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 11 of 46
The restoration of traffic after a failed Layer 3 access switch comes back online depends on the following factors:
●
The time needed for the routes to be learned again by the routing protocol: If traffic from the servers goes to
the newly rebooted Layer 3 switch and this switch does not have routes to the core, traffic would be
dropped.
●
The time needed for the Address Resolution Protocol (ARP) table to be repopulated and, as a result, for the
adjacency table to be rebuilt: If flows from the core reach the newly rebooted Layer 3 switch before the ARP
table is fully rebuilt, the flows whose adjacency is missing are dropped.
●
The time needed for HSRP to take over the primary role (if preemption is configured): If HSRP takes over
the primary role on the newly rebooted Layer 3 switch too quickly, it may attract traffic from the servers
before the routing table has been fully rebuilt.
To reduce traffic drop in the server-to-core direction, you should not use HSRP preemption. However, preemption
helps keep the HSRP topology matched to the spanning-tree topology, which also is desirable.
To address both needs, you can configure preemption with the delay minimum option. This configuration helps
ensure that a newly rebooted Layer 3 switch, such as the Cisco Nexus 5500 platform, in the access layer will not
preempt for a configurable time, allowing the routing table to be rebuilt. The preempt delay value needs to be tuned
according to the characteristics of the individual environment because the time needed for routing to converge
depends on factors such as the number of routes.
In a simple topology example with an OSPF totally stubby area, approximately 4 minutes elapsed between the time
when the control plane was up to the moment when the routes were installed in hardware.
Based on this value, the configuration is as follows:
preempt delay minimum 240
In vPC designs, preempt delay does not change the behavior of HSRP. In vPC designs, as soon as a vPC member
port goes up, traffic destined for the HSRP address is handled by the receiving switch regardless of the preempt
delay configuration.
To control this behavior with vPC, you should use the command delay restore under the vPC domain
configuration. This command delays the restoration of vPC member ports to provide more time for the Layer 3
switch to rebuild the Layer 3 forwarding table.
Aggressive Timers
To achieve convergence in less than a second, you can lower the default HSRP timers and use 250 milliseconds
(ms) of hello time and 750 ms of hold time as in the following configuration:
hsrp 1
timers msec 250 msec 750
Aggressive timers control the convergence time if one Layer 3 engine fails or if the entire Cisco Nexus 5500 switch
fails. The lower the timer value, the faster the recovery from the peer Cisco Nexus 5500 switch.
Be careful when using aggressive timers because the lower the timer values, the higher the load on the CPU, so if
you are using hundreds of SVIs, do not change the timers from the default values.
If you observe HSRP groups flapping, this is clear indication that the control plane is oversubscribed, and you
should use the default timers.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 12 of 46
Aggressive timers do not help in vPC designs because in vPC both HSRP peers forward traffic, even before a
failure.
At the time of this writing, ISSU is not supported with Layer 3 features, but if and when ISSU is supported with
Layer 3 features, aggressive timers may be incompatible with the ISSU upgrade procedure.
Connectivity Between Aggregation and Access Layers
Layer 3 connectivity between the aggregation and access layers uses Layer 3 equal-cost multipathing (ECMP).
ECMP
In the ECMP configuration, each Layer 3 switch has two routes and two associated hardware forwarding adjacency
entries. Before a failure, traffic is forwarded using both these forwarding entries. When an adjacent link or neighbor
fails, the switch hardware and software immediately remove the forwarding entry associated with the lost neighbor.
After the removal of the route and forwarding entries associated with the lost path, the switch still has a remaining
valid route and associated Layer 3 forwarding entry. Because the switch still has an active and valid route, it does
not need to trigger or wait for routing protocol convergence and is immediately able to continue forwarding all traffic
using the remaining Layer 3 entry.
Figure 5 illustrates this concept.
Figure 5.
Rerouting Traffic with ECMP
In a typical design, the links between the aggregation and access layers may be Layer 3 PortChannels, as shown
in Figure 6. If a single PortChannel member fails, the routing cost of the link is recalculated, and the topology
forwards traffic accordingly.
Figure 6 illustrates the failure of a link in a Layer 3 PortChannel. The link may be configured as follows:
Interface Portchannel 21
no switchport
ip address 10.50.3.1/31
ip ospf 1 area 1.1.1.1
ip ospf network point-to-point
If the OSPF reference bandwidth is 1000000, the cost of the link before the link failure is 50, and after the failure
the cost becomes 100.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 13 of 46
Figure 6.
PortChannel Cost Calculation with Layer 3 Links
Subnets for Layer 3 Links
A routed access design requires the allocation of a subnet for each Layer 3 link. The aggregation layer will then
summarize these subnets together with the subnets that are used for server connectivity. You should to use /30 or
/31 to reduce the consumption of IP addresses.
Note:
For more information about /31 addressing, see RFC 3021 at http://www.faqs.org/rfcs/rfc3021.html.
Link Cost Tuning
In OSPF, the reference bandwidth by default is 40 Gbps. As a result, if you have PortChannels with more than four
10 Gigabit Ethernet links, you may want to adjust the reference bandwidth so that it reflects the real bandwidth of
the link. In the case of EIGRP, the cost is calculated as a function of several parameters, including bandwidth and
delay.
For OSPF you can change the reference bandwidth globally as follows:
L3-N5548-1(config-router)# auto-cost reference-bandwidth
<1-4000000>
Rate in Mbps (bandwidth) (Default) *Default value is 40000
<1-4000> Rate in Gbps (bandwidth)
*Default value is 40
Considering that the maximum bandwidth of an individual link on the Cisco Nexus 5500 platform is 160 Gbps (16 x
10 Gigabit Ethernet), if you set the reference bandwidth at 1 Tbps instead of 40 Gbps, OSPF will be able to
perform routing decisions that can distinguish between a 160-Gbps PortChannel and a 40-Gbps PortChannel. After
you change the reference bandwidth, the cost of a 160-Gbps PortChannel is 6, and the cost of a 40-Gbps
PortChannel is 25.
router ospf 1
auto-cost reference-bandwidth 1000000
The SVI cost does not reflect the available bandwidth, because an SVI does not have a one-to-one mapping with a
finite number of links. Sometimes SVIs are used to build Layer 3 connectivity: for instance, in a routed access
topology, the Layer 3 link between two Layer 3 access-layer devices is just a VLAN on a PortChannel. In this case,
routing adjacency is built from an SVI as illustrated in Figure 7.
Figure 7 illustrates the case of a Layer 3 link built with SVIs over an existing PortChannel (which could be the peer
link in vPC topologies). The default cost on the SVIs does note reflect the available bandwidth on this link.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 14 of 46
Figure 7.
Layer 3 Peering Between two Cisco Nexus 5500 Switches
You can change the cost of the Layer 3 route built over an SVI to a meaningful value (which takes into account the
reference bandwidth) by using the command ip ospf cost. For instance, if there are two 10 Gigabit Ethernet links
between the Cisco Nexus 5500 switches, you can set the cost to 50; if there are four 10 Gigabit Ethernet links, you
can set the cost to 25.
In the case of EIGRP, you can also tune the SVI cost to reflect the actual bandwidth, by using the command ip
bandwidth eigrp.
Reducing Polarization
Traffic polarization is a well-known phenomenon that manifests itself when using topologies that distribute traffic by
using hashing of the same type. In this kind of topology, the combined effect of cascading several layers of ECMP
(or PortChannel) hashing is unequal traffic distribution.
Figure 8 illustrates this concept.
In the figure, the traffic that goes from the Cisco Nexus 5500 switch in the access layer to the Cisco Nexus 7000
Series Switch in the aggregation is hashed by ECMP on the left link. Traffic that arrives on the Cisco Nexus 7000
Series Switch will be hashed again to choose the next-hop ECMP route. If we assume that the hashing polynomial
used by the Cisco Nexus 7000 Series line card and by the Cisco Nexus 5500 switch in the access layer is the
same, the traffic may always take the same link because what the Cisco Nexus 7000 Series Switch receives is
already the result of a hashing operation of the same kind as the one that this switch is going to perform; hence, it
will yield the same result.
To reduce the possibility of traffic polarization, you should add different universal ID keys to the Layer 3 loadbalancing configuration on the Cisco Nexus 5500 switch in the access layer compared to those used on the Cisco
Nexus 7000 Series Switch in the aggregation layer:
L3-N5548-1(config)# ip load-sharing address source-destination universal-id
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 15 of 46
Figure 8.
When the Universal ID Is Used, Layer 3 Hashing Polarization Does Not Occur
Summarization
To help ensure the optimal recovery time for data traffic flows in the data center, you need to optimize the routing
design to provide a minimal and deterministic convergence time for failure cases.
The length of time needed for EIGRP, OSPF, or any routing protocol to restore traffic flows is limited by the
following three main factors:
●
The time required to detect the loss of a valid forwarding path.
●
The time required to determine a new best path (which is partially determined by the number of routers
involved in determining the new path, or the number of routers that must be informed of the new path before
the network can be considered converged).
●
The time required to update software and associated Cisco Express Forwarding hardware forwarding tables
with the new routing information.
If the switch has redundant equal-cost paths, all three of these processes are performed locally within the switch
and controlled by the internal interaction of software and hardware. If there is no second equal-cost path, EIGRP or
OSPF must determine a new route, and this process plays a large role in determining network convergence time.
In the case of EIGRP, the time primarily depends on the number of EIGRP queries that the switch needs to
generate and the amount of time needed for the response to each of those queries to return to calculate a feasible
successor (path). The time required for each of these queries to be completed depends on the distance they have
to propagate in the network before a definite response can be returned. To reduce the time required to restore
traffic flows when a full EIGRP routing convergence is required, the design must provide strict limits on the number
and range of queries generated.
In the case of OSPF, the time required to flood and receive link-state advertisements (LSAs) in combination with
the time needed to run the Dijkstra shortest path first (SPF) algorithm to determine the shortest-path tree (SPT)
provides a limit on the time required to restore traffic flows. Optimizing network recovery involves tuning the design
of the network to reduce the time and resources required to complete these two processes.
By limiting the number of peers that an EIGRP router must query or the number of LSAs that an OSPF peer must
process, you can optimize the rerouting of the traffic when a link fails.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 16 of 46
These objectives can be achieved with a combination of:
●
Stub routing
●
Summarization
For summarization to be possible, the IP addressing allocation should map to a route summarization scheme,
which should be based on the building blocks of the network. When a link or a node in a building block fails, this
failure should not result in routing updates in other building blocks.
With EIGRP, you can implement a tiered summarization scheme by summarizing directly at the access layer for a
Layer 3 access topology and performing additional summarizing at the aggregation layer.
With OSPF summarization performed at the area border router (ABR), Layer 3 access summarization is performed
at the aggregation layer. With a Layer 2 access layer, summarization is also still performed at the aggregation
layer.
The OSPF area for the data center should be a stub or totally stubby area.
Design with OSPF
OSPF implements a two-tier hierarchical routing model that uses a core or backbone tier known as area zero (0).
Attached to that backbone through ABRs are a number of secondary tier areas. In a typical data center design, the
data center constitutes a separate area, with the aggregation-layer switches configured as ABRs. The ABRs
control summarization and route propagation from the core to the data center area and from the data center to the
core.
ABRs forward the following LSAs:
●
ABRs for regular areas forward summary LSAs of type 3 (summary of type-1 and type-2 LSAs from an
adjacent area), autonomous system boundary router (ASBR) summary (information about routes to the
ASBR), and external (routes outside this autonomous system).
●
Stub area ABRs forward summary LSAs of type 3 and the summary default (0.0.0.0).
●
Totally stubby area ABRs forward the summary default (0.0.0.0).
The convergence time for network failures is proportional to the number of routes that are present in a given area;
therefore, it is good practice to limit the propagation of routes by configuring a stub or a totally stubby area. The
use of a stub area configuration for the aggregation block area prevents the propagation of external routes to the
aggregation block. However, configuring the area as totally stubby also prevents the propagation of inter-area
routes. In this configuration, each of the distribution switch ABRs creates a default route that provides the
forwarding from the routed access layer to the aggregation layer.
To create a totally stubby area, the configuration is as follows:
router ospf 1
area 1.1.1.1 stub no-summary
area 1.1.1.1 range 10.50.0.0/16
auto-cost reference-bandwidth 1000000
The stub portion of the configuration blocks external LSAs from entering the area through the ABR. The nosummary option of the stub parameter blocks inter-area summary LSAs from entering the area. The ABRs also
inject a default route (0.0.0.0) into the stub area to provide access to the routes not propagated into the area.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 17 of 46
Figure 9 illustrates the OSPF area design with a Layer 3 access layer.
Figure 9.
Scope of Totally Stubby Area
Other OSPF best practices include:
●
Tuning the reference bandwidth.
●
Establishing Layer 3 peering between access devices.
●
Disabling peering (passive interfaces) on all SVIs except for the Layer 3 VLAN interface.
For faster convergence of the Layer 3 topology, you can also consider tuning the SPF timers.
OSPF Timers
OSPF configuration provides the capability to customize the default timers that control the LSA pacing and the
pacing of SPF algorithm execution. In Cisco IOS® Software, you should reduce these values from their default
settings to improve network convergence while helping ensure that multiple iterations have a dampening effect
applied to reduce processor utilization if intermittent or flapping connectivity occurs in the network.
The three parameters interoperate to determine how long an SPF calculation takes to run after notification of a
topology change event (arrival of an LSA).
At the arrival of the first topology notification, the spf-start or initial hold timer controls the wait time before the SPF
calculation starts.
If no subsequent topology change notification arrives (no new LSA) during the hold interval, the SPF is free to run
again as soon as the next topology change event is received.
However, if a second topology change event is received during the hold interval, the SPF calculation is delayed
until the hold interval expires. Additionally, in this second case, the hold interval is temporarily doubled.
After the expiration of any hold interval, the timer is reset, and any future topology changes trigger an SPF again
based on the initial timer. Typical values used when tuning Cisco Catalyst® Family products running Cisco IOS
Software are as follows:
●
sfp-start: 10 ms
●
spf-hold: 100 to 500 ms
●
spf-max-wait: 5000 ms (5 seconds)
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 18 of 46
LSA throttle timers control the generation of type-1 and type-2 LSAs in the event of a network interface change.
Three configuration values are used: an initial delay timer, a hold timer, and a maximum hold timer. Typical values
used with Cisco Catalyst Family and Cisco IOS Software are as follows:
●
lsa-start: 10 ms
●
lsa-hold: 100 to 500 ms
●
lsa-max-wait: 5000 ms (5 seconds)
To optimize OSPF for fast convergence in the data center and to be consistent with Cisco IOS Software, the
default throttle timers in Cisco NX-OS can be updated using the command syntax timers throttle spf 10 100 5000.
Design with EIGRP
If you configure the EIGRP process to run in the stub-connected state, the access switch advertises all connected
subnets. The switch also advertises to its neighbor routers that it is a stub or nontransit router and thus should
never be sent queries to learn of a path to any subnet other than the advertised connected routes. Use of the
EIGRP stub approach isolates the EIGRP queries in case of failure.
In addition, the aggregation-layer switch summarizes the data center subnets to the core.
Figure 10 illustrates the EIGRP topology.
Figure 10.
Hierarchical Summarization with EIGRP
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 19 of 46
Figure 11 illustrates the configuration of EIGRP stub and summarization.
Figure 11.
EIGRP Stub
Multicast Considerations
This section is not meant to replace the existing well-known best practices for multicast, which you can find at
http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6552/ps6592/whitepaper_c11-474791.html.
Proven multicast best practices apply to the routed access design. It is particularly important to remember that
multicast cannot function and better or converge any faster than the IGP infrastructure. A solid multicast design
starts with a solid IGP design and tuning.
The rendezvous point will typically be a device at either the aggregation layer or the core. The Cisco Nexus 5500
platform can be used as a PIM Rendezvous Point, but normally the Cisco Nexus 5500 platform at the access layer
will not be used as the Rendezvous Point.
Note:
The Cisco Nexus 5500 platform supports Auto-RP, BSR, PIM Anycast-RP, and MSDP.
In non-vPC designs, you can use the Cisco Nexus 5500 platform for PIM-ASM or PIM-SSM designs. Bidirectional
PIM (Bidir-PIM) is not supported. The Cisco Nexus 5500 platform in Layer 3 mode supports up to 2000 multicast
groups.
Note:
In vPC designs, the Cisco Nexus 5500 platform used in Layer 3 mode supports up to 1000 multicast
groups. In vPC designs, the Cisco Nexus 5500 platform does not support PIM-SSM.
The preferred mechanism for distributing the Rendezvous Point information is Anycast-RP combined with Auto-RP,
which offers the advantages of faster convergence and the benefits of not having to manually configure the
Anycast-RP address on every router. If the device that provides the Rendezvous Point function supports PIM
Anycast-RP, you can use PIM Anycast-RP instead of MSDP.
Table 1 summarizes the options for synchronizing the source information for the Rendezvous Point and the way
that the Rendezvous Point information can be propagated.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 20 of 46
Table 1.
Rendezvous Point Information
Source Information Synchronization Between Rendezvous Points
Distribution of Rendezvous Point Information
MSDP (RFC 3618)
Auto-RP
PIM Anycast-RP
Bootstrap router (BSR; RFC 5059)
To configure the Cisco Nexus 5500 platform to learn about the Rendezvous Point, you can configure the
Rendezvous Point manually (static Rendezvous Point approach):
ip pim rp-address <IP>
Alternatively, you can configure the Rendezvous Point to learn through Auto-RP (equivalent to Cisco IOS Software
ip pim autorp listener command):
ip pim auto-rp forward listen
As usual, all interfaces (Layer 3 interfaces, Layer 3 VLAN used for peering, and the SVIs that provide the gateway
to the server) need to be configured with this command:
ip pim sparse-mode
Regarding IGMP, if you do not configure the version explicitly, IGMPv2 will run; otherwise, for PIM-SSM designs or
for faster removal of a group, you can configure IGMPv3 with explicit tracking:
interface Vlan50
ip pim sparse-mode
ip igmp version 3
vlan50
ip igmp snooping explicit-tracking
The access layer always has two Cisco Nexus 5500 switches for redundancy. One is the PIM designated router
(PIM-DR), and one is the IGMP querier.
The Cisco Nexus 5500 platform may provide routing for multicast sources. In this case, the PIM-DR is in charge of
registering local sources with the Rendezvous Point. The Rendezvous Point will then send PIM joins to the source
(which is behind the PIM-DR) to build an SPT. This process creates a PIM (S,G) state along the path between the
source and the Rendezvous Point. In the presence of equal-cost paths to the source, any router in the path will
choose the highest next-hop IP address.
The Cisco Nexus 5500 platform may provide routing for multicast receivers. In this case, upon receipt of IGMP
reports for a given group, the PIM-DR installs the (*,G) entry for that group and sends a PIM join to the
Rendezvous Point. This process creates a PIM (*,G) state along the path between the Rendezvous Point and the
receiver, which is behind a Cisco Nexus 5500 switch.
The PIM-DR device is chosen based on the highest IP address, but you can have a more deterministic
configuration by defining a priority:
ip pim dr-priority
The timer for PIM-DR health monitoring can be tuned to reduce the failover time associated with a failure of the
designated router, with the following command:
ip pim hello-interval <period> [msec]
The IGMP querier can also be tuned to accelerate the recovery of the multicast entries in the Layer 2 forwarding
table upon failover.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 21 of 46
The access layer typically is a multicast stub network, and having two routing devices on the same segment
implies that one of the routing devices will see the multicast stream destined for local receivers on an interface that
does not support reverse-path forwarding (RPF). As is the case in all multicast deployments, this non-RPF traffic
must be rate limited to avoid saturating the CPU.
On the Cisco Nexus 5500 platform, the non-RPF traffic is rate limited to allow the PIM control plane still to function
(PIM needs to see a copy of the traffic) and at the same time to prevent the CPU from being overwhelmed.
Layer 3 Access Design with vPC
Designing the access layer with Layer 3 on the Cisco Nexus 5500 platform and vPC combines the benefits of a
Layer 3 access layer with the benefits of being able to connect fabric extenders redundantly and to connect servers
redundantly with a PortChannel.
Most of the design best practices for vPC designs apply, plus specific tuning that is related to the use of the Layer
3 engine on the Cisco Nexus 5500 platform.
Preliminary considerations
The Cisco Nexus 5500 platform with a Layer 3 engine can forward traffic regardless of whether it is vPC primary or
secondary traffic and regardless of whether the HSRP group is in active or standby mode on a given SVI. As long
as one SVI is active for HSRP, the vPC peer can also route traffic for that SVI.
The Cisco Nexus 5500 platform supports the peer-gateway option, which allows the vPC peer to route traffic
whose destination MAC address is the burned-in MAC address (BIA) of the vPC peer.
Using the Cisco Nexus 5500 platform as a vPC Layer 3 access switch allows the use of PIM-ASM but not the use
of PIM-SSM. In this mode, the Cisco Nexus 5500 platform can support up to 1000 multicast groups.
In PIM-ASM mode, vPC peers will switch to the SPT as soon as they find the information about a source. Changing
this behavior to “use shared-tree” is not supported in vPC mode.
Cisco Nexus 5000 Series vPC Baseline Configuration
The Layer 3 design recommendations included in this document build on the existing best practices for connectivity
from the Cisco Nexus 5500 platform to servers, fabric extenders, and blade switches in the presence of vPC.
Figure 12 shows the components of a Cisco Nexus 5500 platform vPC deployment.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 22 of 46
Figure 12.
vPC Components
The following list provides a summary of vPC configuration best practices:
●
Connect the two Cisco Nexus 5500 switches through redundant 10 Gigabit Ethernet links to form the peer
link between vPC peers. There is no specific benefit of separating vPC and non-vPC VLANs. This link
carries both vPC VLANs and non-vPC VLANs. When the peer link is configured, Bridge Assurance is
automatically enabled on this link.
●
The peer keepalive is an out-of-band monitoring mechanism that is used for vPC peers to arbitrate roles
and to resolve peer-link failures. You should configure the peer-keepalive connectivity either through the
mgmt0 interface or through an SVI and a separate port. When using a Cisco Nexus 5500 platform with a
Layer 3 card, you can allocate a VRF instance for to carry the peer keepalive.
●
Unlike on the Cisco Nexus 7000 Series, on the Cisco Nexus 5000 Series direct connectivity of the peer
keepalive through mgmt0 from one vPC peer to the other is an acceptable practice, although routing over
the management network (or any out-of-band network) is still preferred for deployment.
●
The peer-keepalive traffic should never be carried on a VLAN over the peer link; such a configuration would
make the peer keepalive useless.
●
You should use Link Aggregation Control Protocol (LACP) on vPCs that connect switches and, when
supported by NIC team software, also on the servers.
Peer Link
Because of its importance, the peer link should always be configured in a redundant manner. The loss of the peer
link is recovered in a way that prevents split subnets and continuous flooding.
The peer link carries the control traffic used to synchronize MAC address tables and IGMP entries. When
connectivity is lost on this link, MAC addresses and IGMP entries can no longer be synchronized. In a dual-active
scenario, traffic for existing MAC entries and existing multicast group members may continue to flow correctly, but
if a new unicast MAC address was learned after the loss of the peer link, traffic destined for this MAC address
would cause flooding. Also, generation of a new IGMP report after the loss of the peer link would trigger proper
IGMP snooping processing on one vPC peer only. As a result, multicast traffic arriving on the other vPC peer would
be dropped.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 23 of 46
For these reasons, when the peer link is lost, vPC shuts down vPC member ports on the operational secondary
switch to avoid a dual-active scenario, as illustrated in Figure 13.
In the figure, Host1, Host2, and Host3 are connected to the Cisco Nexus 5000 Series and 2000 Series Switches,
respectively, with a PortChannel in a vPC configuration.
Upon failure of the peer link, the vPC secondary device verifies that the vPC primary device is still alive (by using
the heartbeat verification mechanism of the peer keepalive link) and correctly shuts down the vPC member port on
the secondary Cisco Nexus 5000 Series Switch and associated Cisco Nexus 2000 Series Fabric Extenders.
Because of this behavior, unicast and multicast traffic continues flowing correctly through the vPC primary device.
Figure 13 illustrates failure of the peer link.
Figure 13.
Behavior of vPC with Peer-Link Failure
Behavior of vPC with Routing
vPC peers run routing protocols independently, so most design best practices that have been analyzed for nonvPC environments apply:
●
The spanning-tree root and secondary root should be configured the same way as in non-vPC designs.
●
You should make all interface VLANs (SVI) passive except for the one used as a backup route when links to
the core are down.
●
Summarization schemes are configured on the vPC peers exactly the same way as in non-vPC designs.
The following list summarizes some differences between vPC and non-vPC designs:
●
In vPC designs, you do not need to tune the HSRP timers, because both HSRP interfaces are forwarding.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 24 of 46
●
In vPC designs, you do not need to configure a HSRP preempt delay because both HSRP interfaces are
forwarding regardless of whether their status is active or standby.
●
You should configure the vPC restore delay with a value similar to the one you would use for the HSRP
preempt-delay minimum. Based on current testing (and subject to change with additional data from further
testing), you should configure a value of 4 minutes or more.
Interaction of vPC with Gateway Redundancy Protocols
The use of HSRP in the context of vPC does not require any special configuration. The active HSRP interface
answers ARP requests just as regular HSRP deployments do, but with vPC both HSRP interfaces (active and
standby) can forward traffic.
The configuration on the HSRP primary device looks like this:
interface vLAN50
no shutdown
ip address 10.50.0.251/24
hsrp 1
preempt delay minimum 240
priority 150
timers
1
3
ip 10.50.0.1
The configuration on the HSRP secondary device looks like this:
interface vLAN50
no shutdown
ip address 10.50.0.252/24
hsrp 1
preempt delay minimum 240
priority 130
timers
1
3
ip 10.50.0.1
The most significant difference between the HSRP implementation of a non-vPC configuration and a vPC
configuration is that both vPC peers program the HSRP MAC addresses with the routed flag.
Given this fact, routable traffic can be forwarded by both the vPC primary device (with HSRP) and the vPC
secondary device (with HSRP), with no need to send this traffic to the HSRP primary device.
The same behavior applies to VRRP configurations with vPC.
Peer Gateway
If a host or a switch forwards a frame to the Layer 3 gateway and this Layer 3 gateway is present on a vPC switch
pair, so long as the destination MAC is destined for the HSRP MAC address, everything works as expected.
If the frame that is sent to the Layer 3 gateway uses the MAC BIA instead of the HSRP MAC address, the
PortChannel hashing of the frame may forward it to the wrong vPC peer, which would then just bridge the frame to
the other vPC peer.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 25 of 46
On the Cisco Nexus 5500 platform, the vPC peer would then just route the frame out to the vPC instead of
dropping it as it would do with a nonrouted frame.
Figure 14 shows the case in which a server sends traffic to a remote MAC address (RMAC-B), but the traffic gets
hashed to switch RMAC-A. As a result, the frame is bridged to the vPC peer (switch RMAC-B), which then routes
the frame to vPC2, thus reaching the server.
Figure 14.
If servers use the BIA MAC instead of the HSRP MACt raffic May Be Routed Across the Peer Link
Although this approach is fine from a data forwarding correctness perspective it is suboptimal because traffic is
crossing the peer link while a direct path to vPC2 exists. To optimize the forwarding path, you should configure the
peer-gateway command under the vPC domain. This command enables the vPC peers to exchange information
about their respective BIA MAC addresses so that they can route the traffic locally without having to send it over
the peer link.
The use of the peer gateway is recommended.
Interactions of vPC with SVIs
When the peer link fails, the vPC shuts down vPC member ports on the vPC secondary switch. When Layer 3 is
used, shutting down the vPC member ports and leaving the associated SVIs up would cause traffic dropped,
because the vPC secondary peer would still advertise reachability for the subnets associated with the vPC VLANs
that have been shut down.
To avoid this problem, the vPC shuts down the SVIs on the vPC secondary device to help ensure that routing
forwards traffic exclusively to the vPC primary device, as shown in Figure 15.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 26 of 46
Figure 15.
SVIs Go Down When the vPC Peer Link Goes Down.
What Is a Layer 3 vPC?
Using Layer 3 interfaces on Cisco Nexus 5500 platforms that are operating in vPC mode is fully supported to
provide the gateway function for servers and to forward traffic from clients to servers. However, Layer 3 peering
over vPCs (often referred to as Layer 3 vPC) is rarely necessary, and it requires an understanding of the
interoperation of vPC and routing protocols and is subject to restrictions. Because of this Layer 3 peering over vPC
is not recommended.
A Layer 3 vPC is a vPC between a Layer 3 switch or router and two vPC peers whereby there are three routing
entities peering over the Layer 2 segment provided by the vPC itself.
Figure 16 clarifies this concept. In the figure, router R has a PortChannel that is connected to both N5k1 and N5k2.
This PortChannel carries a VLAN, for which router R has an IP address. Both N5k1 and N5k2 also have IP
addresses on this VLAN.
From a Layer 3 perspective, this topology is logically equivalent to the topology at the far right of the figure in which
three routing entities are peering over a Layer 2 segment.
The challenge with these topologies is that they include two load-balancing mechanisms that are acting
independently:
●
ECMP, which load-balances traffic from R to N5k1 and N5k2 based on hashing of the IP addresses and
rewrites the destination MAC address of the packets accordingly
●
Layer 2 PortChannel hashing, which may choose to send a packet to N5k1 even if ECMP chose to send the
packet to N5k2
A Layer 3 vPC can function correctly from a data forwarding perspective when a peer gateway is configured, but
this approach may have repercussions for the IGP peering operations that are outside the scope of this document.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 27 of 46
Figure 16.
Layer 3 and vPC Interaction
For this reason, the topology that uses Layer 3 peering over vPC (the topology in Figure 16) is not recommended.
These topologies do not provide additional benefits compared to Layer 3 links with ECMP, and they can be
modified to use Layer 3 links instead of Layer 3 peering over a vPC. Layer 3 connectivity between vPC peers and
the aggregation and core layers is the recommended topology.
ECMP Between Aggregation and Access Layers
At the time of this writing, the recommended practice is the use of Layer 3 links to connect the Layer 3 access layer
to the Layer 3 aggregation layer instead of the use of a Layer 3 vPC.
Figure 17 illustrates the topology of a vPC access layer connected with Layer 3 links to the aggregation layer. This
topology uses ECMP for traffic distribution on all links between the access and aggregation layers just as in the
non-vPC designs.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 28 of 46
Figure 17.
Layer 3 Access Topology with vPC
With ECMP, as previously stated, each Layer 3 switch has two routes and two associated hardware forwarding
adjacency entries. Before a failure, traffic is being forwarded using both of these forwarding entries. If an adjacent
link or neighbor fails, the switch hardware and software immediately remove the forwarding entry associated with
the lost neighbor. After the removal of the route and forwarding entries associated with the lost path, the switch still
has a remaining valid route and associated Layer 3 forwarding entry. Because of this, the switch does not need to
trigger or wait for routing protocol convergence and is immediately able to continue forwarding all traffic using the
remaining Layer 3 entry.
Establishing IGP Peering over the Peer Link
In vPC designs, you should make sure to include a Layer 3 link or VLAN between the Layer 3 switching vPC peers
so that the routing areas are adjacent. If you do this, if the Layer 3 links to the aggregation layer fail, a backup route
exists to route traffic to the vPC peer so that the traffic is forwarded to the core through the routing peer.
Figure 18 illustrates this concept.
Figure 18.
Layer 3 Peering over the vPC Peer Link
You can change the cost of the Layer 3 route built over an SVI to a meaningful value (which takes into account the
reference bandwidth) by using the command ip ospf cost. For instance, if there are two 10 Gigabit Ethernet links
between the Cisco Nexus 5500 switches, you can set the cost to 50; if there are four 10 Gigabit Ethernet links, you
can set the cost to 25.
The following code shows how to create a Layer 3 link to connect the access-layer switches at Layer 3. Just as in
the non-vPC design, make sure to change the cost of the SVI to reflect the bandwidth available on the peer link.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 29 of 46
interface Vlan3
ip address 10.50.3.8/31
ip router ospf 1 area 1.1.1.1
ip ospf cost 50
ip ospf network point-to-point
Multicast
The vPC multicast implementation is designed to address the following requirements:
●
Avoiding duplicate frames.
●
Helping ensure that multicast traffic reaches orphan ports on non-vPC VLANs and Layer 3 ports.
●
Optimizing traffic forwarding between sources and receivers without crossing the vPC peer link when
possible.
To understand how the Cisco Nexus 5500 platform operates with multicast in a vPC configuration, consider these
two distinct forwarding scenarios:
●
Forwarding with directly connected sources and receivers
●
Forwarding with remote sources and receivers
You should also understand that vPC synchronizes IGMP join information between the vPC peers, but it does not
synchronize PIM state information.
The general rules of routing for multicast traffic with vPC are as follows:
●
If a vPC peer receives multicast packets from a directly connected source, it performs a local multicast route
(mroute) lookup and replicates packets for each outgoing interface (OIF). In a non-vPC design, only one of
the routing peers would have installed mroutes for source-to- receiver traffic, and traffic would cross the
peer link. With vPC, the multicast implementation is referred to as dual-DR because traffic from directly
connected sources to directly connected receivers can be routed by either vPC peer.
●
If the OIF is a VLAN trunked over a vPC peer link, one copy is sent over the peer link for each VLAN
present in the OIF list. By default, the vPC peer link is considered to be an mrouter port. Therefore, the
multicast packet will be sent over to peer-link for each receiver VLAN. This action helps ensure that orphan
ports on a given VLAN can receive this traffic.
●
Multicast traffic received through the peer link cannot go out to a vPC member port at Layer 2, nor can it be
routed. This behavior prevents duplicates.
●
A special VLAN must be used for traffic that may require routing on the vPC peer.
●
The traffic received by a vPC peer over this “special” VLAN can be routed only to non-vPC ports, to avoid
introducing duplicates.
Each vPC peer sends one copy of the multicast traffic to the vPC peer through a reserved VLAN on the peer link.
This reserved VLAN is configured with the following CLI command:
N5596-L3-1(config)# vpc bind-vrf <vrf name> vlan <VLAN ID>
N5596-L3-1(config)# vpc bind-vrf default vlan 4000
One reserved VLAN is required for each VRF instance. Without this CLI command, the receivers in non-VPC
VLANs and the receivers connected to Layer 3 interfaces on the vPC peer may not receive multicast traffic.
Note:
Non-vPC VLANs are the VLANs that are not trunked over the peer link.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 30 of 46
Note:
Unlike regular VLANs used for server connectivity, the VLAN used in the “bind-vrf” command must not be
“created”, if you have “created” this VLAN, you need to do “no vlan <number>”, the bind-vrf <vrf_name> vlan
<VLAN number>, will then work.
PIM Dual-DR Implementation for Directly Connected Sources and Receivers
Just as in non-vPC designs, one vPC peer is the PIM-DR. Unlike non-vPC designs, both vPC peers install
multicast routes in the event of directly connected sources and receivers. If this was not the case, all multicast
traffic would have to be sent to the PIM-DR for it to route the traffic and possibly send it back again on the peer link
to reach the final destination. With the use of PIM dual-DR, the traffic forwarded between directly connected
sources and receivers does not need to cross the peer link.
In the case of traffic flowing from a directly connected source to remote receivers, the (S,G) state and outgoing
interface list (OIL) are programmed on only one of the vPC peers, depending on how PIM joins were routed to the
source. In this case, traffic may have to cross the peer link before being routed.
PIM-DR and Source Registration
The elected PIM-DR is responsible for sending source registration to the Rendezvous Point (RP). When multicast
traffic from a directly connected source is received by a vPC peer that is not a designated router, the designated
router is informed through a Cisco Fabric Services message about the source and group addresses. The PIM-DR
generates source registration packets to the RP. The RP will then joins the SPT to the source through either vPC
peer. The vPC peer then installs the (S,G) state with the appropriate OIL for the source to send traffic to the RP.
No synchronization of state occurs between vPC peers for this (S,G) state.
At some point, the last-hop router next to the receiver also switches from the shared tree to the (S,G) state, which
creates the (S,G) state on either the same vPC peer or the other vPC peer.
To help ensure that the traffic coming from the source behind a vPC is routed to both to the RP and the SPT to the
receiver, vpc bind-vrf must be configured. This configuration is required because the vPC may hash the traffic
from the source to either vPC peer, not both. The vpc bind-vrf command helps ensure that both vPC peers get a
copy of the traffic so that they can route it as needed based on the configured (S,G) state and associated OIL.
PIM Prebuilt SPT
The implementation of vPC on the Cisco Nexus 5500 platform supports the concept of PIM prebuilt SPT by adding
the configuration “ip pim pre-build-spt”. With PIM prebuilt SPT, both vPC peers send PIM joins to the Rendezvous
Point to receive a copy of the multicast traffic.
The IGMP reports from a receiver in the vPC domain are synchronized between vPC peers. Both the PIM-DR and
the non-PIM-DR install the (*,G) state and the OIL information, and both vPC peers send a PIM join to the
Rendezvous Point.
After the multicast traffic is received, both vPC peers join the SPT to the source, but only one of the two vPC peers
actively forwards traffic. The vPC that performs this task is selected on the basis of which vPC peer has the
shortest path to the source, and in case of identical distance, on the basis of the vPC role, with the vPC primary
device having greater priority in forwarding traffic than the vPC secondary device.
During the transition from shared tree to SPT, duplicate frames temporarily exist.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 31 of 46
Sizing the Peer Link for Multicast Traffic
Under normal conditions, the peer link is not used because traffic enters and exits the Cisco Nexus 5000 Series
Switch from one of the vPC member ports. Certain types of traffic use the peer link:
●
Traffic destined for orphan ports
●
Multicast traffic sent
●
Broadcast traffic
When a server joins a multicast group; it sends an IGMP report, which is synchronized between the vPC peers
(N5k1 and N5k2) to associate the multicast MAC addresses of that given multicast group with the vPC member
port on the peer Cisco Nexus 5000 Series Switch (N5k2).
When a multicast stream arrives, it may be hashed to either Cisco Nexus 5000 Switch (N5k1 or N5k2). This
multicast stream not only is sent to the receivers but also over the vPC peer link, in case there are other singleattached subscribers on N5k1.
In the presence of routing, the vPC peer may also route traffic to other VLANs, in which case this additional traffic
is also sent over the vPC peer link.
In addition to these copies, a copy of the multicast traffic is sent to the vPC peer through the bind-VRF VLAN for
the purpose of routing to Layer 3 ports connected to the vPC peer.
To optimize the utilization of the peer link, you can use the following CLI commands to avoid sending multicast
traffic over the peer link for each receiver VLAN when there are no orphan ports on the vPC peer:
N5596-L3-1(config)# no ip igmp snooping mrouter vpc-peer-link
Warning: IGMP Snooping mrouter vpc-peer-link should be globally disabled on peer
VPC switch as well.
N5596-L3-1(config)#
With this CLI command configured, the multicast packet will be sent over to peer link only for those VLANs that
have orphan ports.
Even with this optimization in place, in the presence of sustained multicast traffic you may need to properly size the
vPC peer link to prevent the peer link from becoming a bottleneck.
Example Configurations
This section includes some example configurations of a Layer 3 access layer with and without vPC. Figure 19
illustrates the topology that is used in the examples.
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 32 of 46
Figure 19.
Topology Used in Configuration Examples
Layer 3 Access Design Without vPC
Cisco Nexus 5000 Series Switch 1 (N5k1)
feature ospf
feature interface-vlan
feature hsrp
vlan 3
name l3-vlan
vlan 50
name 10.50.1.x
vlan 60
name 10.50.2.x
!
! Spanning-tree primary root
!
spanning-tree vlan 1-3967,4048-4093 priority 24576
interface Vlan3
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 33 of 46
description l3vlan
no shutdown
ip address 10.50.3.10/31
ip ospf cost 50
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
interface Vlan50
description server-vlan
no shutdown
ip address 10.50.1.2/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
hsrp 1
preempt delay minimum 240
priority 110
timers 1 3
ip 10.50.1.1
interface Vlan60
description server-vlan
no shutdown
ip address 10.50.2.2/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
hsrp 1
preempt delay minimum 240
priority 110
timers 1 3
ip 10.50.2.1
interface port-channel10
description 5k1-to-5k2
switchport mode trunk
speed 10000
interface port-channel21
description to-7k1
no switchport
speed 10000
ip address 10.50.3.1/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
interface port-channel23
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 34 of 46
description to-7k2
no switchport
speed 10000
ip address 10.50.3.5/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
interface loopback0
ip address 128.0.0.5/32
ip router ospf 1 area 1.1.1.1
router ospf 1
area 1.1.1.1 stub no-summary
auto-cost reference-bandwidth 1000000
ip load-sharing address source-destination universal-id 60
Cisco Nexus 5000 Series Switch 2 (N5k2)
feature ospf
feature interface-vlan
feature hsrp
vlan 3
name l3-vlan
vlan 50
name 10.50.1.x
vlan 60
name 10.50.2.x
!
! Spanning-tree secondary root
!
spanning-tree vlan 1-3967,4048-4093 priority 28672
interface Vlan3
description l3-vlan
ip address 10.50.3.11/31
ip ospf cost 50
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
no shutdown
interface Vlan50
description server-vlan
ip address 10.50.1.3/24
ip ospf passive-interface
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 35 of 46
ip router ospf 1 area 1.1.1.1
hsrp 1
preempt delay minimum 240
timers 1 3
ip 10.50.1.1
no shutdown
interface Vlan60
description server-vlan
ip address 10.50.2.3/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
hsrp 1
preempt delay minimum 240
timers 1 3
ip 10.50.2.1
no shutdown
interface port-channel22
description to-7k1
no switchport
speed 10000
ip address 10.50.3.3/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
interface port-channel24
description to-7k2
no switchport
speed 10000
ip address 10.50.3.7/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
interface loopback0
ip address 128.0.0.5/32
ip router ospf 1 area 1.1.1.1
router ospf 1
area 1.1.1.1 stub no-summary
auto-cost reference-bandwidth 1000000
ip load-sharing address source-destination universal-id 60
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 36 of 46
N5k1 with Multicast
feature ospf
feature interface-vlan
feature hsrp
feature pim
vlan 3
name l3-vlan
vlan 50
name 10.50.1.x
vlan 60
name 10.50.2.x
!
! Spanning-tree primary root
!
spanning-tree vlan 1-3967,4048-4093 priority 24576
interface Vlan3
description l3vlan
no shutdown
ip address 10.50.3.10/31
ip ospf cost 50
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
interface Vlan50
description server-vlan
no shutdown
ip address 10.50.1.2/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
ip pim dr-priority 10
hsrp 1
preempt delay minimum 240
priority 110
timers 1 3
ip 10.50.1.1
interface Vlan60
description server-vlan
no shutdown
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 37 of 46
ip address 10.50.2.2/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
ip pim dr-priority 10
hsrp 1
preempt delay minimum 240
priority 110
timers 1 3
ip 10.50.2.1
interface port-channel10
description 5k1-to-5k2
switchport mode trunk
speed 10000
interface port-channel21
description to-7k1
no switchport
speed 10000
ip address 10.50.3.1/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
interface port-channel23
description to-7k2
no switchport
speed 10000
ip address 10.50.3.5/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
interface loopback0
ip address 128.0.0.4/32
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
router ospf 1
area 1.1.1.1 stub no-summary
auto-cost reference-bandwidth 1000000
ip pim auto-rp forward listen
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 38 of 46
ip load-sharing address source-destination universal-id 60
N5k2 with Multicast
feature ospf
feature interface-vlan
feature hsrp
feature pim
vlan 3
name l3-vlan
vlan 50
name 10.50.1.x
vlan 60
name 10.50.2.x
!
! Spanning-tree secondary root
!
spanning-tree vlan 1-3967,4048-4093 priority 28672
interface Vlan3
description l3-vlan
ip address 10.50.3.11/31
ip ospf cost 50
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
no shutdown
interface Vlan50
description server-vlan
ip address 10.50.1.3/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
hsrp 1
preempt delay minimum 240
timers 1 3
ip 10.50.1.1
no shutdown
interface Vlan60
description server-vlan
ip address 10.50.2.3/24
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 39 of 46
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
hsrp 1
preempt delay minimum 240
timers 1 3
ip 10.50.2.1
no shutdown
interface port-channel10
description 5k1-to-5k2
switchport mode trunk
speed 10000
interface port-channel22
description to-7k1
no switchport
speed 10000
ip address 10.50.3.3/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
interface port-channel24
description to-7k2
no switchport
speed 10000
ip address 10.50.3.7/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
interface loopback0
ip address 128.0.0.5/32
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
router ospf 1
area 1.1.1.1 stub no-summary
auto-cost reference-bandwidth 1000000
ip pim auto-rp forward listen
ip load-sharing address source-destination universal-id 60
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 40 of 46
Layer 3 Access Design with vPC
N5k1
feature ospf
feature interface-vlan
feature hsrp
feature pim
feature vpc
vlan 3
name l3-vlan
vlan 50
name 10.50.1.x
vlan 60
name 10.50.2.x
!
! Spanning-tree primary root
!
spanning-tree vlan 1-3967,4048-4093 priority 24576
vpc domain 2
role priority 100
peer-keepalive destination 128.0.0.5 source 128.0.0.4 vrf default
peer-gateway
auto-recovery
graceful-consistency-check
!
! Configuring a restoration delay of 180s = 3 minutes
!
delay restore 180
no ip igmp snooping mrouter vpc-peer-link
vpc bind-vrf default vlan 300
interface Vlan3
description l3vlan
no shutdown
ip address 10.50.3.10/31
ip ospf cost 50
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 41 of 46
ip pim sparse-mode
interface Vlan50
description server-vlan
no shutdown
ip address 10.50.1.2/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
ip pim dr-priority 10
hsrp 1
! delay values are not useful in a vpc topology
preempt
priority 110
! hsrp timers are not useful in a vpc configuration
ip 10.50.1.1
interface Vlan60
description server-vlan
no shutdown
ip address 10.50.2.2/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
ip pim dr-priority 10
hsrp 1
! delay values are not useful in a vpc topology
preempt
priority 110
! hsrp timers are not useful in a vpc configuration
ip 10.50.2.1
interface port-channel10
description 5k1-to-5k2
switchport mode trunk
speed 10000
vpc peer-link
interface port-channel21
description to-7k1
no switchport
speed 10000
ip address 10.50.3.1/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 42 of 46
ip pim sparse-mode
interface port-channel23
description to-7k2
no switchport
speed 10000
ip address 10.50.3.5/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
interface loopback0
ip address 128.0.0.4/32
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
router ospf 1
area 1.1.1.1 stub no-summary
auto-cost reference-bandwidth 1000000
ip pim auto-rp forward listen
ip load-sharing address source-destination universal-id 60
N5k2
feature ospf
feature interface-vlan
feature hsrp
feature pim
feature vpc
vlan 3
name l3-vlan
vlan 50
name 10.50.1.x
vlan 60
name 10.50.2.x
! Spanning-tree secondary root
!
spanning-tree vlan 1-3967,4048-4093 priority 28672
vpc domain 2
role priority 100
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 43 of 46
peer-keepalive destination 128.0.0.4 source 128.0.0.5 vrf default
peer-gateway
auto-recovery
graceful-consistency-check
!
! Configuring a restoration delay of 180s = 3 minutes
!
delay restore 180
no ip igmp snooping mrouter vpc-peer-link
vpc bind-vrf default vlan 300
interface Vlan3
description l3-vlan
ip address 10.50.3.11/31
ip ospf cost 50
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
no shutdown
interface Vlan50
description server-vlan
ip address 10.50.1.3/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
hsrp 1
preempt
ip 10.50.1.1
no shutdown
interface Vlan60
description server-vlan
ip address 10.50.2.3/24
ip ospf passive-interface
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
hsrp 1
preempt
ip 10.50.2.1
no shutdown
interface port-channel10
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 44 of 46
description 5k1-to-5k2
switchport mode trunk
speed 10000
vpc peer-link
interface port-channel22
description to-7k1
no switchport
speed 10000
ip address 10.50.3.3/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
interface port-channel24
description to-7k2
no switchport
speed 10000
ip address 10.50.3.7/31
ip ospf network point-to-point
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
interface loopback0
ip address 128.0.0.5/32
ip router ospf 1 area 1.1.1.1
ip pim sparse-mode
router ospf 1
area 1.1.1.1 stub no-summary
auto-cost reference-bandwidth 1000000
ip pim auto-rp forward listen
ip load-sharing address source-destination universal-id 60
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
Page 45 of 46
For More Information
For additional information about Layer 3 and multicast designs please refer to well known best practices:
●
“High Availability Campus Network Design - Routed Access Layer using EIGRP or OSPF”:
http://www.cisco.com/application/pdf/en/us/guest/netsol/ns432/c649/ccmigration_09186a0080811468.pdf
●
IP Multicast Best Practices for Enterprise Customers:
http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6552/ps6592/whitepaper_c11-474791.html
For more information about the Cisco Nexus 5000 Series Product Family please refer to this url:
●
http://www.cisco.com/go/nexus5000/
Printed in USA
© 2011 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information.
C07-673997-00
07/11
Page 46 of 46