Cisco SFS InfiniBand Redundancy
Configuration Guide
Release 2.10
December 2007
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
Text Part Number: OL-12957-02
FINAL REVIEW—CISCO CONFIDENTIAL
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
CCVP, the Cisco logo, and Welcome to the Human Network are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of
Cisco Systems, Inc.; and Access Registrar, Aironet, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS,
Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, Follow Me
Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, LightStream, Linksys,
MeetingPlace, MGX, Networkers, Networking Academy, Network Registrar, PIX, ProConnect, ScriptShare, SMARTnet, StackWise, The Fastest Way to Increase Your Internet
Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship
between Cisco and any other company. (0711R)
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the
document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
Cisco SFS InfiniBand Redundancy Configuration Guide
© 2007 Cisco Systems, Inc. All rights reserved.
CONTENTS
Preface
vii
Audience
vii
Organization
vii
Conventions
viii
Related Documentation
ix
x
CHAPTER
1
Overview
CHAPTER
2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
1-1
2-1
Cisco SFS 7008P Server Switch Redundancy 2-2
Software Redundancy 2-2
Power Supply Module Redundancy 2-4
Fan Tray Redundancy 2-4
Management Interface Module Redundancy 2-5
Fabric Controller Redundancy 2-6
Line Interface Module Redundancy 2-6
IB Fabric Redundancy 2-8
Cisco SFS 7000P and SFS 7000D Server Switch Redundancy
Power Supply Redundancy 2-9
Port Redundancy 2-10
IB Fabric Redundancy 2-10
2-9
CHAPTER
3
InfiniBand Server Switch Module Redundancy for the IBM BladeCenter
CHAPTER
4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3504 Server Switch Redundancy 4-2
Software Redundancy 4-2
IB Switch Module Redundancy 4-2
Fabric Redundancy 4-2
AC Power-Fan Module Redundancy 4-3
Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy
3-1
4-1
4-4
Cisco SFS InfiniBand Redundancy Configuration Guide
78-12957-02
iii
Contents
Cisco SFS 3012R Server Switch Redundancy 4-5
Power Supply Redundancy 4-5
Blower Redundancy 4-6
Controller Module Redundancy 4-6
InfiniBand Switch Module Redundancy 4-7
Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy
Fabric Redundancy 4-7
4-7
Cisco SFS 3001 Server Switch Redundancy 4-9
Power Supply Module Redundancy 4-9
Fan Redundancy 4-9
Fabric Redundancy 4-10
CHAPTER
5
Subnet Manager Redundancy
Embedded Subnet Manager
5-1
5-3
High-Performance Subnet Manager
5-4
Setting up Master and Standby Subnet Managers 5-4
Setting up Master and Standby Subnet Managers Using Embedded Subnet Managers 5-4
Setting up Master and Standby Subnet Managers with High-Performance Subnet Managers
Setting Up Database Synchronization 5-8
Setting up Database Synchronization for Embedded Subnet Managers 5-9
Setting up Database Synchronization for High-Performance Subnet Managers
CHAPTER
6
Host Redundancy, and IPoIB and SRP Redundancies
HCA Redundancy 6-1
Single HCA Redundancy 6-1
Multiple HCA Redundancy 6-3
Two HCAs with the IBM BladeCenter
5-6
5-10
6-1
6-3
IPoIB High Availability 6-4
Cisco SFS IPoIB High Availability 6-4
Merging Physical Ports 6-4
Unmerging Physical Ports 6-5
OFED IPoIB High Availability 6-6
Configuring IPoIB High Availability 6-6
Verifying IPoIB High Availability 6-8
OFED SRP High Availability
CHAPTER
7
6-8
Ethernet Gateway and IPoIB Redundancies
7-1
Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch
7-3
Cisco SFS InfiniBand Redundancy Configuration Guide
iv
78-12957-02
Contents
Verifying Redundancy Configuration for Cisco SFS 3504 Server Switches 7-6
Verifying Bridge Group Configuration for Cisco SFS 3504 Server Switches 7-8
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch 7-10
Configuring Ethernet Gateway Redundancy Using a Single Cisco SFS 3012R Server Switch 7-10
Verifying Redundancy Group Configuration for a Single Cisco SFS 3012R Server Switch 7-13
Verifying Bridge Group Configuration for a Single Cisco SFS 3012R Server Switch 7-14
Configuring Ethernet Gateway Redundancy Using Dual Cisco SFS 3012R Server Switches 7-15
Verifying Redundancy Group Configuration for Dual Cisco SFS 3012R Server Switches 7-19
Verifying Bridge Group Configuration for Dual Cisco SFS 3012R Server Switches 7-20
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch 7-23
Verifying Redundancy Group Configuration for Cisco SFS 3001 Server Switches 7-27
Verifying Bridge Group Configuration for Cisco SFS 3001 Server Switches 7-28
CHAPTER
8
Fibre Channel Gateway and SRP Redundancies
Dynamic Load Balancing
8-2
Dynamic Gateway Failover
Path Affinity
8-1
8-2
8-2
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch
Verifying Configured Initiator 8-5
Verifying IT 8-6
Verifying LU 8-7
8-3
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch 8-8
Configuring Fibre Channel Gateway Redundancy Using a Single Cisco SFS 3012 Server Switch 8-8
Verifying Configuration for a Single Cisco SFS 3012R Server Switch 8-10
Configuring Fibre Channel Gateway Redundancy Using Two Cisco SFS 3012R Server Switches 8-12
Verifying Configuration for Two Cisco SFS 3012R Server Switches 8-13
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch
Configuring Two Cisco SFS 3001 Server Switches 8-16
Verifying Redundancy Configuration for Cisco SFS 3001 Server Switches 8-17
CHAPTER
9
APPENDIX
A
Typical Redundancy Use Case
Acronyms and Abbreviations
8-16
9-1
A-1
INDEX
Cisco SFS InfiniBand Redundancy Configuration Guide
78-12957-02
v
Contents
Cisco SFS InfiniBand Redundancy Configuration Guide
vi
78-12957-02
Preface
This preface describes who should read the Cisco SFS InfiniBand Redundancy Configuration Guide,
how it is organized and its document conventions. It contains the following sections:
•
Audience, page vii
•
Organization, page vii
•
Conventions, page viii
•
Related Documentation, page ix
•
Obtaining Documentation and Submitting a Service Request, page x
Audience
The intended audience for this document is the administrator responsible for the Enterprise data center
who configures redundancy in a Server Fabric Switch environment. This administrator should have
experience configuring and managing equipment such as server switches, the Subnet Manager, Ethernet
gateways, Host Channel Adapters, and software drivers.
Organization
This guide is organized as follows:
Chapter
Title
Description
Chapter 1
Overview
This chapter provides an overview about the
guide.
Chapter 2
Cisco SFS 7008P and SFS 7000 Series This chapter includes redundancy
Server Switch Redundancy
information for the Cisco SFS 7008P and
Cisco SFS 7000 Series Server Switches.
Chapter 3
InfiniBand Server Switch Module
Redundancy for the IBM BladeCenter
This chapter includes redundancy
information for the
InfiniBand Server Switch for the IBM
BladeCenter Redundancy.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
vii
Preface
Conventions
Chapter
Title
Description
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000
Series Server Switch Redundancy
This chapter includes redundancy
information for the Cisco SFS 3504 and the
Cisco SFS 3000 Series Server Switches.
Chapter 5
Subnet Manager Redundancy
This chapter includes the Subnet Manager
redundancy information.
Chapter 6
Host Redundancy, and IPoIB and SRP
Redundancies
This chapter includes information about
HCA, IPoIB, and SRP redundancies.
Chapter 7
Ethernet Gateway and IPoIB
Redundancies
This chapter includes information about
Ethernet gateway redundancy.
Chapter 8
Fibre Channel Gateway and SRP
Redundancies
This chapter includes information about
Fibre Channel gateway redundancy.
Chapter 9
Typical Redundancy Use Case
This chapter describes a typical redundancy
use case.
Appendix A
Acronyms and Abbreviations
This chapter defines the acronyms and
abbreviations that are used in this
publication.
Conventions
This document uses the following conventions:
Convention
Description
boldface font
Commands, command options, and keywords are in
boldface. Bold text indicates Chassis Manager elements or
text that you must enter as-is.
italic font
Arguments in commands for which you supply values are in
italics. Italics not used in commands indicate emphasis.
Menu1 > Menu2 >
Item…
Series indicate a pop-up menu sequence to open a form or
execute a desired function.
[ ]
Elements in square brackets are optional.
{x|y|z}
Alternative keywords are grouped in braces and separated by
vertical bars. Braces can also be used to group keywords
and/or arguments; for example, {interface interface type}.
[x|y|z]
Optional alternative keywords are grouped in brackets and
separated by vertical bars.
string
A nonquoted set of characters. Do not use quotation marks
around the string or the string will include the quotation
marks.
screen
font
Terminal sessions and information the system displays are in
font.
screen
boldface screen
Information you must enter is in boldface screen font.
font
Cisco SFS InfiniBand Redundancy Configuration Guide
viii
OL-12957-02
Preface
Related Documentation
Convention
italic screen
Description
font
Arguments for which you supply values are in italic
screen font.
^
The symbol ^ represents the key labeled Control—for
example, the key combination ^D in a screen display means
hold down the Control key while you press the D key.
< >
Nonprinting characters, such as passwords are in angle
brackets.
!, #
An exclamation point (!) or a pound sign (#) at the beginning
of a line of code indicates a comment line.
Notes use the following convention:
Note
Means reader take note. Notes contain helpful suggestions or references to material not covered in the
manual.
Cautions use the following convention:
Caution
Means reader be careful. In this situation, you might do something that could result in equipment
damage or loss of data.
Related Documentation
For additional information related to Cisco SFS InfiniBand Redundancy Configuration Guide, see the
following documents:
•
Cisco SFS InfiniBand Software Configuration Guide
•
Cisco SFS 3504 Multifabric Server Switch Hardware Installation Guide
•
Cisco SFS 3012R Multifabric Server Switch Hardware Installation Guide
•
Cisco SFS 3001 Multifabric Server Switch Hardware Guide
•
Cisco SFS 7008P InfiniBand Server Switch Hardware Installation Guide
•
Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide
•
Cisco High-Performance Subnet Manager for InfiniBand Server Switches
•
Cisco SFS Product Family Chassis Manager User Guide
•
Cisco SFS Product Family Element Manager User Guide
•
Cisco SFS Product Family Command Reference
•
Cisco SFS InfiniBand Fibre Channel Gateway User Guide
•
Cisco SFS InfiniBand Ethernet Gateway User Guide
•
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
ix
Preface
Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, submitting a service request, and gathering additional
information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and
revised Cisco technical documentation, at:
http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html
Subscribe to the What’s New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed
and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free
service and Cisco currently supports RSS version 2.0.
Cisco SFS InfiniBand Redundancy Configuration Guide
x
OL-12957-02
CH A P T E R
1
Overview
The Cisco SFS InfiniBand Redundancy Configuration Guide contains redundancy configuration
information for use with Enterprise solutions that contain Cisco InfiniBand (IB) Server Fabric Switches
(SFS). It describes hardware and software redundancies and redundant configurations that are available
in the Cisco SFS IB environment.
Note
For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and
Abbreviations.”
This guide describes the various types of hardware and software redundancies available with the
different products that are available to build an SFS IB environment. It also describes the various typical
redundant configurations that a user may build in this environment and includes a typical use case that
has no single point of failure.
The redundancies described in this guide include the following topics:
•
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
•
InfiniBand Server Switch Module Redundancy for the IBM BladeCenter
•
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
•
Subnet Manager Redundancy
•
Host Redundancy, and IPoIB and SRP Redundancies
•
Ethernet Gateway and IPoIB Redundancies
•
Fibre Channel Gateway and SRP Redundancies
•
Typical Redundancy Use Case
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
1-1
Chapter 1
Overview
Cisco SFS InfiniBand Redundancy Configuration Guide
1-2
OL-12957-02
CH A P T E R
2
Cisco SFS 7008P and SFS 7000 Series Server
Switch Redundancy
This chapter describes the Cisco SFS 7008P and SFS 7000 Series Server Switch redundancy and
includes the following sections:
•
Cisco SFS 7008P Server Switch Redundancy, page 2-2
•
Cisco SFS 7000P and SFS 7000D Server Switch Redundancy, page 2-9
For information related to Subnet Manager Redundancy, see Chapter 5, “Subnet Manager Redundancy”.
Note
For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and
Abbreviations.”
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
2-1
Chapter 2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
Cisco SFS 7008P Server Switch Redundancy
Cisco SFS 7008P Server Switch Redundancy
This section describes redundancies in the Cisco SFS 7008P Server Switch and includes the following
topics:
•
Software Redundancy, page 2-2
•
Power Supply Module Redundancy, page 2-4
•
Fan Tray Redundancy, page 2-4
•
Management Interface Module Redundancy, page 2-5
•
Fabric Controller Redundancy, page 2-6
•
Line Interface Module Redundancy, page 2-6
•
IB Fabric Redundancy, page 2-8
For more details about the Cisco SFS 7008P Server Switch, see the Cisco SFS 7008P InfiniBand Server
Switch Hardware Installation Guide.
Software Redundancy
This section describes redundancy in the Cisco SFS 7008P Server Switch software.
The Cisco SFS 7008P Server Switch supports the hot-standby feature. When the primary controller fails,
a standby controller assumes management of the server switch without having to reboot or reset other
cards in the chassis.
When two controllers are installed in a Cisco SFS 7008P Server Switch, one controller acts as the
primary controller, and the other acts as the standby controller. The primary controller is responsible for
managing the chassis. The standby controller waits to take-over if the primary controller fails or is
rebooted.
Verify the primary switch controller by entering the show card command in the CLI. The oper-code of
the primary controller card is normal and for the standby controller is standby. An asterix marks the
controller card that services this CLI session. So, from a console CLI session, your console port is on
the card marked with an asterix.
The following is sample output from the show card command and verifies the status of each controller
card:
SFS-7008P# show card
=========================================================================
Card Information
=========================================================================
admin
oper
admin
oper
oper
slot type
type
status
status
code
------------------------------------------------------------------------11* controllerFabric12x
controllerFabric12x
up
up
normal
12
controllerFabric12x
controllerFabric12x
up
up
standby
Cisco SFS InfiniBand Redundancy Configuration Guide
2-2
OL-12957-02
Chapter 2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
Cisco SFS 7008P Server Switch Redundancy
The following is sample output from the show card command from the console of slot 12 of the server
switch and verifies the status of each controller card:
SFS-7008P# show card
=========================================================================
Card Information
=========================================================================
admin
oper
admin
oper
oper
slot type
type
status
status
code
------------------------------------------------------------------------11
controllerFabric12x
controllerFabric12x
up
up
normal
12* controllerFabric12x
controllerFabric12x
up
up
standby
When a Cisco SFS 7008P Server Switch is powered on, the fabric card in slot 11 assumes the primary
card status, and the fabric card in slot 12 is the standby card. A fabric card is only eligible to be a
controller card if the fabric card is installed in either slot 11 or 12 and a corresponding management
interface module is available.
If a fabric card is operating in the recovery mode (such as, when it is executing the OS Recovery Image
software), it is not eligible to be a primary or standby controller. The master and standby controllers
automatically synchronize the state and configuration information. When a standby controller assumes
managing a chassis, the service of other cards in the chassis (such as the Line Interface Modules, fabric
controllers, and management interface modules) are not impacted. The other cards are not rebooted,
reset, or interrupted. The standby controller is accessible through the serial console port. A user cannot
access the standby controller using Telnet, SSH, SNMP, or HTTP.
The OS CLI is available on the standby controller (through the serial console only). The CLI on a standby
controller is limited to read-only operations. A user can enter show commands but cannot enter config
commands.
A card is only placed in-service if that card is running the same software as the primary controller. When
the standby controller card has a different version than the primary controller card, the standby controller
card shows a wrong image for its card opercode. In this event, no synchronization occurs between the
two cards. The sys-sync-state for both the controller cards stay at not started.
When a hot-standby controller card takes over in the event of a primary controller card failure, the
hot-standby controller card behaves according to its sys-sync-state.
When the sys-snyc-state is complete, the hot-standby controller card continues management without
disturbing the services. No additional configuration file is executed, and there is no reboot to the node
cards.
When the sys-sync-state is not started, the hot-standby controller card executes its startup-config, if it is
present and continues management. There is no reboot to the node cards.
When the sys-sync-state is in progress, the primary controller card and the hot-standby controller card
are partially synchronized. The hot-standby controller card reboots itself to avoid unpredictable results.
The synchronization begins only after the operStatus of the hot-standby controller card changes to up.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
2-3
Chapter 2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
Cisco SFS 7008P Server Switch Redundancy
Power Supply Module Redundancy
This section describes the power supply module redundancy in the Cisco SFS 7008P Server Switch.
The Cisco SFS 7008P Server Switch has two AC-DC bulk power supply modules (see Figure 2-1). Each
power supply has self-contained fans for cooling. Only one power supply, in either of the two slots is
required to power the system. The second power supply acts as a redundant power supply. Each power
supply has its own AC inlet and runs on an independent AC circuit. If the active power supply fails, the
second power supply automatically takes over the full load of the server switch. There is no user
intervention required in case of a failover. The current is shared in an active-active mode.
Figure 2-1
Cisco SFS 7008P Server Switch Front View
Power supplies
Fan
Slot 9
Node slot 2
Slot 10
Core slot 1
Slot 11
Core slot 2
Node slot 3
Slot 12
Slot 13
Node slot 4
Slot 14
191986
Fan
Node slot 1
If a power supply module fails, it must remain within the chassis until a replacement is available. If it is
removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7008P
Server Switch is in operation, the power supply module bay can remain empty for no more than three
minutes.
Fan Tray Redundancy
This section describes fan tray redundancy in the Cisco SFS 7008P Server Switch.
The Cisco SFS 7008P Server Switch has two fan trays. Only one fan tray in either of the two slots is
required to cool the system. The fan trays are hot swappable. The fan trays operate in an active-active
mode.
If a fan tray fails, it must remain within the chassis until a replacement is available. If it is removed, a
blanking panel must be installed instead. During replacement, when the Cisco SFS 7008P Server Switch
is in operation, the fan tray bay can remain empty for no more than three minutes.
Cisco SFS InfiniBand Redundancy Configuration Guide
2-4
OL-12957-02
Chapter 2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
Cisco SFS 7008P Server Switch Redundancy
Management Interface Module Redundancy
This section describes management interface module redundancy in the Cisco SFS 7008P Server Switch.
The Cisco SFS 7008P Server Switch supports redundant, hot-swappable management interface modules.
Each management interface module is paired to one of the fabric controller core modules. There are two
core slots in the Cisco SFS 7008P Server Switch and two management interface modules (see Figure 2-1
and Figure 2-2). The controller in each of the core slots uses a management interface module to
communicate with the outside network. Each management interface module provides its own serial and
Ethernet port. Both sets of ports must be connected. The failover of the management interface module
is paired with the failover of the fabric controller cards (see the “Fabric Controller Redundancy” section
on page 2-6).
If a management interface module fails, it must remain within the chassis until a replacement is
available. If it is removed, a blanking panel must be installed instead. During replacement, when the
Cisco SFS 7008P Server Switch is in operation, the management interface module bay can remain empty
for no more than three minutes.
Figure 2-2
Cisco SFS 7008P Rear View - Management Interface Modules and Line Interface
Modules
Line Interface
Modules
183386
Management I.O
modules
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
2-5
Chapter 2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
Cisco SFS 7008P Server Switch Redundancy
Fabric Controller Redundancy
This section describes the fabric controller redundancy in the Cisco SFS 7008P Server Switch.
The behavior and responsibility of each fabric controller is determined by the type of slot into which it
is inserted. A fabric controller can be installed in either a node slot or a core slot. When the software on
a fabric controller module detects that a module is inserted in a core slot, it arbitrates for system
mastership and runs. When you power on the Cisco SFS 7008P Server Switch, by default, the card in
slot 11 becomes the active card and the card in slot 12 becomes the standby card (see Figure 2-1). The
master and standby controller cards automatically synchronize state and configuration information.
Thus, the switch does not have to be rebooted and none of the cards in the chassis require resetting if a
switch failover occurs.
Fabric controller cards in the core slots are paired to the management interface modules of the
Cisco SFS 7008P Server Switch (see the “Management Interface Module Redundancy” section on
page 2-5). For redundancy, the pairing of both fabric controller cards and the management interface
modules should be installed and operational. The two core cards operate in an active-active mode and
are required for 100% throughput. If one core card fails, 50% of bandwidth is available to the system. If
the failing card is the active master, the standby master assumes control of the chassis.
Removing the fabric controller in the core slot that currently acts as master, causes a failover to the
standby pair of the fabric controller and management interface module pair. Before removing a fabric
controller from one of the core slots, make sure that the redundant core fabric controller is functional.
The fabric controllers in the node slots act as slaves and do not operate in an active-standby
configuration. Each node card is paired with two Line Interface Modules (see the “Line Interface Module
Redundancy” section on page 2-6). If a node card fails, only ports connected to it and its corresponding
Line Interface Modules are affected.
If a core or node card fails, it can be left within the chassis until a replacement is available. If it is
removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7008P
Server Switch is in operation, the card bay can be left empty for no more than three minutes.
Line Interface Module Redundancy
This section describes the Line Interface Module redundancy in the Cisco SFS 7008P Server Switch.
Line Interface Modules support redundant connection from the HCAs (see Figure 2-2). Line Interface
Modules are hot-swappable and redundant components. Each Line Interface Module is paired with a
fabric controller node card (see the “Fabric Controller Redundancy” section on page 2-6).
If a Line Interface Module fails, it can be left within the chassis until a replacement is available. If it is
removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7008P
Server Switch is in operation, the Line Interface Module bay can be left empty for no more than three
minutes.
Figure 2-3 is an illustration that shows the core cards and Line Interface Module redundancy within a
Cisco SFS 7008P Server Switch.
Cisco SFS InfiniBand Redundancy Configuration Guide
2-6
OL-12957-02
Chapter 2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
Cisco SFS 7008P Server Switch Redundancy
Figure 2-3
Redundancy within a Cisco SFS 7008P Server Switch
InfiniBand Fabric
SFS 7008P
Line interface modules pair 1,2
Line interface modules pair 3,4
Line interface modules pair 5,6
Line interface modules pair 7,8
SFS 7000D-2
InfiniBand
Host
InfiniBand
Host
183340
SFS 7000D-1
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
2-7
Chapter 2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
Cisco SFS 7008P Server Switch Redundancy
IB Fabric Redundancy
This section describes the IB fabric redundancy using Cisco SFS 7008P Server Switches.
For redundancy at the fabric level, IB HCAs can be dual-connected to a redundant pair of
Cisco SFS 7008P Server Switches (see Figure 2-4). The IB links are active. Traffic over redundant IB
links varies and is based on upper-level protocols and applications.
Redundancy with Dual Cisco SFS 7008P Server Switches
SFS 7008P-1
SFS 7008P-2
SFS 7000D-1
SFS 7000D-2
SFS 7000D-3
SFS 7000D-4
InfiniBand
Host-1
InfiniBand
Host-2
InfiniBand
Host-3
InfiniBand
Host-4
183339
Figure 2-4
Cisco SFS InfiniBand Redundancy Configuration Guide
2-8
OL-12957-02
Chapter 2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
Cisco SFS 7000P and SFS 7000D Server Switch Redundancy
Cisco SFS 7000P and SFS 7000D Server Switch Redundancy
This section describes the Cisco SFS 7000P and SFS 7000D Server Switch redundancy and includes the
following topics:
•
Power Supply Redundancy, page 2-9
•
Port Redundancy, page 2-10
•
IB Fabric Redundancy, page 2-10
Redundancy in the Cisco SFS 7000P and SFS 7000D Server Switches is supported at the hardware, port,
and fabric levels. For more details about the Cisco SFS 7000P and SFS 7000D Server Switches, see the
Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide.
Power Supply Redundancy
This section describes power supply redundancy in the Cisco SFS 7000P and SFS 7000D Server
Switches.
The Cisco SFS 7000P and SFS 7000D Server Switches power supply module is an integrated power
supply and fan unit. A server switch can have up to two power supplies installed (see Figure 2-5). The
switch requires only one power supply to function. The second power supply acts as a redundant power
supply. The power supply modules are hot swappable. The replacement of any one power supply module
does not disrupt the operation of the device and can be successfully completed without removing the
device from a rack or disconnecting any cables.
Each power supply has its own AC inlet and runs on an independent AC circuit. The server switch
automatically has the power supplies operating in active-active or active-standby mode. If one power
supply were to fail, the second power supply automatically takes over the full load of the server switch.
There is no user intervention required in case of a failover.
If a power supply module fails, it can be left within the chassis until a replacement is available. If it is
removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7000P or
the SFS 7000D Server Switch is in operation, the power supply module bay can be left empty for no
more than three minutes.
Cisco SFS 7000P and SFS 7000D Server Switches
191995
Figure 2-5
Power supply
modules
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
2-9
Chapter 2
Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy
Cisco SFS 7000P and SFS 7000D Server Switch Redundancy
Port Redundancy
This section describes port redundancy using the Cisco SFS 7000P and SFS 7000D Server Switches.
The Cisco SFS 7000P and SFS 7000D Server Switches each have 24 IB ports. Redundancy at the port
level is such that if any single IB port fails, none of other ports have interrupted service.
In addition, to achieve port redundancy, users can employ two server switches. If the IB ports on one
server switch fails, the second server switch automatically takes over the load of the first server switch.
IB Fabric Redundancy
This section describes fabric redundancy in the Cisco SFS 7000P and SFS 7000D Server Switches.
For redundancy at the fabric level, IB HCAs can be dual-connected to a redundant pair of
Cisco SFS 7000 Series Server Switches. The Cisco SFS 7000 Series Server Switch redundant
configuration is active-active. No hardware configuration is required. The IB links are active-active. But
applications and upper-level protocol use of redundant IB links varies and could be active-active or
active-standby, depending on the application.
In this typical configuration, a dual-port HCA is connected to a pair of Cisco SFS 7000 Series Server
Switches (see Figure 2-6). This configuration provides server redundancy.
One Dual-Port HCA Connected to a Redundant Pair of Cisco SFS 7000 Series Server
Switches
InfiniBand Fabric 1
InfiniBand Fabric 2
SFS 7000D-1
SFS 7000D-2
InfiniBand Host
182855
Figure 2-6
For greater redundancy, connect two single-port HCAs to a redundant pair of server switches. Such a
configuration provides host and IB fabric redundancy.
Note
Figure 2-6 shows the Cisco SFS 7000D Server Switches in a redundant configuration. The
Cisco SFS 7000P Server Switches can also be connected in similar redundant configurations.
Cisco SFS InfiniBand Redundancy Configuration Guide
2-10
OL-12957-02
CH A P T E R
3
InfiniBand Server Switch Module Redundancy
for the IBM BladeCenter
This chapter contains redundancy information about the InfiniBand Server Switch for the
IBM BladeCenter.
For more information on the InfiniBand Server Switch for the IBM BladeCenter, see the
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide.
Note
For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and
Abbreviations.”
The user can create a redundant dual-switch topology in an IBM BladeCenter. To enable IB redundancy
for the IBM BladeCenter chassis, you must install one server switch module in each available slot. HCA
expansion cards do not support redundant links to a single server switch module slot. When you add a
second server switch module to the BladeCenter chassis, each port of each HCA expansion card connects
to a server switch module. Figure 3-1 shows an example of a configuration in which there are 14
dual-port HCAs in the IBM BladeCenter. Each HCA, in this redundant configuration, is connected to the
two server switch modules in the BladeCenter.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
3-1
Chapter 3
Figure 3-1
InfiniBand Server Switch Module Redundancy for the IBM BladeCenter
InfiniBand Switch Redundancy for the IBM BladeCenter
Redundant
InfiniBand Fabric
IBM BladeCenter
Port-1
Note
Server
Switch
Module-2
Port-2
Port-1
Port-2
HCA-1
HCA-14
InfiniBand Host
Blade 1
InfiniBand Host
Blade 14
183338
Server
Switch
Module-1
Connect the server switch modules to a redundant outside fabric.
The server switch modules do not connect to each other within the BladeCenter chassis. To connect the
modules to enable the Subnet Manager failover, connect the modules with an IB cable through the
external connectors. Before you connect the two server switch modules in your chassis, configure the
priority of the Subnet Managers on the modules. For more details about Subnet Manager redundancy,
see Chapter 5, “Subnet Manager Redundancy.” For more details about the High-Performance Subnet
Manager, see the Cisco High-Performance Subnet Manager for InfiniBand Server Switches.
After you configure your subnet manager priority, connect your server switch module to the external IB
fabric. By default, the external interfaces on your server switch module auto-negotiate speed with the
fabric.
Cisco SFS InfiniBand Redundancy Configuration Guide
3-2
OL-12957-02
CH A P T E R
4
Cisco SFS 3504 and Cisco SFS 3000 Series Server
Switch Redundancy
This chapter describes redundancy information about the Cisco SFS 3504 and Cisco SFS 3000 Series
Server Switches and includes the following sections:
•
Cisco SFS 3504 Server Switch Redundancy, page 4-2
•
Cisco SFS 3012R Server Switch Redundancy, page 4-5
•
Cisco SFS 3001 Server Switch Redundancy, page 4-9
The Cisco SFS 3000 Series includes the Cisco SFS 3001, Cisco SFS 3012R Server Switches. For
Ethernet gateway and Fibre Channel gateway redundancy related to the Cisco SFS 3504 and SFS 3000
Series Server Switches, see Chapter 7, “Ethernet Gateway and IPoIB Redundancies” and Chapter 8,
“Fibre Channel Gateway and SRP Redundancies.”
Note
For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and
Abbreviations.”
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
4-1
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3504 Server Switch Redundancy
Cisco SFS 3504 Server Switch Redundancy
This section describes the redundancies supported by the Cisco SFS 3504 Server Switch and includes
the following topics:
•
Software Redundancy, page 4-2
•
IB Switch Module Redundancy, page 4-2
•
Fabric Redundancy, page 4-2
•
AC Power-Fan Module Redundancy, page 4-3
•
Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy, page 4-4
For details about the Cisco SFS 3504 Server Switch, see the Cisco SFS 3504 Multifabric Server Switch
Hardware Installation Guide.
Software Redundancy
This section describes software redundancy in the Cisco SFS 3504 Server Switch.
With the Cisco SFS 3504 Server Switch, you can store two operating systems, one as active and one as
dormant. Use the A/B partitioning feature in one of the following two ways to achieve redundancy:
•
Both A and B partitions arrive from the factory preloaded with the same operating system software
on the ‘active’ and ‘dormant’ partitions. You may reload an existing operating system software or
install a new generation operating system on the ‘active’ operating system partition, thus leaving the
‘dormant’ partition with the factory-installed operating system.
•
Alternate operating system upgrades between the ‘active’ and ‘dormant’ partitions (first partition A,
then partition B, and then back to partition A), thereby enabling a rollback to the previous operating
system without having to reload or install software image files. This method optimizes the ease and
speed of switching operating systems and is similar to a dual boot scheme.
The advantage to using A/B partitions is that a recovery image is readily available. There is no need to
fix an image on a bad partition. Instead you use the CLI to boot the new or desired partition.
IB Switch Module Redundancy
This section describes IB switch module redundancy in the Cisco SFS 3504 Server Switch.
The IB switch card is not redundant. However, redundant IB links from a host can be set up by
connecting to a switch and using each alternate port, such as ports 1, 3, 5 and such. This spreads
connections across the multi-IC switches within the card.
Fabric Redundancy
This section describes fabric redundancy in the Cisco SFS 3504 Server Switch.
The user can configure redundancy with two Cisco SFS 3504 Server Switches. A server with a two-port
HCA can be attached to two IB switch modules on two separate Cisco SFS 3504 Server Switches (see
Figure 4-1).
Cisco SFS InfiniBand Redundancy Configuration Guide
4-2
OL-12957-02
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3504 Server Switch Redundancy
Figure 4-1
Redundancy with Two Cisco SFS 3504 Server Switches
SAN Fabric
Ethernet Fabric
SFS 3504-1
SFS 3504-2
FCGW-1
ENGW-1
FCGW-2
IB Switch Card-1
ENGW-2
IB Switch Card-2
IB Fabric
FCGW - Fibre Channel Gateway
ENGW - Ethernet Gateway
250311
InfiniBand Host
AC Power-Fan Module Redundancy
This section describes AC power-fan module redundancy in the Cisco SFS 3504 Server Switch.
The Cisco SFS 3504 Server Switch provides dual AC power-fan module redundancy (see Figure 4-2). If
one AC power-fan module fails, the other AC power-fan module assumes control immediately. The
redundant AC power-fan modules also support hot swaps. When the Cisco SFS 3504 Server Switch
includes only one AC power-fan module, you may add a second AC power-fan module while the chassis
is in operation. If you have two AC power-fan modules installed, you may remove either one of them
without removing power from the chassis. The AC power-fan modules are active-active.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
4-3
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3504 Server Switch Redundancy
Cisco SFS 3504 Dual Redundant Power-Fan Modules
250153
Figure 4-2
1
2
1-2
AC-DC power-fan modules
Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy
This section describes the Ethernet and Fibre Channel gateway redundancies in the Cisco SFS 3504
Server Switch.
Ethernet and Fibre Channel gateways connect the Cisco SFS 3504 Server Switch to IP and Fibre Channel
networks. There are four gateway slots in the chassis, and gateways can be installed in any of the slots
in any combination desired. All gateways have an IB connection to the IB switch. The gateway modules
can be configured to provide multiple-gateway redundancy if two or more Ethernet gateways or Fibre
Channel gateways are installed in one chassis. The gateways are hot pluggable, so you can add or swap
gateways while the chassis is in operation.
For Ethernet gateway and Fibre Channel gateway redundancy related to the Cisco SFS 3504 Server
Switch, see Chapter 7, “Ethernet Gateway and IPoIB Redundancies” and Chapter 8, “Fibre Channel
Gateway and SRP Redundancies.”
Cisco SFS InfiniBand Redundancy Configuration Guide
4-4
OL-12957-02
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3012R Server Switch Redundancy
Cisco SFS 3012R Server Switch Redundancy
This section describes the Cisco SFS 3012R Server Switch redundancies and includes the following
topics:
•
Power Supply Redundancy, page 4-5
•
Blower Redundancy, page 4-6
•
Controller Module Redundancy, page 4-6
•
InfiniBand Switch Module Redundancy, page 4-7
•
Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy, page 4-7
•
Fabric Redundancy, page 4-7
For details about the Cisco SFS 3012R Server Switch, see the Cisco SFS 3012R Multifabric Server
Switch Hardware Installation Guide.
Power Supply Redundancy
This section describes power supply redundancy in the Cisco SFS 3012R Server Switch.
The Cisco SFS 3012R Server Switch provides dual power supply redundancy (see Figure 4-3). If one
power supply fails, the other power supply assumes control immediately. The redundant power supplies
also support hot swaps. When the Cisco SFS 3012R Server Switch includes only one power supply, you
can add a second power supply while the chassis is in operation. If you have two power supplies
installed, you can remove either one of them without removing power from the chassis. The power
supplies are active-active.
Figure 4-3
Cisco SFS 3012R Server Switch Power Supply and Blower Modules
191985
Blower Modules
Power supply
modules
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
4-5
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3012R Server Switch Redundancy
Blower Redundancy
This section describes blower modules redundancy in the Cisco SFS 3012R Server Switch.
The two, hot-swappable blower modules of the Cisco SFS 3012R Server Switch maintain the internal
temperature in the chassis. Each module contains two blowers, so a fully functional system has four
operational blowers (see Figure 4-3). At least three blowers must be functional for continuous operation.
You do not need to turn off power to the Cisco SFS 3012R Server Switch to replace a blower module.
During replacement, when the Cisco SFS 3012R Server Switch is in operation, the blower can remain
empty for no more than three minutes.
Controller Module Redundancy
This section describes controller module redundancy in the Cisco SFS 3012R Server Switch.
Controller modules manage the Cisco SFS 3012R Server Switch and provide Ethernet and serial console
port access to the Cisco SFS 3012R Server Switch. Each Cisco SFS 3012R Server Switch contains two
controller modules. Figure 4-4 displays the Cisco SFS 3012R Server Switch controller modules with its
slot numbers.
Figure 4-4
Cisco SFS 3012R Server Switch Controller Module Side
1
3
5
7
9
11
13
15
2
4
6
8
10
12
14
180740
16
1
Slot 1—Default master controller module
14
Slot 14—Default standby controller module
2-13
Slots 2-13—Gateway slots
15-16
Slots 15, 16—IB switches
The Cisco SFS 3012R Server Switch contains an active controller and one active standby controller for
redundancy. Upon power-up, the controller in slot 1 becomes the master or active controller. The
controller in slot 14 becomes the standby controller. If the master controller fails or reboots, the standby
Cisco SFS InfiniBand Redundancy Configuration Guide
4-6
OL-12957-02
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3012R Server Switch Redundancy
controller automatically takes over as the new master. All I/O modules such as IB switch modules, Fibre
Channel gateways, and Ethernet gateways reboot when a failover occurs. During the software upgrade
and the installation process, both the master controller and the standby controller get upgraded at the
same time. A Subnet Manager runs on each controller module. The Subnet Manager can manage a single
Cisco SFS 3012R Server Switch or redundant Cisco SFS 3012R Server Switches. For more details on
Subnet Manager, see Chapter 4, “Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch
Redundancy.”
InfiniBand Switch Module Redundancy
This section describes IB switch module redundancy in the Cisco SFS 3012R Server Switch.
IB switch modules connect the Cisco SFS 3012R Server Switch to IB-attached hosts and other switches
in the IB network. The Cisco SFS 3012R Server Switch supports one or two IB switch modules in slots
15 and 16.
Note
The current software release requires an IB switch card with an operational CPU to remain installed in
slot 16 (see Figure 4-4).
The IB switch cards are not redundant. However, redundant IB links from a host can be set up by
connecting to a switch and using each alternate port, such as ports 1, 3, 5 and such. This spreads
connections across the multi-IC switches within the card.
Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy
This section describes Ethernet and Fibre Channel gateway redundancies in the Cisco SFS 3012R Server
Switch.
Ethernet and Fibre Channel gateways connect the Cisco SFS 3012R Server Switch to IP and Fibre
Channel networks. All gateways have an IB connection to each of the IB switches.
For Ethernet gateway and Fibre Channel gateway redundancy related to the Cisco SFS 3012R Server
Switch, see Chapter 7, “Ethernet Gateway and IPoIB Redundancies” and Chapter 8, “Fibre Channel
Gateway and SRP Redundancies.”
Fabric Redundancy
This section describes fabric redundancy in the Cisco SFS 3012R Server Switch.
The Cisco SFS 3012R Server Switch is designed for various types of redundancies within a single
chassis. Every component in the data path can be dual-connected. A server with a two-port HCA can be
dual attached to the two IB switch modules on the Cisco SFS 3012R Server Switch. Each gateway is also
dual attached to the dual switch module. In turn, the gateway modules are dual attached to Ethernet or
Fibre Channel networks in pairs. With the addition of centralized load balancing and port aggregation,
Ethernet and Fibre Channel uplink connections can be trunked and load balanced, allowing the
Cisco SFS 3012R Server Switch to reroute around failures. Each independently removable component
is hot-swappable.
Figure 4-5 shows the internal Cisco SFS 3012R Server Switch architecture. Components within the
dotted line reside within the Cisco SFS 3012R Server Switch chassis.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
4-7
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3012R Server Switch Redundancy
Figure 4-5
Cisco SFS 3012R Server Switch Redundant System Architecture
Ethernet Fabric
SAN Fabric
SFS 3012R
FCGW-1
FCGW-2
ENGW-1
IB Switch Card-1
ENGW-2
IB Switch Card-2
IB Fabric
FCGW - Fibre Channel Gateway
ENGW - Ethernet Gateway
250094
InfiniBand
Host
Cisco SFS InfiniBand Redundancy Configuration Guide
4-8
OL-12957-02
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3001 Server Switch Redundancy
Cisco SFS 3001 Server Switch Redundancy
This section describes the Cisco SFS 3001 Server Switch redundancies and includes the following
topics:
•
Power Supply Module Redundancy, page 4-9
•
Fan Redundancy, page 4-9
•
Fabric Redundancy, page 4-10
For more details about the Cisco SFS 3001 Server Switch, see the Cisco SFS 3001 Multifabric Server
Switch Hardware Guide.
Power Supply Module Redundancy
This section describes power supply module redundancy in the Cisco SFS 3001 Server Switch.
The Cisco SFS 3001 Server Switch provides two power supplies for 1:1 redundancy (see Figure 4-6). If
one power supply module fails, the other immediately assumes control. A power supply module can be
hot swapped without any disruption in power. The power supplies are active-active.
Figure 4-6
Cisco SFS 3001 Server Switch -- Front with Bezel Removed
Power supply
modules
191987
Fan tray
module
Fan Redundancy
This section describes fan redundancy in the Cisco SFS 3001 Server Switch.
A single Cisco SFS 3001 Server Switch provides fan redundancy. The Cisco SFS 3001 Server Switch
has a fan tray module with three individual fans (see Figure 4-6). The fans in the fan tray module provide
1:N redundancy. Two of the three fans in the fan module are required to cool the system and to keep it
operational.
Note
If more than one fan fails, or you want to hot swap the fan tray, it must be replaced within three minutes
after it is removed if the system is in operation.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
4-9
Chapter 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
Cisco SFS 3001 Server Switch Redundancy
Fabric Redundancy
This section describes fabric redundancy in the Cisco SFS 3001 Server Switch.
Cisco SFS 3001 Server Switches can be arranged in a dual chassis redundant configuration (see
Figure 4-7).
Figure 4-7
Cisco SFS 3001 Server Switches in a Dual-Chassis Redundant Configuration
Ethernet Fabric
or SAN Fabric
SFS 3001-1
SFS 3001-2
IB Fabric
191994
InfiniBand
Host
For redundancy, IB HCAs can be dual connected to a redundant pair of Cisco SFS 3001 Server Switches.
In an IB fabric that includes more than one Cisco SFS 3001 Server Switch, the Subnet Manager manages
the task of assigning a new master in the event of a switch failure. For more about Subnet Managers, see
Chapter 5, “Subnet Manager Redundancy.”
Cisco SFS InfiniBand Redundancy Configuration Guide
4-10
OL-12957-02
CH A P T E R
5
Subnet Manager Redundancy
This chapter describes Subnet Manager redundancy and includes the following sections:
Note
•
Embedded Subnet Manager
•
High-Performance Subnet Manager
•
Setting up Master and Standby Subnet Managers
•
Setting Up Database Synchronization
For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and
Abbreviations.”
Cisco Subnet Managers support redundancy as described in the IB specifications. There is a master
Subnet Manager and there are one or more standby Subnet Managers. In the event that something
happens to the master Subnet Manager, the next-in-line standby Subnet Manager assumes control of the
IB fabric.
There are two types of Cisco Subnet Managers. They are as follows:
•
Embedded Subnet Manager
•
High-Performance Subnet Manager
The Embedded Subnet Manager runs on a chassis, and the High-Performance Subnet Manager runs on
hosts. Both types of Subnet Managers support the IB standard master/standby failover between each
other.
The Cisco Subnet Managers have a proprietary database synchronization protocol that synchronizes
important data between the master Subnet Manager and one or more standby Subnet Managers. This
provides high-availability redundancy, enabling a database synchronized standby Subnet Manager to
assume control as master without disrupting the IB fabric.
Note
For redundancy, we recommend that you either have two Embedded Subnet Managers operating together
or two High-Performance Subnet Managers operating together. Database-synchronization is not
supported between the Embedded Subnet Manager and the High-Performance Subnet Manager, so
configuring an Embedded Subnet Manager and a High-Performance Subnet Manager could cause
disruption in data traffic in the case of a failure. For more information about database synchronization,
see the “Setting Up Database Synchronization” section on page 5-8.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
5-1
Chapter 5
Subnet Manager Redundancy
Figure 5-1 shows the IB fabric with the Embedded Subnet Manager and Figure 5-2 shows the IB fabric
with the High-Performance Subnet Manager. Figure 5-1 and Figure 5-2 show typical Subnet Manager
configurations. The user can set up different configurations as required for use.
InfiniBand Fabric and Embedded Subnet Manager
Ethernet
Fabric
SAN Fabric
SFS 3504-1
SFS 3504-2
SFS 7008P-1
SFS 7008P-2
Master Embedded
Subnet Manager
Standby Embedded
Subnet Manager
SFS 7000D-1
SFS 7000D-2
SFS 7000D-3
SFS 7000D-4
InfiniBand
Host-1
InfiniBand
Host-2
InfiniBand
Host-3
InfiniBand
Host-4
182853
Figure 5-1
Cisco SFS InfiniBand Redundancy Configuration Guide
5-2
OL-12957-02
Chapter 5
Subnet Manager Redundancy
Embedded Subnet Manager
InfiniBand Fabric and High-Performance Subnet Manager
Ethernet
Fabric
SAN Fabric
SFS 3504-1
SFS 3504-2
SFS 7008P-1
SFS 7000D-1
InfiniBand
Host-1
SFS 7008P-2
SFS 7000D-2
SFS 7000D-3
Master
HighPerformance
Subnet
Manager
Standby
HighPerformance
Subnet
Manager
InfiniBand
Host-2
InfiniBand
Host-3
SFS 7000D-4
InfiniBand
Host-4
182854
Figure 5-2
For more information about High-Performance Subnet Managers, see the Cisco High-Performance
Subnet Manager for InfiniBand Server Switches.
Embedded Subnet Manager
This section describes the Embedded Subnet Manager.
The Embedded Subnet Manager operates on the Cisco SFS 3504, the Cisco SFS 3000 series, the
Cisco SFS 7000 series, and the Cisco SFS 7008P Server Switches. When deployed in pairs, the
Embedded Subnet Manager prevents single points of failure at the system level. The Embedded Subnet
Manager is recommended for use in subnets of up to 1,000 nodes, when available. With the Embedded
Subnet Manager, a user can detect changes in large subnets within a short duration.
Note
The Cisco SFS 3012R Server Switch and the Cisco SFS 7008P Server Switch have a Subnet Manager
running on each controller card. There are two Subnet Managers in every chassis.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
5-3
Chapter 5
Subnet Manager Redundancy
High-Performance Subnet Manager
High-Performance Subnet Manager
This section describes the High-Performance Subnet Manager.
The High-Performance Subnet Manager is a standalone software package that centrally manages and
controls an IB subnet. It provides high availability when configured in an N+1 configuration and
provides high performance and scalability to large clusters as well.
For fabrics containing over 1,000 nodes, we recommend that you use the High-Performance Subnet
Manager. Although the Embedded Subnet Manager operates on the Cisco SFS 3504, the Cisco SFS 3000
series, the Cisco SFS 7000 series, and the Cisco SFS 7008P Server Switches, due to larger memory
capacity and faster CPU performance the High-Performance Subnet Manager scales more effectively in
larger fabrics.
The Cisco High-Performance Subnet Manager complements the Embedded Subnet Manager by off
loading the Subnet Manager function from the embedded processors on the IB switches.
The High-Performance Subnet Manager is also required in networking configurations of IB fabrics
where chassis types do not contain Embedded Subnet Managers.
Note
The Subnet Manager maintains optimal routing decisions. When a switch fails, the Subnet Manager is
notified through an in-band trap mechanism, and it resets by re-running the routing calculation for
subnets and reprogramming the routes.
Setting up Master and Standby Subnet Managers
This section describes how to set up the master and standby Subnet Managers and includes the following
topics:
Note
•
Setting up Master and Standby Subnet Managers Using Embedded Subnet Managers
•
Setting up Master and Standby Subnet Managers with High-Performance Subnet Managers
In the following sections, values are provided as examples only. We do not recommend that you use
non-default values.
Setting up Master and Standby Subnet Managers Using Embedded Subnet
Managers
This section describes how to set up master and standby Subnet Managers using the Embedded Subnet
Manager.
Typically Embedded Subnet Managers are set up with two chassis in the IB fabric and should be disabled
on all other chassis. The priority number of the Subnet Manager determines which is the master. We
recommend that you keep the priority of all Subnet Managers in a network equal, to ensure that when a
new Subnet Manager is added to the network, it does not take over mastership of the network.
Cisco SFS InfiniBand Redundancy Configuration Guide
5-4
OL-12957-02
Chapter 5
Subnet Manager Redundancy
Setting up Master and Standby Subnet Managers
To configure and verify the priority between Subnet Managers using CLI commands, perform the
following steps:
Step 1
Configure the priority between Subnet Managers to determine which is the master Subnet Manager and
which is the standby Subnet Manager.
Note
The priority range is between 0 and 15. The Subnet Manager that is the master, is the one that is
assigned the highest number in this range. The default priority number is 10. The
High-Performance Subnet Manager is assigned a higher priority than the Embedded Subnet
Manager to ensure that the High-Performance Subnet Manager takes preference over the
Embedded Subnet Manager if they are both present in the IB fabric at the same time.
The following example shows how to configure priority between Embedded Subnet Managers in two
Cisco SFS 3504 Server Switches:
SFS-3504# config
SFS-3504(config)# ib sm subnet-prefix fe:80:00:00:00:00:00:00 priority 12
SFS-3504(config)# exit
Step 2
Configure the master-poll-interval to set the time interval at which the master Subnet Manager is polled
to see whether it is active.
The following example shows how to configure the master-poll-interval to 5:
SFS-3504(config)# ib sm subnet-prefix fe:80:00:00:00:00:00:00 master-poll-intval 5
Note
Step 3
Decrease the master-poll-interval value to hasten the standby Subnet Manager takeover and
increase the value to slow it.
Configure the master-poll-retries to set the number of times it polls the master.
The following example shows how to configure the master-poll-retries to 0:
SFS-3504(config)# ib sm subnet-prefix fe:80:00:00:00:00:00:00 master-poll-retries 0
Note
Step 4
Decrease the master-poll-retries value to hasten the standby Subnet Manager takeover and
increase the value to slow it.
Verify the configuration.
The following example shows how to verify the Subnet Manager configuration, the master-poll-interval
value, and the master-poll-retries value:
SFS-3504# show ib sm configuration subnet-prefix all
================================================================================
Subnet Manager Information
================================================================================
subnet-prefix : fe:80:00:00:00:00:00:00
guid : 00:05:ad:00:00:01:0c:19
priority : 12
sm-key : 00:00:00:00:00:00:00:00
oper-status : master
act-count : 12938
sweep-interval(sec) : 10
response-timeout(msec) : 200
master-poll-intval(sec) : 5
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
5-5
Chapter 5
Subnet Manager Redundancy
Setting up Master and Standby Subnet Managers
master-poll-retries
max-active-sms
LID-mask-control
switch-life-time
switch-hoq-life-time
host-hoq-life-time
max-hops
mad-retries
node-timeout(sec)
wait-report-response
sa-mad-queue-depth
qos-admin-state
max-operational-v1
min-vl-cap-detected
:
:
:
:
:
:
:
:
:
:
:
:
:
:
0
0
0
18
18
18
64
5
10
false
256
disabled
auto-link
vl0-vl7
The output in this instance verifies that the operational status of this Subnet Manager is that of master,
the master-poll-interval is 5 seconds, and the master-poll-retries is 0.
Step 5
Verify that the Subnet Managers are present in the IB fabric.
The following is sample output from the show ib sm sm-info subnet-prefix command and shows how
to verify that the Subnet managers are present in the IB fabric:
SFS-3504# show ib sm sm-info subnet-prefix fe:80:00:00:00:00:00:00
================================================================================
Discovered Subnet Managers in Fabric
================================================================================
subnet-prefix : fe:80:00:00:00:00:00:00
port-guid : 00:05:ad:00:00:01:1d:20
priority : 0
sm-state : standby
sm-key : 00:00:00:00:00:00:00:00
act-count : 219
The sm-state in this instance shows that this standby Subnet Manager is discovered in the fabric.
Setting up Master and Standby Subnet Managers with High-Performance
Subnet Managers
This section describes how to set up Master and Standby Subnet Managers using the High-Performance
Subnet Managers.
Typically High-Performance Subnet Managers are set up with two hosts in the IB fabric and should be
disabled on all other chassis. The priority number of the High-Performance Subnet Manager determines
which is the master. We recommend that you keep the priority of all Subnet Managers in a network equal,
to ensure that when a new Subnet Manager is added to the network, it does not take over mastership of
the network.
To configure and verify the priority between Subnet Managers using CLI commands, perform the
following steps:
Step 1
Configure the priority between the High-Performance Subnet Managers to determine which is the master
Subnet Manager, and which is the standby Subnet Manager.
Cisco SFS InfiniBand Redundancy Configuration Guide
5-6
OL-12957-02
Chapter 5
Subnet Manager Redundancy
Setting up Master and Standby Subnet Managers
Note
The priority range is between 0 and 15. The Subnet Manager that is the master, is the one that is
assigned the highest number in this range. The default priority number is 11. The
High-Performance Subnet Manager is assigned a higher priority than the Embedded Subnet
Manager to ensure that the High-Performance Subnet Manager takes preference over the
Embedded Subnet Manager if they are both present in the IB fabric at the same time.
The following example shows how to configure priority between High-Performance Subnet Managers in
two hosts:
ib_sm> config priority 12
Step 2
Configure the master-poll-interval to set the time interval at which the master Subnet Manager is polled
to see whether it is active.
The following example shows how to configure the master-poll-interval to 5:
ib_sm> config master-poll-interval 5
Step 3
Configure the master-poll-retries to set the number of times it polls the master.
The following example shows how to configure the master-poll-retries to 0:
ib_sm> config master-poll-retries 0
Step 4
Verify the configuration.
The following is sample output from the show config command and shows how to verify the Subnet
Manager configuration, the master-poll-interval value, and the master-poll-retries value:
ib_sm> show config
================================================================================
Subnet Manager Configuration
================================================================================
subnet-prefix : fe:80:00:00:00:00:00:00
guid : 00:05:ad:00:00:01:0c:19
priority : 12
sm-key : 00:00:00:00:00:00:00:00
oper-status : master
act-count : 2923
sweep-interval(sec) : 10
response-timeout(msec) : 200
mad-retries : 5
node-timeout : 10
master-poll-interval(sec) : 5
master-poll-retries : 0
max-active-sms : 0
LID-mask-control : 0
switch-life-time : 18
sw-link-hoqlife : 18
ca-link-hoqlife : 18
max-hops : 64
wait-report-response : false
sa-mad-queue-depth : 256
local-node-retries : 10
qos-admin-state : disabled
max-operational-vl : default
min-vl-cap-detected : vl0-vl7
ib_sm>
The output in this instance verifies that the operational status of this Subnet Manager is that of master,
the master-poll-interval is 5 seconds, and the master-poll-retries is 0.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
5-7
Chapter 5
Subnet Manager Redundancy
Setting Up Database Synchronization
Step 5
Verify the High-Performance Subnet Managers on the IB fabric.
The following is sample output from the show other-sm command and shows how to verify the Subnet
Managers on the IB fabric:
ib_sm> show other-sm
================================================================================
Subnet Managers in the subnet
================================================================================
subnet-prefix : fe:80:00:00:00:00:00:00
port-guid : 00:05:ad:00:00:01:1d:20
sm-key : 00:00:00:00:00:00:00:00
priority : 0
sm-state : standby
act-count : 1133
ib_sm>
This instance shows that one standby Subnet Manager is discovered in the fabric.
Setting Up Database Synchronization
This section describes how to set up database synchronization and includes the following topics:
•
Setting up Database Synchronization for Embedded Subnet Managers, page 5-9
•
Setting up Database Synchronization for High-Performance Subnet Managers, page 5-10
Cisco Subnet Managers have a proprietary database synchronization protocol that synchronizes
important data between the master Subnet Manager and one or more standby Subnet Managers. This
provides high-availability redundancy, enabling a database synchronized standby Subnet Manager to
take over as master without disrupting the IB fabric. It is critical in large clusters with enterprise-class
IB fabrics and where MTBF is required to be minimal.
Note
We recommend that you keep the priority of all Subnet Managers in a network equal. Thus, when a new
Subnet Manager is added to the network, it enters as a standby and synchronizes itself to the master.
Cisco SFS InfiniBand Redundancy Configuration Guide
5-8
OL-12957-02
Chapter 5
Subnet Manager Redundancy
Setting Up Database Synchronization
Setting up Database Synchronization for Embedded Subnet Managers
To set up database synchronization configurations for the Embedded Subnet Manager, perform the
following steps:
Step 1
Verify that database synchronization is enabled (enable : true), and view the current configurations.
Note
By default the database synchronization feature is enabled.
The following is sample output from the show ib db-sync subnet-prefix command:
SFS-3504# show ib sm db-sync subnet-prefix fe:80:00:00:00:00:00:00
================================================================================
Subnet Manager Database Synchronization Information
================================================================================
subnet-prefix : fe:80:00:00:00:00:00:00
enable : true
max-dbsync-sms : 1
session-timeout(sec) : 10
poll-interval(sec) : 3
cold-sync-timeout(sec) : 10
cold-sync-limit : 2
cold-sync-period(sec) : 900
new-session-delay(sec) : 120
resync-interval(sec) : 3600
state : in-sync
SFS-3504#
Step 2
(Optional) Configure max-dbsync-sms to set the maximum number of standby Subnet Managers with
which the master Subnet Manager database can synchronize.
The following example shows how to set the max-dbsync-sms to 2:
SFS-3504(config)# ib sm db-sync subnet-prefix fe:80:00:00:00:00:00:00 max-dbsync-sms 2
Other values displayed in Step 1 under the Subnet Manager Database Synchronization Information can
similarly be configured by the user.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
5-9
Chapter 5
Subnet Manager Redundancy
Setting Up Database Synchronization
Step 3
Verify that the Subnet Managers are synchronized.
The following is sample output from the show ib sm db-sync subnet-prefix command and shows how
to view the standby Subnet Managers:
SFS-3504# show ib sm db-sync subnet-prefix fe:80:00:00:00:00:00:00 sm-list
================================================================================
DB Synchronizing SMs
================================================================================
subnet-prefix : fe:80:00:00:00:00:00:00
port-guid : 00:05:ad:00:00:01:1d:20
entry-state : active
session-state : active
session-timeout-current(sec) : 8
poll-interval-current(sec) : 1
new-session-delay-current(sec) : 120
resync-interval-current(sec) : 3589
state : in-sync
SFS-3504#
The display verifies that there is one standby Subnet Manager as listed.
Setting up Database Synchronization for High-Performance Subnet Managers
To set up database synchronization configurations for the High-Performance Subnet Manager, perform
the following steps:
Step 1
Verify that database synchronization is enabled (admin-state : enabled), and see the current
configurations.
Note
By default the database synchronization feature is enabled.
The following is sample output from the show db-sync command:
ib_sm> show db-sync
================================================================================
DB Sync Configuration and Status
================================================================================
protocol-version : 10
admin-state : enabled
max-dbsync-sms : 1
session-timeout(sec) : 10
poll-interval(sec) : 3
cold-sync-timeout(sec) : 10
cold-sync-limit : 2
cold-sync-period(sec) : 900
new-session-delay(sec) : 120
resync-interval(sec) : 3600
state : in-sync
ib_sm>
Cisco SFS InfiniBand Redundancy Configuration Guide
5-10
OL-12957-02
Chapter 5
Subnet Manager Redundancy
Setting Up Database Synchronization
Step 2
(Optional) Configure max-dbsync-sms to set the maximum number of standby Subnet Managers with
which the master Subnet Manager database can synchronize.
The following example shows how to set the max-dbsync-sms to 2:
ib_sm> config db-sync max-dbsync-sms 2
Other values displayed in Step 1 under the Database Synchronization Configuration and Status display
can similarly be configured by the user.
Step 3
Verify that the Subnet Managers are synchronized.
The following is sample output from the show db-sync sm-list command and shows how to list the
standby Subnet Managers:
ib_sm> show db-sync sm-list
================================================================================
DB Synchronizing SMs
================================================================================
port-guid : 00:05:ad:00:00:01:1d:20
entry-state : active
session-state : active
session-timeout-current(sec) : 8
poll-interval-current(sec) : 1
new-session-delay-current(sec) : 120
resync-interval-current(sec) : 3373
state : in-sync
ib_sm>
The display verifies that there is one standby Subnet Manager as listed.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
5-11
Chapter 5
Subnet Manager Redundancy
Setting Up Database Synchronization
Cisco SFS InfiniBand Redundancy Configuration Guide
5-12
OL-12957-02
CH A P T E R
6
Host Redundancy, and IPoIB and SRP
Redundancies
This chapter describes host redundancy, IPoIB redundancy, and SRP redundancy and includes the
following sections:
•
HCA Redundancy, page 6-1
•
IPoIB High Availability, page 6-4
•
OFED SRP High Availability, page 6-8
IPoIB and SRP are drivers that currently support redundancy.
Note
For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and
Abbreviations.”
HCA Redundancy
This section describes HCA redundancy and includes the following topics:
•
Single HCA Redundancy, page 6-1
•
Multiple HCA Redundancy, page 6-3
•
Two HCAs with the IBM BladeCenter, page 6-3
Single HCA Redundancy
This section describes how a single HCA may be configured to provide redundancy.
Single HCAs can each have two ports for redundancy within a single unit. Because such HCAs contain
two ports, port-to-port redundancy can be achieved with a single, dual-port HCA (see Figure 6-1). In
such cases, the HCA hardware and software drivers handle failovers between ports on the same HCA.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
6-1
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
HCA Redundancy
See the Cisco InfiniBand Host Channel Adapter Hardware Installation Guide for further details about
your HCA hardware installation.
Single HCA Redundancy with Dual Ports
InfiniBand Fabric 1
InfiniBand Fabric 2
SFS 7000D-1
SFS 7000D-2
InfiniBand Host
182855
Figure 6-1
Cisco SFS InfiniBand Redundancy Configuration Guide
6-2
OL-12957-02
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
HCA Redundancy
Multiple HCA Redundancy
This section describes how multiple HCAs may be configured to provide redundancy.
Multiple HCAs can be installed in a single host. This enables network traffic to failover from one HCA
to another HCA. For example, redundancy can be provided with two HCAs serving one host. Installing
two HCAs in one host is the minimum recommended configuration for a redundant IB fabric (see
Figure 6-2). Such a configuration provides an extra level of redundancy at the host level.
Two HCAs in a Single Host for Redundancy
InfiniBand Fabric 1
InfiniBand Fabric 2
SFS 7000D-1
SFS 7000D-2
Port 1
Port 2
HCA-1
HCA-2
InfiniBand Host
183267
Figure 6-2
Two HCAs with the IBM BladeCenter
For a description about this redundant configuration, see Chapter 3, “InfiniBand Server Switch Module
Redundancy for the IBM BladeCenter”.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
6-3
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
IPoIB High Availability
IPoIB High Availability
This section describes IPoIB high availability and includes the following topics:
Note
•
Cisco SFS IPoIB High Availability, page 6-4
•
OFED IPoIB High Availability, page 6-6
Every host that complies with the RFC-4391 IPoIB specification can use Ethernet gateway redundancies.
For more information about Ethernet gateway redundancies, see Chapter 7, “Ethernet Gateway and
IPoIB Redundancies.”
Cisco SFS IPoIB High Availability
This section describes IPoIB high availability and includes the following topics:
•
Merging Physical Ports
•
Unmerging Physical Ports
IPoIB supports active/passive port failover high availability between two or more ports. When you
enable the high availability feature, the ports on the HCA (for example, ib0 and ib1) merge into one
virtual port. If you configure high availability between the ports on the HCA(s), only one of the physical
ports passes traffic. The other ports are used as standby in the event of a failure.
For more details about the Cisco SFS host drivers, see the Cisco SFS InfiniBand Host Drivers User Guide
for Linux.
Merging Physical Ports
To configure IPoIB high availability on HCA ports in a Linux host, perform the following steps:
Step 1
Log in to your Linux host.
Step 2
Display the available interfaces by entering the ipoibcfg list command. The following example shows
how to configure IPoIB high availability between two ports on one HCA.
The following example shows how to display the available interfaces:
host1# /usr/local/topspin/sbin/ipoibcfg list
ib0 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/1, Active: InfiniHost0/1)
ib1 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/2, Active: InfiniHost0/2)
Step 3
Take the interfaces offline.
Note
You cannot merge interfaces until you take them offline.
The following example shows how to take the interfaces offline:
host1# ifconfig ib0 down
host1# ifconfig ib1 down
Cisco SFS InfiniBand Redundancy Configuration Guide
6-4
OL-12957-02
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
IPoIB High Availability
Step 4
Merge the two ports into one virtual IPoIB high availability port by entering the ipoibcfg merge
command with the IB identifiers of the first and the second IB ports on the HCA.
The following example shows how to merge the two ports into one virtual IPoIB high availability port:
host1# /usr/local/topspin/sbin/ipoibcfg merge ib0 ib1
Step 5
Display the available interfaces by entering the ipoibcfg list command.
The following example shows how to display the available interfaces:
host1# /usr/local/topspin/sbin/ipoibcfg list
ib0 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/1, Active: InfiniHost0/1)
Note
Step 6
The ib1 interface no longer appears, as it is merged with ib0.
Enable the interface by entering the ifconfig command with the appropriate port identifier ib# argument
and the up keyword.
The following example shows how to enable the interface with the ifconfig command:
host1# ifconfig ib0 up
Step 7
Assign an IP address to the merged port just as you would assign an IP address to a standard interface.
Unmerging Physical Ports
To unmerge physical ports and disable active-passive IPoIB high availability, perform the following
steps:
Step 1
Disable the IPoIB high availability interface that you want to unmerge by entering the ifconfig command
with the appropriate IB interface argument and the down argument.
The following example shows how to unmerge by disabling the IPoIB high availability interface:
host1# ifconfig ib0 down
Step 2
Unmerge the port by entering the ipoibcfg unmerge command with the identifier of the port that you
want to unmerge.
The following example shows how to unmerge the port:
host1# /usr/local/topspin/sbin/ipoibcfg unmerge ib0 ib1
Note
Step 3
After unmerging the port, ib1 no longer has an IP address and must be configured.
Display the available interfaces by entering the ipoibcfg list command.
The following example shows how to display the available interfaces:
host1# /usr/local/topspin/sbin/ipoibcfg list
ib0 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/1, Active: InfiniHost0/1)
ib1 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/2, Active: InfiniHost0/2)
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
6-5
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
IPoIB High Availability
Step 4
Enable the interfaces by entering the ifconfig command with the appropriate IB interface argument and
the up argument.
The following example shows how to enable the interfaces:
host1# ifconfig ib0 up
OFED IPoIB High Availability
This section describes the OFED IPoIB high availability and includes the following topics:
•
Configuring IPoIB High Availability
•
Verifying IPoIB High Availability
This section describes IPoIB high availability. IPoIB supports active/passive port failover high
availability between two or more ports. When you enable the high availability feature, the ports on the
HCA(s) (such as ib0 and ib1) bond into one virtual port. If you configure high availability between the
ports on the HCA(s), only one of the physical ports passes traffic. The other ports are used as standby in
the event of a failure.
IPoIB high availability is implemented through the IPoIB bonding driver. This driver is based on the
Linux Ethernet bonding driver and has been changed to work with IPoIB. The ib-bonding package
contains the bonding driver and a utility named ib-bond to manage and control the driver operation.
For more details about OFED host drivers, see the Cisco OpenFabrics Enterprise Distribution
InfiniBand Host Drivers User Guide for Linux.
Configuring IPoIB High Availability
To configure IPoIB high availability, perform the following steps:
Step 1
Remove the existing IP addresses from the interfaces.
The IP address from ib0 will be reassigned to the bonding interface.
The following example shows how to remove the existing IP addresses:
host1# ifconfig ib0 0.0.0.0
host1# ifconfig ib1 0.0.0.0
Step 2
Bond the two ports into one virtual IPoIB high availability port by using the ib-bond command.
The following example shows how to bond two ports into one virtual IPoIB high availability port in
verbose mode:
host1# ib-bond --bond-ip 192.168.0.1/24 --slaves ib0,ib1 -v
enslaving ib0
enslaving ib1
bonding is up: 192.168.0.1
bond0: 80:00:04:04:fe:80:00:00:00:00:00:00:00:05:ad:02:00:23:f0:d0 192.168.0.1/24
slave0: ib0 *
slave1: ib1
In the preceding output, ib0 * indicates that ib0 is the active interface, and ib1 is the passive interface.
Partition interfaces such as ib0.8002 can also be used with IPoIB high availability. In addition, /24 is the
subnet mask.
Cisco SFS InfiniBand Redundancy Configuration Guide
6-6
OL-12957-02
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
IPoIB High Availability
Step 3
(Optional) Enter the ifconfig command.
The following example shows how to enter the ifconfig command:
host1# ifconfig bond0
bond0
Link encap:InfiniBand HWaddr
80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::205:ad00:20:849/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1
RX packets:33523452 errors:0 dropped:0 overruns:0 frame:0
TX packets:165408699 errors:2 dropped:3 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:175570845580 (163.5 GiB) TX bytes:619840713192 (577.2 GiB)
The IPoIB high availability status information can be printed at any time with the ib-bond --status-all
command.
The following example shows how to print the IPoIB high availability status information:
host1# ib-bond --status-all
bond0: 80:00:04:04:fe:80:00:00:00:00:00:00:00:05:ad:02:00:23:f0:d0 192.168.0.1/24
slave0: ib0 *
slave1: ib1
The IPoIB high availability configuration can be removed with the ib-bond --stop-all command.
The following example shows how to remove the IPoIB high availability configuration:
host1# ib-bond --stop-all
IPoIB high availability interfaces that are configured manually are not persistent across reboots. You
must use the configuration file /etc/infiniband/openib.conf to configure IPoIB high availability when the
host boots. You must also remove any existing IPoIB boot-time configuration files such as ifcfg-ib0.
The following example shows the portion of openib.conf that must be edited to configure IPoIB high
availability at boot time:
# Enable the bonding driver on startup
IPOIBBOND_ENABLE=yes
# Set bond interface names
IPOIB_BONDS=bond0
# Set specific bond params; address and slaves
bond0_IP=192.168.0.1/24
bond0_SLAVES=ib0,ib1
The drivers can be restarted for the change to take effect without rebooting.
The following example shows how the drivers can be restarted:
host1# /etc/init.d/openibd restart
Unloading HCA driver:
Loading HCA driver and Access Layer:
Setting up InfiniBand network interfaces:
No configuration found for ib0
No configuration found for ib1
Setting up service network . . .
Setting up bonding interfaces:
Bringing up interface bond0
[
[
OK
OK
[
done
[
OK
]
]
]
]
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
6-7
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
OFED SRP High Availability
Verifying IPoIB High Availability
To force an IPoIB high availability failover while IPoIB traffic is running, perform the following steps:
Step 1
Start ping or Netperf between two IPoIB hosts.
For more details about how to start ping or Netperf between two IPoIB hosts, see the Cisco OpenFabrics
Enterprise Distribution InfiniBand Host Drivers User Guide for Linux.
Step 2
Remove the cable connected to ib0, either manually or by using the OS CLI or GUI. Print the ib-bond
--status-all command to verify the IPoIB high availability status.
The following example shows how to print the IPoIB high availability status:
host1# ib-bond --status-all
bond0: 80:00:04:04:fe:80:00:00:00:00:00:00:00:05:ad:02:00:23:f0:d0 192.168.0.1/24
slave0: ib0
slave1: ib1 *
The ib-bond --status-all command prints an asterix next to the primary interface. The primary interface
has now switched to ib1 as shown in the preceding example.
A kernel sys log message is also printed every time there is a failover.
host1# dmesg
bonding: bond0: link status definitely down for interface ib0, disabling it
bonding: bond0: making interface ib1 the new active one.
bonding: send gratuitous arp: bond bond0 slave ib1
Step 3
Note
Verify that the ping or Netperf continues with little or no interruption.
The Element Manager GUI can also be used to display port statistics, which is useful for watching a port
failover. For more information about the Element Manager GUI, see the Cisco SFS Product Family
Element Manager User Guide.
OFED SRP High Availability
This section describes how to configure SRP for use with Device Mapper Multipath, which is included
with both RHEL and SLES.
Device Mapper Multipath supports both active/active (load balancing and failover) and active/passive
(failover only) high availability, depending on the capability of the storage device. SRP should always
be used with multipathing software for high availability, to prevent data corruption and data loss. Other
third-party multipathing software can also be used with SRP, for configuration information. Consult the
relevant documentation for that software.
Device Mapper Multipath allows hosts to route I/O over the multiple paths available to an end storage
unit. A path refers to the connection from a host IB port to a storage controller port. When an active path
through which I/O happens fails, Device Mapper Multipath reroutes the I/O over other available paths.
In a Linux host, when there are multiple paths to a storage controller, each path appears as a separate
block device and hence results in multiple block devices for a single LUN. Device Mapper Multipath
creates a new multipath block device for those devices having the same LUN WWN. For example, a host
with two IB ports attached to a Cisco SFS 3012R Server Switch with two Fibre Channel port(s) attached
Cisco SFS InfiniBand Redundancy Configuration Guide
6-8
OL-12957-02
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
OFED SRP High Availability
to a storage controller, sees two block devices: /dev/sda and /dev/sdb, for example. Device Mapper
Multipath creates a single block device, /dev/mapper/360003ba27cf53000429f82b300016652, that
reroutes I/O through those two underlying block devices.
Device Mapper Multipath includes the following software components:
•
dm-multipath kernel module — routes I/O and does failover to paths
•
multipath configuration tool — provides commands to configure, list, and flush multipath devices
•
multipathd daemon — monitors paths to check if paths have failed or been fixed
Independent of storage device high availability capability, the Device Mapper Multipath provides
active/active high availability on host IB ports. The Cisco SFS Fibre Channel gateway similarly provides
active/active high availability between the SFS chassis and the Fibre Channel fabric.
To configure SRP high availability with Device Mapper Multipath, perform the following steps:
Step 1
Edit the file /etc/infiniband/openib.conf. Change SRPHA_ENABLE=no to SRPHA_ENABLE=yes. This
starts the srp_daemon program at boot time to create block devices for all paths to the SRP storage.
The srp_daemon program also handles dynamic storage reconfiguration, such as new storage being
added after the host is booted.
Note
Step 2
(Optional) Edit the file /etc/srp_daemon.conf to restrict SRP host driver access to a subset of available
SRP targets. By default, srp_daemon configures block devices for all SRP targets. The default
/etc/srp_daemon.conf file contains this information.
Note
Step 3
Both SRP_LOAD and SRPHA_ENABLE must be set to yes for SRP high availability to function
correctly.
For more details about the srp_daemon.conf file, see the Cisco OpenFabrics Enterprise
Distribution InfiniBand Host Drivers User Guide for Linux.
Edit the file /etc/multipath.conf. On RHEL4, the devnode_blacklist section (blacklist on RHEL5) should
be removed, commented out, or modified.
The following example shows the section of the file to be edited:
devnode_blacklist {
devnode "*"
}
(Optional) Change user_friendly_names yes to user_friendly_names no in /etc/multipath.conf on
RHEL as well. The friendly names are not consistent between different hosts and operating systems.
Note
On SLES, /etc/multipath.conf does not exist by default, and devnode_blacklist is not in effect.
On both RHEL and SLES, additional storage-specific configuration information may be required in
/etc/multipath.conf. Consult your storage device documentation for more details. For more information
on multipath.conf, consult the device-mapper-multipath package (RHEL) or multipath-tools (SLES).
Both packages have well-documented sample multipath.conf example files in /usr/share/doc.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
6-9
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
OFED SRP High Availability
Step 4
Configure Device Mapper Multipath to start at boot time.
The following example shows the command to enter on RHEL:
host1# chkconfig multipathd on
The following example shows the commands to enter on SLES:
host1# chkconfig boot.multipath on
host1# chkconfig multipathd on
Step 5
Reboot the Linux host.
After the reboot, multipath SRP devices should be accessible in /dev/mapper. Depending on the
configuration, it may take a few minutes after reboot for /dev/mapper to be fully populated.
The following example shows the output for the Fibre Channel gateway configuration when the IB host
has one HCA with both IB ports connected to the Server Fabric Switch:
host1# ls /dev/mapper
3600c0ff00000000007a6d11b6f245e00
3600c0ff00000000007a6d11b6f245e00p1
3600c0ff00000000007a6d11b6f245e00p2
3600c0ff00000000007a6d11b6f245e01
3600c0ff00000000007a6d11b6f245e02
3600c0ff00000000007a6d11b6f245e02p1
3600c0ff00000000007a6d11b6f245e02p2
3600c0ff00000000007a6d11b6f245e02p3
3600c0ff00000000007a6d11b6f245e03
3600c0ff00000000007a6d11b6f245e03p1
3600c0ff00000000007a6d11b6f245e03p2
3600c0ff00000000007a6d11b6f245e04
3600c0ff00000000007a6d11b6f245e05
3600c0ff00000000007a6d11b6f245e06
3600c0ff00000000007a6d11b6f245e06p1
control
Step 6
View the SCSI devices.
The following example shows how to view the SCSI devices:
host1# lsscsi
[0:0:0:0]
disk
[1:0:0:0]
disk
[1:0:0:1]
disk
[1:0:0:2]
disk
[1:0:0:3]
disk
[1:0:0:4]
disk
[1:0:0:5]
disk
[1:0:0:6]
disk
[2:0:0:0]
disk
[2:0:0:1]
disk
[2:0:0:2]
disk
[2:0:0:3]
disk
[2:0:0:4]
disk
[2:0:0:5]
disk
[2:0:0:6]
disk
Note
IBM-ESXS
SUN
SUN
SUN
SUN
SUN
SUN
SUN
SUN
SUN
SUN
SUN
SUN
SUN
SUN
MAY2036RC
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
StorEdge 3510
T107
327P
327P
327P
327P
327P
327P
327P
327P
327P
327P
327P
327P
327P
327P
/dev/sda
/dev/sdb
/dev/sdd
/dev/sdk
/dev/sdl
/dev/sdm
/dev/sdn
/dev/sdo
/dev/sdc
/dev/sde
/dev/sdf
/dev/sdg
/dev/sdh
/dev/sdi
/dev/sdj
The lsscsi command is supported by RHEL5 and SLES10 only. The lsscsi command is not
supported by RHEL4.
Cisco SFS InfiniBand Redundancy Configuration Guide
6-10
OL-12957-02
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
OFED SRP High Availability
Step 7
List the relationship between the SRP block devices and multipath devices by using the multipath -l
command.
The following example shows how to use the multipath -l command:
host1# multipath -l
3600c0ff00000000007a6d11b6f245e06dm-13 SUN,StorEdge 3510
[size=13G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 2:0:0:6 sdj 8:144 [active][undef]
\_ 1:0:0:6 sdo 8:224 [active][undef]
3600c0ff00000000007a6d11b6f245e05dm-12 SUN,StorEdge 3510
[size=15G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 2:0:0:5 sdi 8:128 [active][undef]
\_ 1:0:0:5 sdn 8:208 [active][undef]
3600c0ff00000000007a6d11b6f245e04dm-11 SUN,StorEdge 3510
[size=15G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 2:0:0:4 sdh 8:112 [active][undef]
\_ 1:0:0:4 sdm 8:192 [active][undef]
3600c0ff00000000007a6d11b6f245e03dm-3 SUN,StorEdge 3510
[size=15G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 2:0:0:3 sdg 8:96 [active][undef]
\_ 1:0:0:3 sdl 8:176 [active][undef]
3600c0ff00000000007a6d11b6f245e02dm-2 SUN,StorEdge 3510
[size=15G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 2:0:0:2 sdf 8:80 [active][undef]
\_ 1:0:0:2 sdk 8:160 [active][undef]
3600c0ff00000000007a6d11b6f245e01dm-1 SUN,StorEdge 3510
[size=15G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 2:0:0:1 sde 8:64 [active][undef]
\_ 1:0:0:1 sdd 8:48 [active][undef]
3600c0ff00000000007a6d11b6f245e00dm-0 SUN,StorEdge 3510
[size=15G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 2:0:0:0 sdc 8:32 [active][undef]
\_ 1:0:0:0 sdb 8:16 [active][undef]
In the preceding output, each multipath device corresponds to two SRP block devices, one on each of
the two attached host IB ports.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
6-11
Chapter 6
Host Redundancy, and IPoIB and SRP Redundancies
OFED SRP High Availability
Cisco SFS InfiniBand Redundancy Configuration Guide
6-12
OL-12957-02
CH A P T E R
7
Ethernet Gateway and IPoIB Redundancies
This chapter describes the Ethernet gateway redundancies and includes the following sections:
Note
•
Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch, page 7-3
•
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch, page 7-10
•
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch, page 7-23
For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and
Abbreviations.”
Ethernet gateway redundancy is based on the concept of redundancy group. Redundancy group is a
logical entity bridging an Ethernet VLAN to an IB partition, just like a bridge group, but in redundant
fashion. A redundancy group can contain one or more bridge groups located on the same or different
gateways. Gateways with bridge group members of the same redundancy group can be in the same or
different chassis.
A redundancy group can operate in two modes. The first one, which is the default mode, is
active-passive. In active-passive mode, only one bridge group is active and all others are in hot-standby
state. In case the active bridge group fails, another bridge group from the same redundancy group is
selected and activated. In active-active mode, all bridge groups are active and load balancing is enabled.
Load balancing allows the user to distribute the load of IB nodes among all bridge groups of the
redundancy group. In case of a bridge group failure, the load of the IB nodes is redistributed among the
remaining bridge groups.
Note
Even in an active-active mode, only one bridge group forwards the broadcast and multicast traffic. If the
bridge group forwarding broadcast and multicast fails, a new bridge group member of the same
redundancy group is selected to forward broadcast and multicast. The selection mechanism is identical
to the one used in active-passive mode to select the active bridge group.
A logical diagram of gateways in a redundant configuration is shown in Figure 7-1. Two or more
Ethernet gateways are configured to bridge between the Ethernet fabric and the IB fabric. The gateways
can be in the same or different chassis. The redundancy manager monitors the health of the gateways and
in case of failure elects a new primary bridge group. The redundancy manager is part of the SFS OS and
runs on the controller card of each chassis.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-1
Chapter 7
Figure 7-1
Ethernet Gateway and IPoIB Redundancies
Logical Diagram of Ethernet Gateways in a Redundant Configuration
Ethernet Fabric
Ethernet Gateway
Additional Ethernet
Ethernet Gateway gateways are
added as required
182641
InfiniBand Fabric
Different topologies work in similar ways but provide different levels of redundancy. For example, a
redundancy group with two bridge groups in the same chassis but different gateways protects against
gateway failure but does not provide chassis redundancy. Alternatively, a redundancy group with two
bridge groups on gateways in different chassis provides the same level of gateway redundancy but also
provides chassis redundancy. Examples are provided later in this chapter.
When a bridge group becomes a member of a redundancy group, some parameters are overwritten with
the corresponding parameters from the redundancy group. These parameters are restored when the
bridge group is removed from the redundancy group. Some examples of such parameters are broadcast
and multicast forwarding.
The examples in the sections that follow show the most popular deployment of different types of I/O
chassis. To simplify configuration, only one IP subnet is bridged (Data IP subnet). The Ethernet ports
are not VLAN tagged. The configuration on the Ethernet switch connected to the gateways determines
which VLAN is bridged. This VLAN is mapped to the default IB partition using the Ethernet gateway.
One IP subnet is allocated for in-band IB management (Management IP subnet). The in-band IB
management interface must be configured in order for the redundancy to work. This is true even in a
single chassis configuration.
Each bridge group must have an IP address assigned from the data subnet for the redundancy with
load-balancing (active-active mode) to work. The following addresses are assigned for the purpose of
the examples in this chapter:
Data Subnet: 10.0.0.0/8
Default Gateway (on the Ethernet switch): 10.0.0.1
IB Management Subnet: 192.168.0.0/8
Note
Every host that complies with the RFC-4391 IPoIB specification, can use Ethernet gateway
redundancies. For more information on IPoIB redundancies, see Chapter 6, “Host Redundancy, and
IPoIB and SRP Redundancies.”
Ethernet gateway uses gratuitous ARPs to redirect traffic from one gateway to another during failover,
fail-back, new member join/leave events and such. Because gratuitous ARPs are not guaranteed to reach
all network nodes, some ARP entries may become out of sync for a period of time up to the ARP cache
timeout. That is why ARP cache timeout on all network nodes must be set to the maximum acceptable
outage time
Cisco SFS InfiniBand Redundancy Configuration Guide
7-2
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch
Configuring Ethernet Gateway Redundancy with the
Cisco SFS 3504 Server Switch
This section describes how to configure redundant Ethernet gateways with the
Cisco SFS 3504 Server Switch to provide high availability redundancy and includes the following
topics:
•
Verifying Redundancy Configuration for Cisco SFS 3504 Server Switches, page 7-6
•
Verifying Bridge Group Configuration for Cisco SFS 3504 Server Switches, page 7-8
Two or more Cisco SFS 3504 Server Switches must be used to provide high availability redundancy. A
very typical deployment would consist of two Cisco SFS 3504 Server Switches with two Ethernet
gateways in each chassis (see Figure 7-2). The switches are configured with one bridge group for every
gateway and each bridge group having six Ethernet ports aggregated in single link aggregation groups
(trunk). All four bridge groups are in a single redundancy group.
Data subnet: 10.0.0.0/8
Management subnet: 192.168.0.0/24
Figure 7-2
Ethernet Gateway Redundancy with Dual Cisco SFS 3504 Server Switches
Ethernet Fabric
SFS 3504-1
Ethernet Gateway
SFS 3504-2
Ethernet Gateway
Ethernet Gateway
InfiniBand Switch
InfiniBand Switch
Ethernet Gateway
182642
InfiniBand Fabric
Note
For the purpose of the examples in the following sections, the Ethernet gateways are in slots numbers 1
and 2 for both Cisco SFS 3504 chassis.
To configure the first Cisco SFS 3504 Server Switch, perform the following steps:
Step 1
Enter configuration mode.
The following example shows how to enter configuration mode:
SFS-3504-1> enable
SFS-3504-1# configure terminal
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-3
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch
Step 2
Configure and connect an IB in-band management interface.
The IP address must be unique on each chassis.
Note
If an out-of-band Ethernet interface on the controller card is also configured, it must be on a
different IP subnet.
The following example shows how to configure an IB management interface:
SFS-3504-1(config)# interface mgmt-ib
SFS-3504-1(config)# ipaddress 192.168.0.1 255.255.255.0
SFS-3504-1(config-if-mgmt-ib)# no shutdown
SFS-3504-1(config-if-mgmt-ib)# exit
SFS-3504-1(config)#
Step 3
Create and configure two link aggregation groups (trunks) by assigning Ethernet ports to the link
aggregation groups.
The following example shows how to create and configure link aggregation groups by assigning Ethernet
ports:
SFS-3504-1(config)# interface trunk 1
SFS-3504-1(config-if-trunk)# enable
SFS-3504-1(config-if-trunk)# distribution-type src-dst-ip
SFS-3504-1(config-if-trunk)# interface ethernet 1/1-1/6
SFS-3504-1(config-if-ether-1/1-1/6)# trunk-group 1
SFS-3504-1(config-if-ether-1/1-1/6)# exit
SFS-3504-1(config)# interface trunk 2
SFS-3504-1(config-if-trunk)# enable
SFS-3504-1(config-if-trunk)# distribution-type src-dst-ip
SFS-3504-1(config-if-trunk)# interface ethernet 2/1-2/6
SFS-3504-1(config-if-ether-2/1-2/6)# trunk-group 2
SFS-3504-1(config-if-ether-2/1-2/6)# exit
SFS-3504-1(config)#
Step 4
Configure two bridge groups and assign ports to them.
Note
IP addresses must be from the data IP subnet and must be unique for each bridge group.
The following example shows how to configure bridge groups and assign ports to them:
SFS-3504-1(config)# bridge-group 1 subnet-prefix 10.0.0.0 8
SFS-3504-1(config)# bridge-group 1 ip-addr 10.0.0.101
SFS-3504-1(config)# bridge-group 1 ib-next-hop 10.0.0.1
SFS-3504-1(config)# interface trunk 1
SFS-3504-1(config-if-trunk)# bridge-group 1
SFS-3504-1(config-if-trunk)# interface gateway 1
SFS-3504-1(config-if-gw-1/2)# bridge-group 1
SFS-3504-1(config-if-gw-1/2)# exit
SFS-3504-1(config)# bridge-group 2 subnet-prefix 10.0.0.0 8
SFS-3504-1(config)# bridge-group 2 ip-addr 10.0.0.102
SFS-3504-1(config)# bridge-group 2 ib-next-hop 10.0.0.1
SFS-3504-1(config)# interface trunk 2
SFS-3504-1(config-if-trunk)# bridge-group 2
SFS-3504-1(config-if-trunk)# interface gateway 2
SFS-3504-1(config-if-gw-2/2)# bridge-group 2
SFS-3504-1(config-if-gw-2/2)# exit
SFS-3504-1(config)#
Cisco SFS InfiniBand Redundancy Configuration Guide
7-4
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch
Step 5
Configure a redundancy group and assign both bridge groups to it.
Note
The redundancy group ID must be the same in both chassis.
The following example shows how to configure a redundancy group and assign both bridge groups to it:
SFS-3504-1(config)# redundancy-group 1
SFS-3504-1(config)# bridge-group 1 redundancy-group 1
SFS-3504-1(config)# bridge-group 2 redundancy-group 1
Step 6
(Optional) Enable load balancing between bridge groups.
The following example shows how to enable load balancing:
SFS-3504-1(config)# redundancy-group 1 load-balancing
SFS-3504-1(config)# exit
To configure the second Cisco SFS 3504 Server Switch, perform the following steps:
Step 1
Enter configuration mode.
The following example shows how to enter configuration mode:
SFS-3504-2> enable
SFS-3504-2# configure terminal
Step 2
Configure and connect an IB in-band management interface
The IP address must be unique on each chassis.
Note
If an out-of-band Ethernet interface on the controller card is also configured, it must be on a
different IP subnet.
The following example shows how to configure an IB management interface:
SFS-3504-2(config)# interface mgmt-ib
SFS-3504-2(config-if-mgmt-ib)# ip address 192.168.0.2 255.255.255.0
SFS-3504-2(config-if-mgmt-ib)# no shutdown
SFS-3504-2(config-if-mgmt-ib)# exit
SFS-3504-2(config)#
Step 3
Create and configure two link aggregation groups (trunks) by assigning Ethernet ports to the link
aggregation groups.
The following example shows how to create and configure link aggregation groups:
SFS-3504-2(config)# interface trunk 3
SFS-3504-2(config-if-trunk)# enable
SFS-3504-2(config-if-trunk)# distribution-type src-dst-ip
SFS-3504-2(config-if-trunk)# interface ethernet 1/1-1/6
SFS-3504-2(config-if-ether-1/1-1/6)# trunk-group 3
SFS-3504-2(config-if-ether-1/1-1/6)# interface trunk 4
SFS-3504-2(config-if-trunk)# enable
SFS-3504-2(config-if-trunk)# distribution-type src-dst-ip
SFS-3504-2(config-if-trunk)# interface ethernet 2/1-2/6
SFS-3504-2(config-if-ether-2/1-2/6)# trunk-group 4
SFS-3504-2(config-if-ether-2/1-2/6)# exit
SFS-3504-2(config)#
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-5
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch
Step 4
Configure two bridge groups and assign ports to them.
Note
IP addresses must be from the data IP subnet and must be unique for each bridge group.
The following example shows how to configure bridge groups and assign ports to them:
SFS-3504-2(config)# bridge-group 3 subnet-prefix 10.0.0.0 8
SFS-3504-2(config)# bridge-group 3 ip-addr 10.0.0.103
SFS-3504-2(config)# bridge-group 3 ib-next-hop 10.0.0.1
SFS-3504-2(config)# interface trunk 3
SFS-3504-2(config-if-trunk)# bridge-group 3
SFS-3504-2(config-if-trunk)# interface gateway 1
SFS-3504-2(config-if-gw-1/2)# bridge-group 3
SFS-3504-2(config-if-gw-1/2)# exit
SFS-3504-2(config)# bridge-group 4 subnet-prefix 10.0.0.0 8
SFS-3504-2(config)# bridge-group 4 ip-addr 10.0.0.104
SFS-3504-2(config)# bridge-group 4 ib-next-hop 10.0.0.1
SFS-3504-2(config)# interface trunk 4
SFS-3504-2(config-if-trunk)# bridge-group 4
SFS-3504-2(config-if-trunk)# interface gateway 2
SFS-3504-2(config-if-gw-2/2)# bridge-group 4
SFS-3504-2(config-if-gw-2/2)# exit
SFS-3504-2(config)#
Step 5
Configure a redundancy group and assign both bridge groups to it.
Note
The redundancy group ID must be the same in both chassis.
The following example shows how to configure a redundancy group and assign both bridge groups to it:
SFS-3504-2(config)# redundancy-group 1
SFS-3504-2(config)# bridge-group 3 redundancy-group 1
SFS-3504-2(config)# bridge-group 4 redundancy-group 1
SFS-3504-2(config)#
Step 6
(Optional) Enable load balancing between bridge groups.
The following example shows how to enable load balancing:
SFS-3504-2(config)# redundancy-group 1 load-balancing
SFS-3504-2(config)# exit
Verifying Redundancy Configuration for Cisco SFS 3504 Server Switches
To verify the redundancy group configuration and status, use the show redundancy-group CLI
command. This command shows the redundancy group properties and all its members. It is important to
make sure the properties match the configuration and all members are reported. Redundancy groups must
be checked on both chassis.
Cisco SFS InfiniBand Redundancy Configuration Guide
7-6
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch
The following is sample output from show redundancy-group command for the first switch:
SFS-3504-1# show redundancy-group
================================================================================
Redundancy Groups
================================================================================
rlb-id : 1
name :
group-p_key : ff:ff
load-balancing : enabled
broadcast-forwarding : false
directed-broadcast : false
multicast : false
gratuitous-igmp : false
igmp-version : v2
num-members : 4
new-member-force-reelection : false
=================================================================================
Redundancy Group Members
================================================================================
bridge-group src-addr
last-receive
-------------------------------------------------------------------------------1
192.168.0.1
Sun Jan 4 00:27:31 1970
2
192.168.0.1
Sun Jan 4 00:27:31 1970
3
192.168.0.2
Sun Jan 4 00:27:31 1970
4
192.168.0.2
Sun Jan 4 00:27:31 1970
The following example shows how to use the show redundancy command for the second switch:
SFS-3504-2# show redundancy-group
================================================================================
Redundancy Groups
================================================================================
rlb-id : 1
name :
group-p_key : ff:ff
load-balancing : enabled
broadcast-forwarding : false
directed-broadcast : false
multicast : false
gratuitous-igmp : false
igmp-version : v2
num-members : 4
new-member-force-reelection : false
================================================================================
Redundancy Group Members
================================================================================
bridge-group src-addr
last-receive
-------------------------------------------------------------------------------1
192.168.0.1
Sun Jan 4 00:27:12 1970
2
192.168.0.1
Sun Jan 4 00:27:23 1970
3
192.168.0.2
Sun Jan 4 00:25:51 1970
4
192.168.0.2
Sun Jan 4 00:25:54 1970
SFS-3504-2#
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-7
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch
Verifying Bridge Group Configuration for Cisco SFS 3504 Server Switches
To check the bridge group configuration and status, use the show bridge-group CLI command. Check
both the Cisco SFS 3504 Server Switches, and make sure all the bridge groups are members of the same
redundancy group.
When a bridge group is a member of a redundancy group, most of the properties are inherited from the
redundancy group. Also make sure only one bridge group, across all chassis, is primary and the rest are
secondary. This is true even when load balancing is enabled (active-active mode). In active-passive
mode, only the primary bridge group is forwarding. All others are in hot standby state. In active-active
mode, all bridge groups are forwarding unicast traffic and only the primary is forwarding broadcast and
multicast if enabled.
The following is sample output from the show bridge-group command and shows how to verify the
bridge group configuration for the first switch:
SFS-3504-1# show bridge-group
================================================================================
Bridge Groups
================================================================================
bridge-group-id : 1
bridge-group-name :
ip-addr : 10.0.0.101
eth-bridge-port : trunk 1 (not tagged)
ib-bridge-port : 1/2(gw) (pkey: ff:ff)
broadcast-forwarding : false
broadcast-forwarding-mode : inherit-from-redundancy-group
directed-broadcast : false
directed-broadcast-mode : inherit-from-redundancy-group
loop-protection-method : one
multicast : false
multicast-mode : inherit-from-redundancy-group
gratuitous-igmp : false
gratuitous-igmp-mode : inherit-from-redundancy-group
igmp-version : v2
igmp-version-mode : inherit-from-redundancy-group
redundancy-group : 1
status-in-redundancy-group : primary
bridge-group-id
bridge-group-name
ip-addr
eth-bridge-port
ib-bridge-port
broadcast-forwarding
broadcast-forwarding-mode
directed-broadcast
directed-broadcast-mode
loop-protection-method
multicast
multicast-mode
gratuitous-igmp
gratuitous-igmp-mode
igmp-version
igmp-version-mode
redundancy-group
status-in-redundancy-group
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
2
10.0.0.102
trunk 2 (not tagged)
2/2(gw) (pkey: ff:ff)
false
inherit-from-redundancy-group
false
inherit-from-redundancy-group
one
false
inherit-from-redundancy-group
false
inherit-from-redundancy-group
v2
inherit-from-redundancy-group
1
secondary
Cisco SFS InfiniBand Redundancy Configuration Guide
7-8
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch
The following is sample output from the show bridge-group command that shows how to verify the
bridge group configuration for the second switch:
SFS-3504-2# show bridge-group
================================================================================
Bridge Groups
================================================================================
bridge-group-id: 3
bridge-group-name:
I-addr : 10.0.0.103
eth-bridge-port : trunk 3 (not tagged)
ib-bridge-port : 1/2(gw) (pkey: ff:ff)
broadcast-forwarding : false
broadcast-forwarding-mode : inherit-from-redundancy-group
directed-broadcast : false
directed-broadcast-mode : inherit-from-redundancy-group
loop-protection-method : one
multicast : false
multicast-mode : inherit-from-redundancy-group
gratuitous-igmp : false
gratuitous-igmp-mode : inherit-from-redundancy-group
igmp-version : v2
igmp-version-mode : inherit-from-redundancy-group
redundancy-group : 1
status-in-redundancy-group : secondary
bridge-group-id
bridge-group-name
ip-addr
eth-bridge-port
ib-bridge-port
broadcast-forwarding
broadcast-forwarding-mode
directed-broadcast
directed-broadcast-mode
loop-protection-method
multicast
multicast-mode
gratuitous-igmp
gratuitous-igmp-mode
igmp-version
igmp-version-mode
redundancy-group
status-in-redundancy-group
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
4
10.0.0.104
trunk 4 (not tagged)
2/2(gw) (pkey: ff:ff)
false
inherit-from-redundancy-group
false
inherit-from-redundancy-group
one
false
inherit-from-redundancy-group
false
inherit-from-redundancy-group
v2
inherit-from-redundancy-group
1
secondary
SFS-3504#
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-9
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Configuring Ethernet Gateway Redundancy Using the
Cisco SFS 3012R Server Switch
This section describes how to configure the Ethernet gateway redundancy using the Cisco SFS 3012R
Server Switch and includes the following topics:
•
Configuring Ethernet Gateway Redundancy Using a Single Cisco SFS 3012R Server Switch,
page 7-10
•
Configuring Ethernet Gateway Redundancy Using Dual Cisco SFS 3012R Server Switches,
page 7-15
The Cisco SFS 3012R Server Switch is a fully redundant chassis. It has twelve gateway slots, two
controller modules, and two switch cards. Each gateway is connected to both switch cards through the
backplane. A single Cisco SFS 3012R Server Switch is therefore capable of being configured for high
availability.
Configuring Ethernet Gateway Redundancy Using a Single Cisco SFS 3012R
Server Switch
This section describes how to configure Ethernet gateway redundancy using a single Cisco SFS 3012R
Server Switch and includes the following topics:
•
Verifying Redundancy Group Configuration for Cisco SFS 3001 Server Switches, page 7-27
•
Verifying Bridge Group Configuration for Cisco SFS 3001 Server Switches, page 7-28
The example in this section shows a typical single-switch configuration with four Ethernet gateways and
two switch cards. Two of the gateways, slots 2 and 4, are configured to use switch cards in slot 15 and
the other two gateways, slots 3 and 5, are configured to use switch cards in slot 16. (To locate the slot
numbers on the Cisco SFS 3012R Server Switch, see Figure 4-4.) Thus if any of the switch cards fail,
two gateways continue to remain operational. If the Ethernet ports of the gateways are required to be
connected to two different Ethernet switches, connect the gateways to the same IB switch card that are
connected to different Ethernet switches. Thus if any one Ethernet switch and any one IB switch card
were to fail, at least one gateway continues to remain operational.
Note
For the purpose of this example, gateways in slots 2 and 3 must be connected to different Ethernet
switches than gateways in slots 4 and 5.
The topology used in this example is shown in Figure 7-3.
Cisco SFS InfiniBand Redundancy Configuration Guide
7-10
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Figure 7-3
Ethernet Gateway Redundancy with Single Cisco SFS 3012R Server Switch
Ethernet Fabric
SFS 3012R
Ethernet Gateway 1
Ethernet Gateway 2
InfiniBand Switch 1
Ethernet Gateway 3
Ethernet Gateway 4
InfiniBand Switch 2
182639
InfiniBand Fabric
Once the redundancy groups are configured, a primary bridge is selected and forwarding is enabled.
To configure redundancy in a single Cisco SFS 3012R Server Switch, perform the following steps:
Step 1
Enter configuration mode.
The following example shows how to enter configuration mode:
SFS-3012R> enable
SFS-3012R# configure terminal
Step 2
Configure and connect an IB in-band management interface
Although there is only one chassis in this configuration, the interface must be configured and connected
in order for the redundancy to work.
Note
If an out-of-band Ethernet interface on the controller card is also configured, it must be on a
different IP subnet.
The following example shows how to configure and connect an IB in-band management interface:
SFS-3012R(config)# interface mgmt-ib
SFS-3012R(config-if-mgmt-ib)# 192.168.0.1 255.255.255.0
SFS-3012R(config-if-mgmt-ib)# no shutdown
SFS-3012R(config-if-mgmt-ib)# exit
Step 3
Create and configure all four link aggregation groups (trunks) by assigning Ethernet ports to each link
aggregation group.
The following example shows how to create and configure four link aggregation groups:
SFS-3012R(config)# interface trunk 1
SFS-3012R(config-if-trunk)# enable
SFS-3012R(config-if-trunk)# distribution-type src-dst-ip
SFS-3012R(config-if-trunk)# interface ethernet 2/1-2/6
SFS-3012R(config-if-ether-2/1-2/6)# trunk-group 1
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-11
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
SFS-3012R(config)# interface trunk 2
SFS-3012R(config-if-trunk)# enable
SFS-3012R(config-if-trunk)# distribution-type src-dst-ip
SFS-3012R(config-if-trunk)# interface ethernet 3/1-3/6
SFS-3012R(config-if-ether-3/1-3/6)# trunk-group 2
SFS-3012R(config)# interface trunk 3
SFS-3012R(config-if-trunk)# enable
SFS-3012R(config-if-trunk)# distribution-type src-dst-ip
SFS-3012R(config-if-trunk)# interface ethernet 4/1-4/6
SFS-3012R(config-if-ether-4/1-4/6)# trunk-group 3
SFS-3012R(config)# interface trunk 4
SFS-3012R(config-if-trunk)# enable
SFS-3012R(config-if-trunk)# distribution-type src-dst-ip
SFS-3012R(config-if-trunk)# interface ethernet 5/1-5/6
SFS-3012R(config-if-ether-5/1-5/6)# trunk-group 4
SFS-3012R(config-if-ether-5/1-5/6)# exit
Step 4
Configure all four bridge groups and assign ports to them.
Two of the bridge groups use IB port 1 to connect to the switch card in slot 15 and the other two bridge
groups use IB port 2 to connect to the switch card in slot 16.
Note
The IP address must be from the Data IP subnet and must be unique for each bridge group.
The following example shows how to configure bridge groups and assign ports to them:
SFS-3012R(config)# bridge-group 1 subnet-prefix 10.0.0.0 8
SFS-3012R(config)# bridge-group 1 ip-addr 10.0.0.101
SFS-3012R(config)# bridge-group 1 ib-next-hop 10.0.0.1
SFS-3012R(config)# interface trunk 1
SFS-3012R(config-if-trunk)# bridge-group 1
SFS-3012R(config)# interface gateway 2/1
SFS-3012R(config-if-gw-2/1)# bridge-group 1
SFS-3012R(config-if-gw-2/1)# exit
SFS-3012R(config)# bridge-group 2 subnet-prefix 10.0.0.0 8
SFS-3012R(config)# bridge-group 2 ip-addr 10.0.0.102
SFS-3012R(config)# bridge-group 2 ib-next-hop 10.0.0.1
SFS-3012R(config)# interface trunk 2
SFS-3012R(config-if-trunk)# bridge-group 2
SFS-3012R(config)# interface gateway 3/2
SFS-3012R(config-if-gw-3/2)# bridge-group 2
SFS-3012R(config-if-gw-3/2)# exit
SFS-3012R(config)# bridge-group 3 subnet-prefix 10.0.0.0 8
SFS-3012R(config)# bridge-group 3 ip-addr 10.0.0.103
SFS-3012R(config)# bridge-group 3 ib-next-hop 10.0.0.1
SFS-3012R(config)# interface trunk 3
SFS-3012R(config-if-trunk)# bridge-group 3
SFS-3012R(config)# interface gateway 4/1
SFS-3012R(config-if-gw-4/1)# bridge-group 3
SFS-3012R(config-if-gw-4/1)# exit
Cisco SFS InfiniBand Redundancy Configuration Guide
7-12
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
SFS-3012R(config)# bridge-group 4 subnet-prefix 10.0.0.0 8
SFS-3012R(config)# bridge-group 4 ip-addr 10.0.0.104
SFS-3012R(config)# bridge-group 4 ib-next-hop 10.0.0.1
SFS-3012R(config)# interface trunk 4
SFS-3012R(config-if-trunk)# bridge-group 4
SFS-3012R(config)# interface gateway 5/2
SFS-3012R(config-if-gw-5/2)# bridge-group 4
SFS-3012R(config-if-gw-5/2)# exit
Step 5
Configure a redundancy group, and assign all four bridge groups to it.
The following example shows how to configure a redundancy group and assign bridge groups to it:
SFS-3012R(config)#
SFS-3012R(config)#
SFS-3012R(config)#
SFS-3012R(config)#
SFS-3012R(config)#
Step 6
redundancy-group 1
bridge-group 1 redundancy-group
bridge-group 2 redundancy-group
bridge-group 3 redundancy-group
bridge-group 4 redundancy-group
1
1
1
1
(Optional) Enable load balancing between bridge groups.
The following example shows how to enable load balancing between bridge groups:
SFS-3012R(config)# redundancy-group 1 load-balancing
SFS-3012R(config)# exit
Verifying Redundancy Group Configuration for a Single Cisco SFS 3012R Server Switch
This section describes how to verify redundancy group configuration for a single Cisco SFS 3012R
Server Switch.
The following is sample output from the show redundancy-group command to check the redundancy
group configuration and status. This command shows the redundancy group properties and all the
members.
Note
Make sure the properties match the configuration and that all members are reported.
SFS-3012R# show redundancy-group
================================================================================
Redundancy Groups
================================================================================
rlb-id : 1
name :
group-p_key : ff:ff
load-balancing : enabled
broadcast-forwarding : false
multicast : false
gratuitous-igmp : false
igmp-version : v2
num-members : 4
new-member-force-reelection : false
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-13
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
================================================================================
Redundancy Group Members
================================================================================
bridge-group src-addr
last-receive
-------------------------------------------------------------------------------1
192.168.0.1
Thu Jan 1 01:44:48 1970
2
192.168.0.1
Thu Jan 1 01:44:47 1970
3
192.168.0.1
Thu Jan 1 00:02:12 1970
4
192.168.0.1
Thu Jan 1 00:03:04 1970
Verifying Bridge Group Configuration for a Single Cisco SFS 3012R Server Switch
This section describes how to verify bridge group configuration for a single Cisco SFS 3012R Server
Switch.
To check the bridge group configuration and status use the show bridge-group CLI command. Make
sure all bridge groups are members of the same redundancy group. When a bridge group is a member of
a redundancy group, most of the properties are inherited from the redundancy group. Also make sure
only one bridge group is primary and the rest are secondary. This is true even when load balancing is
enabled (active-active mode). In the active-passive mode, only the primary bridge group is forwarding
and all others are in hot standby state. In the active-active mode all bridge groups are forwarding unicast
traffic and only the primary bridge group is forwarding broadcast and multicast if enabled.
The following is sample output from the show bridge-group command, and it checks the bridge group
configuration and status.
SFS-3012R# show bridge-group
================================================================================
Bridge Groups
================================================================================
bridge-group-id : 1
bridge-group-name :
ip-addr : 10.0.0.101
eth-bridge-port : trunk 1 (not tagged)
ib-bridge-port : 2/1(gw) (pkey: ff:ff)
broadcast-forwarding : false
broadcast-forwarding-mode : inherit-from-redundancy-group
loop-protection-method : one
multicast : false
multicast-mode : inherit-from-redundancy-group
gratuitous-igmp : false
gratuitous-igmp-mode : inherit-from-redundancy-group
igmp-version : v2
igmp-version-mode : inherit-from-redundancy-group
redundancy-group : 1
status-in-redundancy-group : primary
bridge-group-id
bridge-group-name
ip-addr
eth-bridge-port
ib-bridge-port
broadcast-forwarding
broadcast-forwarding-mode
loop-protection-method
multicast
multicast-mode
gratuitous-igmp
gratuitous-igmp-mode
:
:
:
:
:
:
:
:
:
:
:
:
2
10.0.0.102
trunk 2 (not tagged)
3/2(gw) (pkey: ff:ff)
false
inherit-from-redundancy-group
one
false
inherit-from-redundancy-group
false
inherit-from-redundancy-group
Cisco SFS InfiniBand Redundancy Configuration Guide
7-14
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
igmp-version
igmp-version-mode
redundancy-group
status-in-redundancy-group
:
:
:
:
v2
inherit-from-redundancy-group
1
secondary
bridge-group-id
bridge-group-name
ip-addr
eth-bridge-port
ib-bridge-port
broadcast-forwarding
broadcast-forwarding-mode
loop-protection-method
multicast
multicast-mode
gratuitous-igmp
gratuitous-igmp-mode
igmp-version
igmp-version-mode
redundancy-group
status-in-redundancy-group
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
3
bridge-group-id
bridge-group-name
ip-addr
eth-bridge-port
ib-bridge-port
broadcast-forwarding
broadcast-forwarding-mode
loop-protection-method
multicast
multicast-mode
gratuitous-igmp
gratuitous-igmp-mode
igmp-version
igmp-version-mode
redundancy-group
status-in-redundancy-group
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
4
10.0.0.103
trunk 3 (not tagged)
4/1(gw) (pkey: ff:ff)
false
inherit-from-redundancy-group
one
false
inherit-from-redundancy-group
false
inherit-from-redundancy-group
v2
inherit-from-redundancy-group
1
secondary
10.0.0.104
trunk 4 (not tagged)
5/2(gw) (pkey: ff:ff)
false
inherit-from-redundancy-group
one
false
inherit-from-redundancy-group
false
inherit-from-redundancy-group
v2
inherit-from-redundancy-group
1
secondary
Configuring Ethernet Gateway Redundancy Using Dual Cisco SFS 3012R Server
Switches
This section describes how to configure a dual Cisco SFS 3012R Server Switch redundancy
configuration and includes the following topics:
•
Verifying Redundancy Group Configuration for Dual Cisco SFS 3012R Server Switches, page 7-19
•
Verifying Bridge Group Configuration for Dual Cisco SFS 3012R Server Switches, page 7-20
This typical example shows a dual Cisco SFS 3012R Server Switch setup with a total of four Ethernet
gateways that are available as two for each server switch. Both gateways in each chassis are configured
to use different switch cards. The gateway in slot 2 uses the switch card in slot 15 and the gateway in
slot 3 uses the switch card in slot 16. (To locate the slot numbers on the Cisco SFS 3012R Server Switch,
see Figure 4-4.) Thus if any of the switch cards fail, the other gateway continues to remain operational.
If the Ethernet ports of the gateways are connected to two different Ethernet switches, the gateways in
the same chassis must be connected to different Ethernet switches. This topology allows one chassis and
one Ethernet switch failure without interruption to the service. The topology used in this example is
shown in Figure 7-4.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-15
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Figure 7-4
Ethernet Gateway Redundancy with Dual Cisco SFS 3012R Server Switches
Ethernet Fabric
SFS 3012R-1
SFS 3012R-2
Ethernet Gateway 1
Ethernet Gateway 2
Ethernet Gateway 1
Ethernet Gateway 2
InfiniBand Switch 1
InfiniBand Switch 2
InfiniBand Switch 1
InfiniBand Switch 2
182640
InfiniBand Fabric
Once redundancy group is configured, a primary bridge group is elected and forwarding is enabled.
To configure the first Cisco SFS 3012R Server Switch, perform the following steps:
Step 1
Enter configuration mode.
The following example shows how to enter configuration mode:
SFS-3012R> enable
SFS-3012R# configure terminal
Step 2
Configure and connect the IB in-band management interface.
The IP address must be unique on each chassis.
Note
If the out-of-band Ethernet interface on the controller card is also configured, it must be on a
different IP subnet.
The following example shows how to configure and connect the IB in-band management interface:
SFS-3012R(config)# interface mgmt-ib
SFS-3012R(config-if-mgmt-ib)# ip address 192.168.0.1 255.255.255.0
SFS-3012R(config-if-mgmt-ib)# no shutdown
SFS-3012R(config-if-mgmt-ib)# exit
Step 3
Create and configure two link aggregation groups (trunks). Assign Ethernet ports to the link aggregation
groups.
The following example shows how to create and configure link aggregation groups:
SFS-3012R(config)# interface trunk 1
SFS-3012R(config-if-trunk)# enable
SFS-3012R(config-if-trunk)# distribution-type src-dst-ip
SFS-3012R(config-if-trunk)# interface ethernet 2/1-2/6
SFS-3012R(configif-ether-2/1-2/6)# trunk-group 1
Cisco SFS InfiniBand Redundancy Configuration Guide
7-16
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
SFS-3012R(config)# interface trunk 2
SFS-3012R(config-if-trunk)# enable
SFS-3012R(config-if-trunk)# distribution-type src-dst-ip
SFS-3012R(config-if-trunk)# interface ethernet 3/1-3/6
SFS-3012R(configif-ether-3/1-3/6)# trunk-group 2
SFS-3012R(configif-ether-3/1-3/6)# exit
Step 4
Configure two bridge groups and assign ports to them.
One of the bridge groups uses the IB port 1 to connect to the switch card in slot 15 and the other bridge
group uses the IB port 2 to connect to the switch card in slot 16.
Note
IP addresses must be from Data IP subnet and must be unique for each bridge group.
The following example shows how to configure two bridge groups and assign ports to them:
SFS-3012R(config)# bridge-group 1 subnet-prefix 10.0.0.0 8
SFS-3012R(config)# bridge-group 1 ip-addr 10.0.0.101
SFS-3012R(config)# bridge-group 1 ib-next-hop 10.0.0.1
SFS-3012R(config)# interface trunk 1
SFS-3012R(config-if-trunk)# bridge-group 1
SFS-3012R(config)# interface gateway 2/1
SFS-3012R(config-if-gw-2/2)# bridge-group 1
SFS-3012R(config-if-gw-2/2)# exit
SFS-3012R(config)# bridge-group 2 subnet-prefix 10.0.0.0 8
SFS-3012R(config)# bridge-group 2 ip-addr 10.0.0.102
SFS-3012R(config)# bridge-group 2 ib-next-hop 10.0.0.1
SFS-3012R(config)# interface trunk 2
SFS-3012R(config-if-trunk)# bridge-group 2
SFS-3012R(config)# interface gateway 3/2
SFS-3012R(config-if-gw-3/2)# bridge-group 2
SFS-3012R(config-if-gw-3/2)# exit
Step 5
Configure the redundancy group and assign both bridge groups to it.
Note
The redundancy group ID must be the same in both chassis.
The following example shows how to configure the redundancy group and assign bridge groups to it:
SFS-3012R(config)# redundancy-group 1
SFS-3012R(config)# bridge-group 1 redundancy-group 1
SFS-3012R(config)# bridge-group 2 redundancy-group 1
Step 6
(Optional) Enable load balancing between bridge groups.
The following example shows how to enable load balancing between bridge groups:
SFS-3012R(config)# redundancy-group 1 load-balancing
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-17
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
To configure the second Cisco SFS 3012R Server Switch, perform the following steps:
Step 1
Enter configuration mode.
The following example shows how to enable configuration mode:
SFS-3012R> enable
SFS-3012R# configure terminal
Step 2
Configure and connect the IB in-band management interface.
The IP address must be unique on each chassis.
Note
If the out-of-band Ethernet interface on the controller card is also configured it must be on a
different IP subnet.
The following example shows how to configure and connect the IB in-band management interface:
SFS-3012R(config)# interface mgmt-ib
SFS-3012R(config-if-mgmt-ib)# ip address 192.168.0.2 255.255.255.0
SFS-3012R(config-if-mgmt-ib)# no shutdown
SFS-3012R(config-if-mgmt-ib)# exit
Step 3
Create and configure two link aggregation groups (trunks). Assign Ethernet ports to link aggregation
groups.
The following example shows how to create and configure two link aggregation groups:
SFS-3012R(config)# interface trunk 3
SFS-3012R(config-if-trunk)# enable
SFS-3012R(config-if-trunk)# distribution-type src-dst-ip
SFS-3012R(config-if-trunk)# interface ethernet 2/1-2/6
SFS-3012R(config-if-ether-2/1-2/6)# trunk-group 3
SFS-3012R(config)# interface trunk 4
SFS-3012R(config-if-trunk)# enable
SFS-3012R(config-if-trunk)# distribution-type src-dst-ip
SFS-3012R(config-if-trunk)# interface ethernet 3/1-3/6
SFS-3012R(config-if-ether-3/1-3/6)# trunk-group 4
SFS-3012R(config-if-ether-3/1-3/6)# exit
Step 4
Configure two bridge groups and assign ports to them.
One of the bridge groups uses the IB port 1 to connect to the switch card in slot 15, and the other bridge
group uses the IB port 2 to connect to the switch card in slot 16.
Note
IP addresses must be from Data IP subnet and must be unique for each bridge group.
The following example shows how to configure two bridge groups and assign ports to them:
SFS-3012R(config)# bridge-group 3 subnet-prefix 10.0.0.0 8
SFS-3012R(config)# bridge-group 3 ip-addr 10.0.0.103
SFS-3012R(config)# bridge-group 3 ib-next-hop 10.0.0.1
SFS-3012R(config)# interface trunk 3
SFS-3012R(config-if-trunk)# bridge-group 3
SFS-3012R(config)# interface gateway 2/1
SFS-3012R(config-if-gw-2/1)# bridge-group 3
SFS-3012R(config-if-gw-2/1)# exit
Cisco SFS InfiniBand Redundancy Configuration Guide
7-18
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
SFS-3012R(config)# bridge-group 4 subnet-prefix 10.0.0.0 8
SFS-3012R(config)# bridge-group 4 ip-addr 10.0.0.104
SFS-3012R(config)# bridge-group 4 ib-next-hop 10.0.0.1
SFS-3012R(config)# interface trunk 4
SFS-3012R(config-if-trunk)# bridge-group 4
SFS-3012R(config)# interface gateway 3/2
SFS-3012R(config-if-gw-3/2)# bridge-group 4
SFS-3012R(config-if-gw-3/2)# exit
Step 5
Configure a redundancy group and assign both bridge groups to it.
Note
The redundancy group ID must be the same in both chassis.
The following example shows how to configure a redundancy group and assign both bridge groups to it:
SFS-3012R(config)# redundancy-group 1
SFS-3012R(config)# bridge-group 3 redundancy-group 1
SFS-3012R(config)# bridge-group 4 redundancy-group 1
Step 6
(Optional) Enable load balancing between bridge groups.
The following example shows how to enable load balancing between bridge groups:
SFS-3012R(config)# redundancy-group 1 load-balancing
SFS-3012R(config)# exit
Verifying Redundancy Group Configuration for Dual Cisco SFS 3012R Server Switches
This section describes how to verify redundancy group configuration for the Cisco SFS 3012R Server
Switch.
To check redundancy group configuration and status use the show redundancy-group CLI command.
This command shows redundancy group properties and all members. It is important to make sure the
properties match the configuration and all members are reported. The redundancy group must be checked
on both chassis.
The following is sample output from the show redundancy-group command for the first switch:
SFS-3012R# show redundancy-group
================================================================================
Redundancy Groups
================================================================================
rlb-id : 1
name :
group-p_key : ff:ff
load-balancing : enabled
broadcast-forwarding : false
multicast : false
gratuitous-igmp : false
igmp-version : v2
num-members : 4
new-member-force-reelection : false
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-19
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
================================================================================
Redundancy Group Members
================================================================================
bridge-group src-addr
last-receive
-------------------------------------------------------------------------------1
192.168.0.1
Thu Jan 1 00:03:44 1970
2
192.168.0.1
Thu Jan 1 00:03:44 1970
3
192.168.0.2
Thu Jan 1 00:04:04 1970
4
192.168.0.2
Thu Jan 1 00:04:03 1970
The following is sample output from the show redundancy-group command for the next switch:
SFS-3012R# show redundancy-group
================================================================================
Redundancy Groups
================================================================================
rlb-id : 1
name :
group-p_key : ff:ff
load-balancing : enabled
broadcast-forwarding : false
multicast : false
gratuitous-igmp : false
igmp-version : v2
num-members : 4
new-member-force-reelection : false
================================================================================
Redundancy Group Members
================================================================================
bridge-group src-addr
last-receive
-------------------------------------------------------------------------------1
192.168.0.1
Thu Jan 1 00:03:44 1970
2
192.168.0.1
Thu Jan 1 00:03:44 1970
3
192.168.0.2
Thu Jan 1 00:04:04 1970
4
192.168.0.2
Thu Jan 1 00:04:03 1970
Verifying Bridge Group Configuration for Dual Cisco SFS 3012R Server Switches
This section describes how to verify bridge group configuration for the Cisco SFS 3012R Server Switch.
To check bridge group configuration and status use the show bridge-group CLI command. Inspect both
chassis and confirm all bridge groups are members of the same redundancy group. When a bridge group
is a member of a redundancy group, most of the properties are inherited from the redundancy group. Also
make sure only one bridge group, across all chassis, is primary and the rest are secondary. This is true
even when load balancing is enabled (active-active mode). In active-passive mode only the primary
bridge group is forwarding. All others are in hot standby state. In active-active mode all bridge groups
are forwarding unicast traffic and only primary is forwarding broadcast and multicast if enabled.
Cisco SFS InfiniBand Redundancy Configuration Guide
7-20
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Step 1
Verify the bridge group configuration for the first Cisco SFS 3012R Server Switch.
The following is sample output from the show bridge-group command that shows how to verify the
bridge group configuration for the first Cisco SFS 3012R Server Switch:
SFS-3012R# show bridge-group
================================================================================
Bridge Groups
================================================================================
bridge-group-id : 1
bridge-group-name :
ip-addr : 10.0.0.101
eth-bridge-port : trunk 1 (not tagged)
ib-bridge-port : 2/1(gw) (pkey: ff:ff)
broadcast-forwarding : false
broadcast-forwarding-mode : inherit-from-redundancy-group
loop-protection-method : one
multicast : false
multicast-mode : inherit-from-redundancy-group
gratuitous-igmp : false
gratuitous-igmp-mode : inherit-from-redundancy-group
igmp-version : v2
igmp-version-mode : inherit-from-redundancy-group
redundancy-group : 1
status-in-redundancy-group : primary
bridge-group-id
bridge-group-name
ip-addr
eth-bridge-port
ib-bridge-port
broadcast-forwarding
broadcast-forwarding-mode
loop-protection-method
multicast
multicast-mode
gratuitous-igmp
gratuitous-igmp-mode
igmp-version
igmp-version-mode
redundancy-group
status-in-redundancy-group
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
2
10.0.0.102
trunk 2 (not tagged)
3/2(gw) (pkey: ff:ff)
false
inherit-from-redundancy-group
one
false
inherit-from-redundancy-group
false
inherit-from-redundancy-group
v2
inherit-from-redundancy-group
1
secondary
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-21
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Step 2
Verify the bridge group configuration for the second Cisco SFS 3012R Server Switch.
The following is sample output from the show bridge-group command that shows how to verify the
bridge group configuration for the second Cisco SFS 3012R Server Switch:
SFS-3012R# show bridge-group
================================================================================
Bridge Groups
================================================================================
bridge-group-id : 3
bridge-group-name :
ip-addr : 10.0.0.103
eth-bridge-port : trunk 3 (not tagged)
ib-bridge-port : 2/1(gw) (pkey: ff:ff)
broadcast-forwarding : false
broadcast-forwarding-mode : inherit-from-redundancy-group
loop-protection-method : one
multicast : false
multicast-mode : inherit-from-redundancy-group
gratuitous-igmp : false
gratuitous-igmp-mode : inherit-from-redundancy-group
igmp-version : v2
igmp-version-mode : inherit-from-redundancy-group
redundancy-group : 1
status-in-redundancy-group : secondary
bridge-group-id
bridge-group-name
ip-addr
eth-bridge-port
ib-bridge-port
broadcast-forwarding
broadcast-forwarding-mode
loop-protection-method
multicast
multicast-mode
gratuitous-igmp
gratuitous-igmp-mode
igmp-version
igmp-version-mode
redundancy-group
status-in-redundancy-group
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
4
10.0.0.104
trunk 4 (not tagged)
3/2(gw) (pkey: ff:ff)
false
inherit-from-redundancy-group
one
false
inherit-from-redundancy-group
false
inherit-from-redundancy-group
v2
inherit-from-redundancy-group
1
secondary
Cisco SFS InfiniBand Redundancy Configuration Guide
7-22
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch
Configuring Ethernet Gateway Redundancy for the
Cisco SFS 3001 Server Switch
This section describes how to configure Ethernet gateway redundancy for two Cisco SFS 3001 Server
Switches. and includes the following topics:
•
Verifying Redundancy Group Configuration for Cisco SFS 3001 Server Switches, page 7-27
•
Verifying Bridge Group Configuration for Cisco SFS 3001 Server Switches, page 7-28
Two or more Cisco SFS 3001 Server Switches must be used to provide high availability redundancy. A
very typical deployment consists of two Cisco SFS 3001 Server Switches with one Ethernet gateway in
each (see Figure 7-5).
Figure 7-5
Ethernet Redundancy with Dual Cisco SFS 3001 Server Switches
Ethernet Fabric
SFS 3001-1
SFS 3001-2
Ethernet Gateway
Ethernet Gateway
InfiniBand Switch
InfiniBand Switch
182638
InfiniBand Fabric
A single Cisco SFS 3001 Server Switch provides power supply redundancy only. A single
Cisco SFS 3001 Server Switch does not provide Ethernet gateway redundancy, because it contains a
single gateway slot. For more on the Cisco SFS 3001 redundancy, see Chapter 4, “Cisco SFS 3504 and
Cisco SFS 3000 Series Server Switch Redundancy.”
For each Cisco SFS 3001 Server Switch, once the redundancy groups are configured, a primary bridge
is selected and forwarding is enabled.
To configure the first Cisco SFS 3001 Server Switch, perform the following steps:
Step 1
Enter configuration mode.
The following example shows how to enter configuration mode:
SFS-3001> enable
SFS-3001# configure terminal
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-23
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch
Step 2
Configure and connect an IB in-band management interface.
The IP address must be unique on each chassis.
Note
If an out-of-band Ethernet interface on the controller card is also configured, it must be on a
different IP subnet.
The following example shows how to configure an IB management interface:
SFS-3001(config)# interface mgmt-ib
SFS-3001(config-if-mgmt-ib)# ip address 192.168.0.1 255.255.255.0
SFS-3001(config-if-mgmt-ib)# no shutdown
SFS-3001(config-if-mgmt-ib)# exit
Step 3
Create and configure a link aggregation group (trunk) by assigning Ethernet ports to the link aggregation
group.
The following example shows how to create and configure a link aggregation group by assigning
Ethernet ports:
SFS-3001(config)# interface trunk 1
SFS-3001(config-if-trunk)# enable
SFS-3001(config-if-trunk)# distribution-type src-dst-ip
SFS-3001(config-if-trunk)# interface ethernet 2/1-2/6
SFS-3001(config-if-ether-2/1-2/6)# trunk-group 1
SFS-3001(config-if-ether-2/1-2/6)# exit
Step 4
Configure a bridge group and assign ports to it.
Note
The IP address must be from the data IP subnet and must be unique for each bridge group.
The following example shows how to configure a bridge group and assign ports:
SFS-3001(config)# bridge-group 1 subnet-prefix 10.0.0.0 8
SFS-3001(config)# bridge-group 1 ip-addr 10.0.0.101
SFS-3001(config)# bridge-group 1 ib-next-hop 10.0.0.1
SFS-3001(config)# interface trunk 1
SFS-3001(config-if-trunk)#bridge-group 1
SFS-3001(config)# interface gateway 2
SFS-3001(config-if-gw-2/2)# bridge-group 1
SFS-3001(config-if-gw-2/2)# exit
Step 5
Configure a redundancy group, and assign a bridge group to it.
Note
The redundancy group ID must be the same in all chassis.
The following example shows how to configure a redundancy group and assign a bridge group.
SFS-3001(config)# redundancy-group 1
SFS-3001(config)# bridge-group 1 redundancy-group 1
Step 6
(Optional) Enable load balancing between bridge groups.
The following example shows how to enable load balancing between bridge groups:
SFS-3001(config)# redundancy-group 1 load-balancing
SFS-3001(config)#
Cisco SFS InfiniBand Redundancy Configuration Guide
7-24
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch
Step 7
(Optional) Change the redundancy group parameters.
Note
Enable multicast forwarding on the redundancy group and not on individual bridge groups.
The following example shows how to change the redundancy group parameters:
SFS-3001(config)# redundancy-group 1 multicast
SFS-3001(config)#
To configure the next Cisco SFS 3001 Server Switch, perform the following steps:
Note
Step 1
While configuring the bridge group IDs, it is good practice to maintain unique bridge group IDs even if
it is not required.
Enter configuration mode.
The following example shows how to enter configuration mode:
SFS-3001> enable
SFS-3001# configure terminal
Step 2
Configure and connect an IB in-band management interface.
The IP address must be unique on each chassis.
Note
If an out-of-band Ethernet interface on the controller card is also configured, it must be on a
different IP subnet.
The following example shows how to configure IB management interface:
SFS-3001(config)# interface mgmt-ib
SFS-3001(config-if-mgmt-ib)# ip address 192.168.0.2 255.255.255.0
SFS-3001(config-if-mgmt-ib)# no shutdown
SFS-3001(config-if-mgmt-ib)# exit
Step 3
Create and configure a link aggregation group (trunk) by assigning Ethernet ports to the link aggregation
group.
The following example shows how to create and configure a link aggregation group by assigning
Ethernet ports:
SFS-3001(config)# interface trunk 2
SFS-3001(config-if-trunk)# enable
SFS-3001(config-if-trunk)# distribution-type src-dst-ip
SFS-3001(config-if-trunk)# interface ethernet 2/1-2/6
SFS-3001(config-if-ether-2/1-2/6)# trunk-group 2
SFS-3001(config-if-ether-2/1-2/6)# exit
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-25
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch
Step 4
Configure a bridge group and assign ports to it.
Note
The IP address must be from the data IP subnet and must be unique for each bridge group.
The following example shows how to configure a bridge group and assign ports:
SFS-3001(config)# bridge-group 2 subnet-prefix 10.0.0.0 8
SFS-3001(config)# bridge-group 2 ip-addr 10.0.0.102
SFS-3001(config)# bridge-group 2 ib-next-hop 10.0.0.1
SFS-3001(config)# interface trunk 2
SFS-3001(config-if-trunk)#bridge-group 2
SFS-3001(config)# interface gateway 2
SFS-3001(config-if-gw-2/2)# bridge-group 2
SFS-3001(config-if-gw-2/2)# exit
Step 5
Configure a redundancy group, and assign a bridge group to it.
Note
The redundancy group ID must be the same in all chassis.
The following example shows how to configure a redundancy group and assign a bridge group.
SFS-3001(config)# redundancy-group 1
SFS-3001(config)# bridge-group 2 redundancy-group 1
Step 6
(Optional) Enable load balancing between bridge groups.
The following example shows how to enable load balancing between bridge groups:
SFS-3001(config)# redundancy-group 1 load-balancing
SFS-3001(config)#
Step 7
(Optional) Change the redundancy group parameters.
Note
Enable multicast forwarding on the redundancy group and not on individual bridge groups.
The following example shows how to change the redundancy group parameters:
SFS-3001(config)# redundancy-group 1 multicast
SFS-3001(config)#
Cisco SFS InfiniBand Redundancy Configuration Guide
7-26
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch
Verifying Redundancy Group Configuration for Cisco SFS 3001 Server Switches
To check the redundancy group configuration and status, use the show redundancy-group CLI
command. This command shows the redundancy group properties and all its members. It is important to
make sure the properties match the configuration and all members are reported. The redundancy groups
must be checked on both chassis.
The following is sample output from the show redundancy-group command for the first switch:
SFS-3001# show redundancy-group
================================================================================
Redundancy Groups
================================================================================
rlb-id : 1
name :
group-p_key : ff:ff
load-balancing : enabled
broadcast-forwarding : false
multicast : true
gratuitous-igmp : false
igmp-version : v2
num-members : 2
new-member-force-reelection : false
================================================================================
Redundancy Group Members
================================================================================
bridge-group src-addr
last-receive
-------------------------------------------------------------------------------1
192.168.0.1
Thu Jan 1 00:06:50 1970
2
192.168.0.2
Thu Jan 1 00:03:39 1970
The following is sample output from the show redundancy-group command for the next switch:
SFS-3001# show redundancy-group
================================================================================
Redundancy Groups
================================================================================
rlb-id : 1
name :
group-p_key : ff:ff
load-balancing : disabled
broadcast-forwarding : false
multicast : true
gratuitous-igmp : false
igmp-version : v2
num-members : 2
new-member-force-reelection : false
================================================================================
Redundancy Group Members
================================================================================
bridge-group src-addr
last-receive
-------------------------------------------------------------------------------1
192.168.0.1
Thu Jan 1 00:06:50 1970
2
192.168.0.2
Thu Jan 1 00:03:39 1970
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-27
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch
Verifying Bridge Group Configuration for Cisco SFS 3001 Server Switches
To check the bridge group configuration and status, use the show bridge-group CLI command. Check
both the Cisco SFS 3001 Server Switches to ensure all the bridge groups are members of the same
redundancy group.
When a bridge group is a member of the redundancy group, most of the properties are inherited from the
redundancy group. Also make sure only one bridge group, across all chassis, is primary and the rest are
secondary. This is true even when load balancing is enabled (active-active mode). In active-passive
mode, only the primary bridge group is forwarding. All others are in hot standby state. In active-active
mode, all bridge groups are forwarding unicast traffic and only the primary bridge group is forwarding
broadcast and multicast traffic if enabled.
Step 1
Verify the bridge group configuration for the first Cisco SFS 3001 Server Switch.
The following is sample output from the show bridge-group command that shows how to verify the
bridge group configuration for the first switch:
SFS-3001# show bridge-group
================================================================================
Bridge Groups
================================================================================
bridge-group-id : 1
bridge-group-name :
ip-addr : 10.0.0.101
eth-bridge-port : trunk 1 (not tagged)
ib-bridge-port : 2/2(gw) (pkey: ff:ff)
broadcast-forwarding : false
broadcast-forwarding-mode : inherit-from-redundancy-group
loop-protection-method : one
multicast : false
multicast-mode : inherit-from-redundancy-group
gratuitous-igmp : false
gratuitous-igmp-mode : inherit-from-redundancy-group
igmp-version : v2
igmp-version-mode : inherit-from-redundancy-group
redundancy-group : 1
status-in-redundancy-group : primary
Cisco SFS InfiniBand Redundancy Configuration Guide
7-28
OL-12957-02
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch
Step 2
Verify the bridge group configuration for the next SFS 3001 Server Switch.
The following is sample output from the show bridge-group command that shows how to verify the
bridge group configuration for the next switch:
SFS-3001# show bridge-group
================================================================================
Bridge Groups
================================================================================
bridge-group-id : 2
bridge-group-name :
ip-addr : 10.0.0.102
eth-bridge-port : trunk 2 (not tagged)
ib-bridge-port : 2/2(gw) (pkey: ff:ff)
broadcast-forwarding : false
broadcast-forwarding-mode : inherit-from-redundancy-group
loop-protection-method : one
multicast : false
multicast-mode : inherit-from-redundancy-group
gratuitous-igmp : false
gratuitous-igmp-mode : inherit-from-redundancy-group
igmp-version : v2
igmp-version-mode : inherit-from-redundancy-group
redundancy-group : 1
status-in-redundancy-group : secondary
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
7-29
Chapter 7
Ethernet Gateway and IPoIB Redundancies
Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch
Cisco SFS InfiniBand Redundancy Configuration Guide
7-30
OL-12957-02
CH A P T E R
8
Fibre Channel Gateway and SRP Redundancies
This chapter describes Fibre Channel gateway redundancies and includes the following sections:
Note
•
Dynamic Load Balancing, page 8-2
•
Dynamic Gateway Failover, page 8-2
•
Path Affinity, page 8-2
•
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch, page 8-3
•
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch,
page 8-8
•
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch, page 8-16
For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and
Abbreviations.”
Fibre Channel gateway redundancy is based on the concept of using all available paths if the configured
policy allows for it. Each ITL has a redundancy policy set and the gateway paths are used according to
those policies. Access to the available paths can either be granted or denied. The connection manager
that is resident on the controller determines the path of the Initiator/Target.
A logical diagram of Fibre Channel gateways in a redundant configuration is shown in Figure 8-1.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-1
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Dynamic Load Balancing
Figure 8-1
Logical Diagram of Fibre Channel Gateways in a Redundant Configuration
SAN Fabric
Fibre Channel Gateway
Additional Fibre Channel
Fibre Channel Gateway gateways are
added as required
183052
InfiniBand Fabric
The examples in the next sections show the most popular deployment of different types of I/O chassis.
The following features enable Fibre Channel gateways to support redundancy and high availability:
•
Dynamic Load Balancing
•
Dynamic Gateway Failover
•
Path Affinity
Dynamic Load Balancing
These features work within a chassis for the Cisco SFS 3504, SFS 3012R, and SFS 3001 Server
Switches. Dynamic load balancing automatically distributes traffic from a host to a target evenly across
all available paths and thus provides increased availability. It prevents against single points of failure or
performance bottlenecks.
Dynamic Gateway Failover
This feature is supported by the Cisco SFS 3504, SFS 3012R, and SFS 3001 Server Switches. Dynamic
gateway failover enables available gateways to assume the traffic of gateways that fail by having one or
more redundant gateways available that are ready to provide service. Fibre Channel gateways support
greater granular high availability at the port level and thereby ensure load balancing is utilized most
efficiently.
Path Affinity
This feature is supported by the SFS 3012R and the Cisco SFS 3001 Server Switches. Path affinity
compensates for the lack of load balancing capabilities of the storage systems, including the storage
systems for which multiple paths are available. When multiple I/Os are initiated to a storage system with
a queue depth that is greater than one, the host has an affinity to the path that was selected and the I/Os
are transmitted on that path until I/O count to that host reaches zero. The next set of I/Os can have a
Cisco SFS InfiniBand Redundancy Configuration Guide
8-2
OL-12957-02
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch
different path affinity. Therefore, at any given time, there is only one path in use, although multiple paths
are available. However, each time the path could be different. This is different from port failover, where
only one path is used until the path no longer exists.
Note
The Cisco SFS 3504 Server Switch does not support path affinity.
Configuring Fibre Channel Gateway Redundancy for the
Cisco SFS 3504 Server Switch
This section describes how to configure Fibre Channel gateway redundancy for the Cisco SFS 3504
chassis and includes the following topics:
•
Verifying Configured Initiator, page 8-5
•
Verifying IT, page 8-6
•
Verifying LU, page 8-7
Two or more Cisco SFS 3504 Server Switches must be used to provide high availability redundancy. A
very typical deployment would consist of two Cisco SFS 3504 Server Switches with two Fibre Channel
gateways in each (see Figure 8-2).
The Product ID number for the Fibre Channel gateway card is SFS-3500-FCGW-4G. Each Fibre Channel
gateway card is a 4-port, 4 Gbps per-port capable InfiniBand-to-Fibre Channel gateway module.
The Cisco SFS 3504 Server Switches support VSANs. For more details about VSANs, see the Cisco SFS
InfiniBand Software Configuration Guide and the Cisco SFS Product Family Command Reference.
Figure 8-2
Fibre Channel Redundancy with Dual Cisco SFS 3504 Server Switches
SAN Fabric
SFS 3504-1
SFS 3504-2
Fibre Channel Gateway Fibre Channel Gateway
InfiniBand Switch
Fibre Channel Gateway Fibre Channel Gateway
InfiniBand Switch
183053
InfiniBand Fabric
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-3
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch
To configure the first Cisco SFS 3504 Server Switch, perform the following steps:
Step 1
Enter configuration mode.
The following example shows how to enter configuration mode:
SFS-3504> enable
SFS-3504# config
Step 2
Configure the initiator.
The following example shows how to configure the initiator:
SFS-3504(config)# fc srp initiator
initiator
initiator-wwpn
SFS-3504(config)# fc srp initiator 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 wwnn ?
<wwnn>
- Initiator wwnn
suggested wwnn = 20:08:00:1b:0d:00:12:00
SFS-3504(config)# fc srp initiator 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 wwnn
20:08:00:1b:0d:00:12:00
SFS-3504(config)#
SFS-3504(config)# fc srp initiator-wwpn 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00
3/1 ?
<wwpn>
- wwpn
suggested wwpn = 20:08:00:1b:0d:00:12:16
SFS-3504(config)# fc srp initiator-wwpn 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00
3/1 20:08:00:1b:0d:00:12:16 vsan 20
SFS-3504(config)#
Step 3
Apply port masking.
The following example shows how to apply port masking:
SFS-3504(config)# no fc srp it 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00
20:01:00:20:c2:03:31:99 gateway-portmask-policy restricted 3/1-3/4
Step 4
Discover the LUNs.
The following example shows how to discover the LUNs:
SFS-3504(config)# fc srp initiator 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00
discover-itl
Step 5
Set the logical unit policy to load balancing, failover, or path-affinity.
The following example shows how to set the logical unit policy to load balancing:
SFS-3504(config)# fc srp lu <64 byte lu-id string> dynamic-gateway-port-loadbalancing
SFS-3504(config)# exit
To configure the second Cisco SFS 3504 Server Switch, perform Step 1 to Step 5 on the switch.
Cisco SFS InfiniBand Redundancy Configuration Guide
8-4
OL-12957-02
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch
Verifying Configured Initiator
The following is sample output from the show fc srp initiator command that shows how to verify the
configured initiator for the first Cisco SFS 3504 Server Switch:
SFS-3504# show fc srp initiator
================================================================================
SRP Initiators
================================================================================
guid: 00:05:ad:00:00:00:22:3c
extension: 00:00:00:00:00:00:00:00
description: svbu-fc-host-12
wwnn: 20:08:00:1b:0d:00:12:00
credit: 0
active-ports: none
pkeys:
bootup-target: 00:00:00:00:00:00:00:00
bootup-lu: 00:00:00:00:00:00:00:00
alt-bootup-target: 00:00:00:00:00:00:00:00
alt-bootup-lu: 00:00:00:00:00:00:00:00
action: none
result: none
wwpns: port
wwpn
fc-addr
vsan
3/1
20:08:00:1b:0d:00:12:16 14:01:01
20
3/2
20:08:00:1b:0d:00:12:16 14:04:01
20
3/3
20:08:00:1b:0d:00:12:16 14:04:01
20
3/4
20:08:00:1b:0d:00:12:16 14:04:01
20
To verify the configured initiator for the second Cisco SFS 3504 Server Switch, use the command after
configuring the switch.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-5
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch
Verifying IT
The following is sample output from the show fc srp it command that shows how to verify IT for the
first Cisco SFS 3504 Server Switch:
SFS-3504# show fc srp it 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00
20:01:00:20:c2:03:31:99
================================================================================
SRP IT
================================================================================
guid: 00:05:ad:00:00:00:22:3c
extension: 00:00:00:00:00:00:00:00
target-wwpn: 20:01:00:20:c2:03:31:99
description: it
non-restricted-ports: 3/1-3/4
active-ports: 3/1
physical-access: 3/1
mode: normal-mode
action: none
result: none
To verify the IT for the second Cisco SFS 3504 Server Switch, use the command after configuring the
switch.
Cisco SFS InfiniBand Redundancy Configuration Guide
8-6
OL-12957-02
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch
Verifying LU
The following is sample output from the show fc srp itl command that shows how to verify LU for the
first Cisco SFS 3504 Server Switch:
SFS-3504# show fc srp itl 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00
20:01:00:20:c2:03:31:99 00:00:00:00:00:00:00:00
================================================================================
SRP ITL
================================================================================
guid: 00:05:ad:00:00:00:22:3c
extension: 00:00:00:00:00:00:00:00
target-wwpn: 20:01:00:20:c2:03:31:99
fc-lunid: 00:00:00:00:00:00:00:00
srp-lunid: 00:00:00:00:00:00:00:00
logical-id (raw 64 bytes): 02:01:00:22:54:4d:53:20:20:20:20:20:46:43:36:35
: 20:20:20:20:20:20:20:20:20:20:20:20:30:33:33:31
: 39:39:30:30:30:30:00:00:00:00:00:00:00:00:00:00
: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
logical-id (formatted display): TMS
FC65
0331990000
description: itl
device-category: random
lun-policy: restricted
non-restricted-ports: 3/1-3/4
active-ports: none
physical-access: none
hi-mark: 16
max-retry: 5
min-io-timeout: 10
dynamic-path-affinity: false
dynamic-gateway-port-loadbalancing: true
dynamic-storage-port-loadbalancing:
dynamic-gateway-port-failover: false
dynamic-storage-port-failover:
active-slots: none
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-7
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Configuring Fibre Channel Gateway Redundancy Using the
Cisco SFS 3012R Server Switch
This section describes how to configure Fibre Channel gateway redundancy using the Cisco SFS 3012R
Server Switch and includes the following topics:
•
Configuring Fibre Channel Gateway Redundancy Using a Single Cisco SFS 3012 Server Switch,
page 8-8
•
Verifying Configuration for a Single Cisco SFS 3012R Server Switch, page 8-10
•
Configuring Fibre Channel Gateway Redundancy Using Two Cisco SFS 3012R Server Switches,
page 8-12
•
Verifying Configuration for Two Cisco SFS 3012R Server Switches, page 8-13
The Cisco SFS 3012R Server Switch has various redundancies built into the chassis. It has twelve
gateway slots, two controller modules, and two switch cards. Each gateway is connected to both switch
cards through the backplane. A single Cisco SFS 3012R Server Switch is therefore capable of being
configured for high availability.
For more information about this product, see Chapter 4, “Cisco SFS 3504 and Cisco SFS 3000 Series
Server Switch Redundancy.”
Configuring Fibre Channel Gateway Redundancy Using a Single Cisco SFS 3012
Server Switch
This section describes how to configure Fibre Channel gateway redundancy using a single
Cisco SFS 3012R Server Switch.
The example in this section shows a typical single-switch configuration with four Fibre Channel
gateways and two switch cards. Two of the gateways, slots 2 and 4, are configured to use switch cards
in the first slot and the other two gateways, slots 3 and 5, are configured to use switch cards in the next
slot. (For information about the location of the slot numbers, see Figure 4-4.) Thus if any of the switch
cards fail, two gateways continue to remain operational. If the Fibre Channel ports of the gateways are
required to be connected to two different Fibre Channel switches, connect the gateways to the same IB
switch card that are connected to different Fibre Channel switches. Thus if any one Fibre Channel switch
and any one IB switch card were to fail, at least one gateway continues to remain operational.
Note
For the purpose of this example, gateways in slots 2 and 3 must be connected to different Fibre Channel
switches than gateways in slots 4 and 5.
The topology used in this example is shown in Figure 8-3.
Cisco SFS InfiniBand Redundancy Configuration Guide
8-8
OL-12957-02
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Figure 8-3
Fibre Channel Gateway Redundancy Using a Single Cisco SFS 3012R Server Switch
SAN Fabric
SFS 3012R
Fibre Channel Gateway 1
Fibre Channel Gateway 2
Fibre Channel Gateway 3
InfiniBand Switch 1
Fibre Channel Gateway 4
InfiniBand Switch 2
183050
InfiniBand Fabric
Once the redundancy groups are configured, a primary bridge is selected and forwarding is enabled.
To configure a single Cisco SFS 3012R Server Switch, perform the following steps:
Step 1
Telnet to the Cisco SFS 3012R Server Switch.
The following example shows how to Telnet to the server switch and enter the configuration mode:
telnet ip-address <switch address>
SFS-3012R-1> enable
SFS-3012R-1# configure
Step 2
Configure the IB initiator using the IB GUID and the GUID extension.
The following example shows how to configure the IB initiator and then change the initiator description:
SFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
auto-bind
SFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
description ib-fc-init-1
Step 3
Apply port masking.
The following example shows how to apply port masking:
SFS-3012R-1(config)# no fc srp it 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
50:02:0f:23:00:00:08:70 gateway-portmask-policy restricted 2/1-2/2,3/1,3/2
Step 4
Discover the LUNs.
The following example shows how to discover the LUNs:
SFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
discover-itl
Step 5
Set the logical unit policy to load balancing, failover, or path-affinity.
The following example shows how to set the logical unit policy to load balancing:
SFS-3012R-1(config)# fc srp lu <64 byte lu-id string> dynamic-gateway-port-loadbalancing
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-9
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Step 6
Exit the switch.
The following example shows how to exit the switch:
SFS-3012R-1(config)# exit
Verifying Configuration for a Single Cisco SFS 3012R Server Switch
To verify the redundancy configuration for a single Cisco SFS 3012R Server Switch, perform the
following steps:
Step 1
Telnet to the server switch.
The following example shows how to Telnet to the server switch:
telnet ip-address <switch address>
SFS-3012R-1> enable
Step 2
Verify the configured initiator.
The following sample output from the show fc srp initiator command shows how to verify the
configured initiator:
SFS-3012R-1# show fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
================================================
SRP Initiators
================================================
guid: 29:01:00:05:ad:00:24:fc
extension: 00:00:00:00:00:00:00:00
description: ib-fc-init-1
wwnn: 20:01:00:05:ad:00:1b:4f
credit: 0
active-ports: none
pkeys:
bootup-target: 00:00:00:00:00:00:00:00
bootup-lu: 00:00:00:00:00:00:00:00
alt-bootup-target: 00:00:00:00:00:00:00:00
alt-bootup-lu: 00:00:00:00:00:00:00:00
action: discover-itl
result: success
wwpns: port
wwpn
fc-addr
2/1
20:01:00:05:ad:20:1b:4f 67:0f:04
2/2
20:01:00:05:ad:24:1b:4f 67:0e:04
3/1
20:01:00:05:ad:30:1b:4f 67:11:18
3/2
20:01:00:05:ad:34:1b:4f 67:10:18
Cisco SFS InfiniBand Redundancy Configuration Guide
8-10
OL-12957-02
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Step 3
Verify the configured Initiator/Target.
The following sample output from the show fc srp it command shows how to verify the configured
Initiator/Target:
SFS-3012R-1# show fc srp it
50:02:0f:23:00:00:08:70
29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
================================================
SRP IT
================================================
guid: 29:01:00:05:ad:00:24:fc
extension: 00:00:00:00:00:00:00:00
target-wwpn: 50:02:0f:23:00:00:08:70
description: it
non-restricted-ports: 2/1-2/2,3/1-3/2
active-ports: none
physical-access: 2/1-2/2,3/1-3/2
mode: normal-mode
action: none
result: none
Step 4
Verify the configured logical unit.
The following is sample output from the show fc srp itl command that shows how to verify the
configured logical unit:
SFS-3012R-1# show fc srp itl 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
50:02:0f:23:00:00:08:70 00:00:00:00:00:00:00:00
================================================
SRP ITL
================================================
guid: 29:01:00:05:ad:00:24:fc
extension: 00:00:00:00:00:00:00:00
target-wwpn: 50:02:0f:23:00:00:08:70
fc-lunid: 00:00:00:00:00:00:00:00
srp-lunid: 00:00:00:00:00:00:00:00
logical-id (raw 64 bytes): 01:03:00:10:60:06:01:60:a2:70:0d:00:70:ad:1c:0a
: 3b:a8:db:11:00:00:00:00:00:00:00:00:00:00:00:00
: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
logical-id (formatted display): 60060160A2700D0070AD1C0A3BA8DB11
description: itl
device-category: random
lun-policy: non restricted
non-restricted-ports: 2/1-2/2,3/1-3/2
active-ports: none
physical-access: 2/1-2/2,3/1-3/2
hi-mark: 16
max-retry: 5
min-io-timeout: 10
dynamic-path-affinity: false
dynamic-gateway-port-loadbalancing: true
dynamic-storage-port-loadbalancing:
dynamic-gateway-port-failover: false
dynamic-storage-port-failover:
active-slots: none
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-11
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Step 5
Exit the switch.
The following example shows how to exit the switch:
SFS-3012R-1(config)# exit
Step 6
To verify the redundancy configuration for the second Cisco SFS 3012R Server Switch, repeat Step 1 to
Step 5 for the next switch.
Configuring Fibre Channel Gateway Redundancy Using Two Cisco SFS 3012R
Server Switches
This section describes how to configure Fibre Channel gateway redundancy using two Cisco SFS 3012R
Server Switches.
This typical example shows a setup of two Cisco SFS 3012R Server Switches with a total of four Fibre
Channel gateways that are available as two for each server switch. Both gateways in each chassis are
configured to use different switch cards. The gateway in slot 2 uses the switch card in one slot and the
gateway in slot 3 uses the switch card in the other slot. Thus if any of the switch cards fail, the other
gateway continues to remain operational. If the Fibre Channel ports of the gateways are connected to two
different Fibre Channel switches, the gateways in the same chassis must be connected to different Fibre
Channel switches. This topology allows one chassis and one Fibre Channel switch failure without
interruption to the service. The topology used in this example is shown in Figure 8-4.
Figure 8-4
Fibre Channel Gateway Redundancy Using Two Cisco SFS 3012R Server Switches
SAN Fabric
SFS 3012R
Fibre Channel Gateway 1
InfiniBand Switch 1
Fibre Channel Gateway 2
InfiniBand Switch 2
Fibre Channel Gateway 3
InfiniBand Switch 1
SFS 3012R-2
Fibre Channel Gateway 4
InfiniBand Switch 2
183051
InfiniBand Fabric
Once a redundancy group is configured, a primary bridge group is elected and forwarding is enabled.
Cisco SFS InfiniBand Redundancy Configuration Guide
8-12
OL-12957-02
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch
To configure two Cisco SFS 3012R Server Switches, perform the following steps:
Step 1
Telnet to the first Cisco SFS 3012R Server Switch.
The following example shows how to Telnet to the server switch:
telnet ip-address <switch address>
SFS-3012R-1> enable
SFS-3012R-1# configure
Step 2
Configure the IB initiator using the IB GUID and the GUID extension.
The following example shows how to configure the IB initiator and then change the initiator description:
SFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
auto-bind
SFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
description ib-fc-init-1
Step 3
Apply port masking.
The following example shows how to apply port masking:
SFS-3012R-1(config)# no fc srp it 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
50:02:0f:23:00:00:08:70 gateway-portmask-policy restricted 2/1-2/2,3/1,3/2
Step 4
Discover the LUNs.
The following example shows how to discover the LUNs:
3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
discover-itl
Step 5
Set the logical unit policy to load balancing, failover, or path-affinity.
The following example shows how to set the logical unit policy to load balancing:
SFS-3012R-1(config)# fc srp lu <64 byte lu-id string> dynamic-gateway-port-loadbalancing
Step 6
Exit the switch.
The following example shows how to exit the switch:
SFS-3012R-1(config)# exit
Step 7
To configure the second Cisco SFS 3012R Server Switch, repeat Step 1 to Step 6 for the next switch.
Verifying Configuration for Two Cisco SFS 3012R Server Switches
To verify configuration for the two Cisco SFS 3012R Server Switches, perform the following steps:
Step 1
Telnet to the Cisco SFS 3012R Server Switch.
The following example shows how to Telnet to the server switch:
telnet ip-address <switch address>
SFS-3012R-1> enable
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-13
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Step 2
Verify the configured initiator.
The following is sample output from the show fc srp initiator command that shows how to verify the
configured initiator:
SFS-3012R-1# show fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
================================================
SRP Initiators
================================================
guid: 29:01:00:05:ad:00:24:fc
extension: 00:00:00:00:00:00:00:00
description: ib-fc-init-1
wwnn: 20:01:00:05:ad:00:1b:4f
credit: 0
active-ports: none
pkeys:
bootup-target: 00:00:00:00:00:00:00:00
bootup-lu: 00:00:00:00:00:00:00:00
alt-bootup-target: 00:00:00:00:00:00:00:00
alt-bootup-lu: 00:00:00:00:00:00:00:00
action: discover-itl
result: success
wwpns: port
wwpn
fc-addr
2/1
20:01:00:05:ad:20:1b:4f 67:0f:04
2/2
20:01:00:05:ad:24:1b:4f 67:0e:04
3/1
20:01:00:05:ad:30:1b:4f 67:11:18
3/2
20:01:00:05:ad:34:1b:4f 67:10:18
Step 3
Verify the configured Initiator/Target.
The following is sample output from the show fc srp it command that shows how to verify the configured
Initiator/Target:
SFS-3012R-1# show fc srp it
50:02:0f:23:00:00:08:70
29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
================================================
SRP IT
================================================
guid: 29:01:00:05:ad:00:24:fc
extension: 00:00:00:00:00:00:00:00
target-wwpn: 50:02:0f:23:00:00:08:70
description: it
non-restricted-ports: 2/1-2/2,3/1-3/2
active-ports: none
physical-access: 2/1-2/2,3/1-3/2
mode: normal-mode
action: none
result: none
Cisco SFS InfiniBand Redundancy Configuration Guide
8-14
OL-12957-02
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch
Step 4
Verify the configured logical unit.
The following is sample output from the show fc srp itl command that shows how to verify the
configured logical unit:
SFS-3012R-1# show fc srp itl 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
50:02:0f:23:00:00:08:70 00:00:00:00:00:00:00:00
================================================
SRP ITL
================================================
guid: 29:01:00:05:ad:00:24:fc
extension: 00:00:00:00:00:00:00:00
target-wwpn: 50:02:0f:23:00:00:08:70
fc-lunid: 00:00:00:00:00:00:00:00
srp-lunid: 00:00:00:00:00:00:00:00
logical-id (raw 64 bytes): 01:03:00:10:60:06:01:60:a2:70:0d:00:70:ad:1c:0a
: 3b:a8:db:11:00:00:00:00:00:00:00:00:00:00:00:00
: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
logical-id (formatted display): 60060160A2700D0070AD1C0A3BA8DB11
description: itl
device-category: random
lun-policy: non restricted
non-restricted-ports: 2/1-2/2,3/1-3/2
active-ports: none
physical-access: 2/1-2/2,3/1-3/2
hi-mark: 16
max-retry: 5
min-io-timeout: 10
dynamic-path-affinity: false
dynamic-gateway-port-loadbalancing: true
dynamic-storage-port-loadbalancing:
dynamic-gateway-port-failover: false
dynamic-storage-port-failover:
active-slots: none
Step 5
Exit the switch.
The following example shows how to exit the switch:
SFS-3012R-1(config)# exit
Step 6
To verify the configuration for the second Cisco SFS 3012R switch, repeat Step 1 to Step 5 for the
switch.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-15
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch
Configuring Fibre Channel Gateway Redundancy for the
Cisco SFS 3001 Server Switch
This section describes how to configure Fibre Channel gateway redundancy for the Cisco SFS 3001
Server Switch and includes the following topics:
•
Configuring Two Cisco SFS 3001 Server Switches, page 8-16
•
Verifying Redundancy Configuration for Cisco SFS 3001 Server Switches, page 8-17
Two or more Cisco SFS 3001 Server Switches must be used to provide high availability redundancy. A
typical deployment consists of two Cisco SFS 3001 Server Switches using one Fibre Channel gateway
in each (see Figure 8-5).
Figure 8-5
Fibre Channel Gateway Redundancy Using Two Cisco SFS 3001 Server Switches
SAN Fabric
SFS 3001-1
SFS 3001-2
Fibre Channel Gateway
Fibre Channel Gateway
InfiniBand Switch
InfiniBand Switch
183049
InfiniBand Fabric
A single Cisco SFS 3001 Server Switch provides power supply redundancy only. A single
Cisco SFS 3001 Server Switch cannot provide Fibre Channel gateway redundancy, because it contains a
single gateway slot. For more information about this product, see Chapter 4, “Cisco SFS 3504 and
Cisco SFS 3000 Series Server Switch Redundancy.”
Configuring Two Cisco SFS 3001 Server Switches
This section describes how to configure two Cisco SFS 3001 Server Switches. To configure two
Cisco SFS 3001 Server Switches, perform the following steps:
Step 1
Telnet to the server switch.
The following example shows how to Telnet to a server switch:
telnet ip-address <switch address>
SFS-3001-1> enable
SFS-3001-1# configure
Cisco SFS InfiniBand Redundancy Configuration Guide
8-16
OL-12957-02
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch
Step 2
Configure the IB initiator using the IB GUID and the GUID-extension.
The following is sample output from the show fc srp initiator command that shows how to configure
the IB initiator and then change the initiator description:
SFS-3001-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
auto-bind
SFS-3001-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
description ib-fc-init-1
Step 3
Apply port masking.
The following example shows how to apply port masking:
SFS-3001-1(config)# no fc srp it 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
50:02:0f:23:00:00:08:70 gateway-portmask-policy restricted 2/1-2/2
Step 4
Discover the LUNs.
The following example shows how to discover the LUNs:
SFS-3001-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
discover-itl
Step 5
Set the logical unit policy to load balancing, failover, or path affinity.
The following example shows how to set the logical unit policy to load balancing:
SFS-3001-1(config)# fc srp lu <64 byte lu-id string> dynamic-gateway-port-loadbalancing
Step 6
To configure the second Cisco SFS 3001 Server Switch, repeat Step 1 to Step 5 for the next switch.
Verifying Redundancy Configuration for Cisco SFS 3001 Server Switches
To verify redundancy for the Cisco SFS 3001 Server Switches, perform the following steps:
Step 1
Telnet to the first Cisco SFS 3001 Server Switch.
The following example shows how to Telnet to the server switch:
telnet ip-address <switch address>
SFS-3001-1> enable
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-17
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch
Step 2
Verify the configured initiator.
The following example shows how to verify the configured initiator:
SFS-3001-1# show fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
================================================
SRP Initiators
================================================
guid: 29:01:00:05:ad:00:24:fc
extension: 00:00:00:00:00:00:00:00
description: ib-fc-init-1
wwnn: 20:01:00:05:ad:00:1b:4f
credit: 0
active-ports: none
pkeys:
bootup-target: 00:00:00:00:00:00:00:00
bootup-lu: 00:00:00:00:00:00:00:00
alt-bootup-target: 00:00:00:00:00:00:00:00
alt-bootup-lu: 00:00:00:00:00:00:00:00
action: discover-itl
result: success
wwpns: port
wwpn
fc-addr
2/1
20:01:00:05:ad:20:1b:4f 67:0f:04
2/2
20:01:00:05:ad:24:1b:4f 67:0e:04
Step 3
Verify the configured Initiator/Target.
The following is sample output from the show fc srp it command that shows how to verify the configured
Initiator/Target:
SFS-3001-1# show fc srp it
50:02:0f:23:00:00:08:70
29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
================================================
SRP IT
================================================
guid: 29:01:00:05:ad:00:24:fc
extension: 00:00:00:00:00:00:00:00
target-wwpn: 50:02:0f:23:00:00:08:70
description: it
non-restricted-ports: 2/1-2/2
active-ports: none
physical-access: 2/1-2/2
mode: normal-mode
action: none
result: none
Step 4
Verify the configured logical unit.
The following is sample output from the show fc srp itl command that shows how to verify the
configured logical unit:
SFS-3001-1# show fc srp itl 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00
50:02:0f:23:00:00:08:70 00:00:00:00:00:00:00:00
================================================
SRP ITL
================================================
guid: 29:01:00:05:ad:00:24:fc
extension: 00:00:00:00:00:00:00:00
target-wwpn: 50:02:0f:23:00:00:08:70
fc-lunid: 00:00:00:00:00:00:00:00
srp-lunid: 00:00:00:00:00:00:00:00
logical-id (raw 64 bytes): 01:03:00:10:60:06:01:60:a2:70:0d:00:70:ad:1c:0a
: 3b:a8:db:11:00:00:00:00:00:00:00:00:00:00:00:00
Cisco SFS InfiniBand Redundancy Configuration Guide
8-18
OL-12957-02
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch
:
:
logical-id (formatted display):
description:
device-category:
lun-policy:
non-restricted-ports:
active-ports:
physical-access:
hi-mark:
max-retry:
min-io-timeout:
dynamic-path-affinity:
dynamic-gateway-port-loadbalancing:
dynamic-storage-port-loadbalancing:
dynamic-gateway-port-failover:
dynamic-storage-port-failover:
active-slots:
Step 5
00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
60060160A2700D0070AD1C0A3BA8DB11
itl
random
non restricted
2/1-2/2
none
2/1-2/2
16
5
10
false
true
false
none
Exit from the switch.
The following example shows how to exit from the switch:
SFS-3001-1# exit
Step 6
To verify the configuration for the second Cisco SFS 3001 Server Switch, repeat Step 1 to Step 5 for the
next switch.
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
8-19
Chapter 8
Fibre Channel Gateway and SRP Redundancies
Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch
Cisco SFS InfiniBand Redundancy Configuration Guide
8-20
OL-12957-02
CH A P T E R
9
Typical Redundancy Use Case
This chapter describes a typical redundancy use case of an Oracle Real Application Clusters (RAC) 10g
installation.
Note
For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and
Abbreviations.”
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
9-1
Chapter 9
Figure 9-1
Typical Redundancy Use Case
Example of a Fully-Redundant Oracle RAC 10g Installation
LUN
VSAN Fabric-1
VSAN Fabric-2
ENGW-1
ENGW-2
ENGW-1
ENGW-2
FCGW-1
FCGW-2
FCGW-1
FCGW-2
SFS 3504-2
SFS 3504-1
SFS 7000D-1
HCA-1
SFS 7000D-2
HCA-1
HCA-2
IB Host 1
16 1B Hosts
HCA-2
IB Host 16
FCGW - Fibre Channel Gateway
ENGW - Ethernet Gateway
241774
LAN Fabric
Figure 9-1 is an example of a fully-redundant Oracle RAC10g installation. The environment for this
cluster is set such that there is no single point of failure. The following redundancies are built into this
environment:
•
HCA redundancy:
For more details about HCA redundancy, see Chapter 6, “Host Redundancy, and IPoIB and SRP
Redundancies.”
•
IPoIB and SRP redundancy:
For more details about IPoIB and SRP redundancy, see Chapter 6, “Host Redundancy, and IPoIB
and SRP Redundancies.”
•
Cisco SFS 7008P and SFS 7000 Series Server Switches redundancy:
For more details about Cisco SFS 7008P and SFS 7000 Series Server Switches redundancy, see
Chapter 2, “Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy.”
Cisco SFS InfiniBand Redundancy Configuration Guide
9-2
OL-12957-02
Chapter 9
Typical Redundancy Use Case
•
Subnet Manager redundancy:
For more details about Subnet Manager redundancy, see Chapter 5, “Subnet Manager Redundancy.”
•
Cisco SFS 3504 and SFS 3000 Series Multifabric Switches redundancy:
For more details about Cisco SFS 3504 and SFS 3000 Series Multifabric Switches redundancy, see
Chapter 4, “Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy.”
•
Ethernet gateways redundancy:
For more details, see Chapter 7, “Ethernet Gateway and IPoIB Redundancies.”
•
Fibre Channel gateway redundancy:
For more details, see Chapter 8, “Fibre Channel Gateway and SRP Redundancies.”
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
9-3
Chapter 9
Typical Redundancy Use Case
Cisco SFS InfiniBand Redundancy Configuration Guide
9-4
OL-12957-02
A P P E N D I X
A
Acronyms and Abbreviations
Table A-1 defines the acronyms and abbreviations that are used in this publication.
Table A-1
List of Acronyms and Abbreviations
Acronym
Expansion
API
Application Program Interface
APM
Automatic Path Migration
CLI
command-line interface
HCA
Host Channel Adapter
HSM
High-Performance Subnet Manager
IB
InfiniBand
IPoIB
IP over InfiniBand
ITL
Initiator/Target/LUN
LID
Local ID
LU
logical unit
LUN
logical unit number
MTBF
mean time between failure
OFED
OpenFabrics Enterprise Distribution
RAC
Real Application Clusters
RHEL
Red Hat Enterprise Linux
SFS
Server Fabric Switch
SLES
SuSE Linux Enterprise Server
SNMP
Simple Network Management Protocol
SRP
SCSI RDMA Protocol
SSH
Secure Shell
VLAN
virtual local area network
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
A-1
Appendix A
Acronyms and Abbreviations
Cisco SFS InfiniBand Redundancy Configuration Guide
A-2
OL-12957-02
INDEX
organization
A
related
abbreviations
acronyms
1-1
dual-port HCA
4-6
6-6
6-1
dynamic gateway failover
active standby controller
audience
ix
driver operation
1-1
active controller
vii
4-6
dynamic load balancing
vii
8-2
8-2
dynamic storage reconfiguration
B
6-9
E
blanking panel
2-4
embedded processors
blower module redundancy
bonding driver
bridge group
4-6
Embedded Subnet Manager
6-6
end storage unit
7-1
broadcast traffic
5-4
6-8
Enterprise solutions
7-1
5-1
1-1
Ethernet gateway redundancy
Ethernet port
4-4, 4-7, 7-1
2-5
C
CLI session
F
2-2
controller card
5-3
fabric card
controller module redundancy
conventions, document
core slot
viii
2-6
4-6
2-3
fabric controller
fabric redundancy
failover
fan tray
protocol
data traffic
4-9
Fibre Channel gateway redundancy
Fibre Channel network
5-1
4-4, 4-7, 8-1
4-4
5-1
Device Mapper Multipath
document
audience
5-8
4-9
2-4
fan tray module
database synchronization
2-8, 4-7, 4-10
6-8
fan redundancy
D
2-5, 2-6
vii
conventions
6-8
H
high availability, IPoIB
viii
6-6
High-Performance Subnet Manager
5-1, 5-4
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
IN-1
Index
host redundancy
hot-standby
hot swap
HTTP
6-1
N
2-2
Netperf
4-3, 4-5
6-8
node card
2-3
2-3
node slot
2-6
I
O
IBM Blade Center
IP network
3-1
operating system
4-4
4-2
optimal routing
IPoIB
configure
Oracle RAC 10g
6-6
failover
P
6-1
IPoIB specification
partition
6-4
4-2
path affinity
7-2
8-2
port aggregation
port failover
L
4-7
6-6
port redundancy
large clusters
large fabric
2-10
power-fan module
5-8
4-3
power supply module
5-4
Line Interface Module
load balancing
power supply redundancy
primary controller
primary interface
M
management interface module
product ID
5-7
merge, physical ports
multiple HCAs
5-4, 5-6
8-3
R
6-4
reboot
2-2
recovery mode
7-1
multi-IC switches
2-2
5-1
5-8
multicast traffic
2-9, 4-5
5-7
master Subnet Manager
MTBF
2-3, 2-5, 2-6
4-9
6-8
priority number
master-poll-interval
2-4
power supply module redundancy
2-6
4-7, 6-8
master-poll-retries
vii
6-6
6-8
IPoIB redundancy
IP subnet
9-1
organization, document
6-8
high availability
verify
5-4
4-7
6-3
2-3
Red Hat Enterprise Linux. See RHEL
redundancy
software
1-1
2-2
related documentation
reset
ix
2-2
Cisco SFS InfiniBand Redundancy Configuration Guide
IN-2
OL-12957-02
Index
RHEL
6-8
V
verbose mode
S
6-6
virtual local area network. See VLAN
Secure Shell. See SSH
virtual port
serial console port
VLAN
serial port
2-3
6-6
7-1
2-5
Server Fabric Switch
1-1
W
SFS. See Server Fabric Switch
Simple Network Management Protocol. See SNMP.
world-wide names. See WWN.
SLES
WWN
SNMP
6-8
2-3
software redundancy
SRP redundancy
SSH
6-8
2-2, 4-2
6-1
2-3
standby controller
2-2
standby Subnet Manager
5-1
Subnet Manager
Embedded
5-1
High-Performance
master
5-1
redundancy
standby
5-1, 5-4
5-1
5-1
SuSE Linux Enterprise Server. See SLES
switch module redundancy
system mastership
4-7
2-6
T
Telnet
2-3
trap mechanism
5-4
U
unmerge, physical ports
6-5
Cisco SFS InfiniBand Redundancy Configuration Guide
OL-12957-02
IN-3
Index
Cisco SFS InfiniBand Redundancy Configuration Guide
IN-4
OL-12957-02