Cisco InfiniBand Host Channel Adapter
Hardware Installation Guide
Linux Release 3.2.0, Windows Release 2.0.3
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
Text Part Number: OL-12622-02
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant
to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial
environment. This equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause
harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required
to correct the interference at their own expense.
The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it is not
installed in accordance with Cisco’s installation instructions, it may cause interference with radio and television reception. This equipment has been tested and found to
comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC rules. These specifications are designed to provide reasonable
protection against such interference in a residential installation. However, there is no guarantee that interference will not occur in a particular installation.
Modifying the equipment without Cisco’s written authorization may result in the equipment no longer complying with FCC requirements for Class A or Class B digital
devices. In that event, your right to use the equipment may be limited by FCC regulations, and you may be required to correct any interference to radio or television
communications at your own expense.
You can determine whether your equipment is causing interference by turning it off. If the interference stops, it was probably caused by the Cisco equipment or one of its
peripheral devices. If the equipment causes interference to radio or television reception, try to correct the interference by using one or more of the following measures:
• Turn the television or radio antenna until the interference stops.
• Move the equipment to one side or the other of the television or radio.
• Move the equipment farther away from the television or radio.
• Plug the equipment into an outlet that is on a different circuit from the television or radio. (That is, make certain the equipment and the television or radio are on circuits
controlled by different circuit breakers or fuses.)
Modifications to this product not authorized by Cisco Systems, Inc. could void the FCC approval and negate your authority to operate the product.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
CCDE, CCENT, Cisco Eos, Cisco StadiumVision, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and
Learn is a service mark; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco,
the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without
Limitation, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient,
IOS, iPhone, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MGX,
Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The
Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United
States and certain other countries.
All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship
between Cisco and any other company. (0803R)
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the
document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
Host Channel Adapter Hardware Installation Guide, Linux Release 3.2.0, Windows Release 2.0.3
© 2006–2008 Cisco Systems, Inc. All rights reserved.
C O N T E N T S
Preface
5
Audience
1-5
Organization
1-5
Conventions
1-6
Related Documentation
1-12
Obtaining Documentation and Submitting a Service Request
CHAPTER
1
About the Host Channel Adapter
1-12
1-1
Introduction 1-1
HCA Hardware 1-1
Firmware 1-2
HCA Host Drivers 1-2
Supported Protocols 1-2
IPoiB 1-3
SDP (Linux only) 1-3
SRP 1-3
Supported APIs 1-3
uDAPL (Linux only)
MPI 1-4
1-3
Supported Operating System Software
Linux Kernels 1-4
Windows 1-4
About Boot over InfiniBand Functionality
Hardware Specifications
CHAPTER
2
Installing an HCA
1-4
1-4
Installing the Host Channel Adapter
Selecting the PCI Slot
PCI-X 2-1
PCI Express 2-2
1-4
2-1
2-1
2-4
Installing the Host Drivers
2-7
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
3
Contents
CHAPTER
Replacing a Tall Bracket with a Short Bracket
3
Required Tools
3-2
Removing a Tall Bracket
3-2
Attaching a Short Bracket
CHAPTER
Interpret HCA LEDs
A
4-1
4-1
Check the InfiniBand Cable
APPENDIX
3-4
Troubleshooting the HCA Installation
4
3-1
Acronyms and Abbreviations
4-2
A-1
INDEX
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
4
OL-12622-02
Preface
This preface describes who should read the Cisco InfiniBand Host Channel Adapter Hardware
Installation Guide, how it is organized, and its document conventions, and it contains these sections:
•
Audience, page 5
•
Organization, page 5
•
Conventions, page 6
•
Related Documentation, page 12
•
Obtaining Documentation and Submitting a Service Request, page 12
Audience
The audience for this document is the administrator responsible for installing, configuring, and
managing Cisco Server Fabric Switch equipment. This administrator should have experience
administering similar networking or storage equipment.
Only trained and qualified service personnel (as defined in IEC 60950 and AS/NZS3260) should install,
replace, or service the equipment described in this publication.
Organization
This publication is organized as follows:
Chapter
Title
Description
Chapter 1
About the Host Channel
Adapter
Provides an overview of the
Host Channel Adapter components and functionality.
Chapter 2
Installing the Host
Channel Adapter
Describes how to install the Host Channel Adapter and
tells where to obtain instructions for installing the
InfiniBand host drivers.
Chapter 3
Replacing a Tall Bracket Shows how to remove a tall bracket from an HCA
expansion card and replace it with a short bracket.
with a Short Bracket
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
5
Preface
Conventions
Chapter
Title
Description
Chapter 4
Troubleshooting the
HCA Installation
Describes how to interpret Host Channel Adapter LEDs
and check InfiniBand cable connections.
Appendix A
Acronyms and
Abbreviations
Defines the acronyms and abbreviations used in this
publication.
Conventions
This publication uses the following conventions:
Convention
Description
boldface font
Commands, command options, and keywords are in boldface.
italic font
Arguments for which you supply values are in italics.
[ ]
Elements in square brackets are optional.
{x|y|z}
Alternative keywords are grouped in braces and separated by vertical bars.
[x|y|z]
Optional alternative keywords are grouped in brackets and separated by
vertical bars.
string
A nonquoted set of characters. Do not use quotation marks around the string
or the string will include the quotation marks.
screen
font
Terminal sessions and information the system displays are in screen font.
boldface screen
italic screen
font
font
Information you must enter is in boldface
screen
font.
Arguments for which you supply values are in italic screen font.
^
The symbol ^ represents the key labeled Control. For example, the key
combination ^D in a screen display means hold down the Control key while
you press the D key.
< >
Nonprinting characters, such as passwords, are in angle brackets.
Notes use the following conventions:
Note
Means reader take note. Notes contain helpful suggestions or references to material not covered in the
publication.
Cautions use the following conventions:
Caution
Means reader be careful. In this situation, you might do something that could result in equipment
damage or loss of data.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
6
OL-12622-02
Preface
Conventions
Warnings use the following conventions:
Warning
IMPORTANT SAFETY INSTRUCTIONS
This warning symbol means danger. You are in a situation that could cause bodily injury. Before you
work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar
with standard practices for preventing accidents. Use the statement number provided at the end of
each warning to locate its translation in the translated safety warnings that accompanied this
device. Statement 1071
SAVE THESE INSTRUCTIONS
Waarschuwing
BELANGRIJKE VEILIGHEIDSINSTRUCTIES
Dit waarschuwingssymbool betekent gevaar. U verkeert in een situatie die lichamelijk letsel kan
veroorzaken. Voordat u aan enige apparatuur gaat werken, dient u zich bewust te zijn van de bij
elektrische schakelingen betrokken risico's en dient u op de hoogte te zijn van de standaard
praktijken om ongelukken te voorkomen. Gebruik het nummer van de verklaring onderaan de
waarschuwing als u een vertaling van de waarschuwing die bij het apparaat wordt geleverd, wilt
raadplegen.
BEWAAR DEZE INSTRUCTIES
Varoitus
TÄRKEITÄ TURVALLISUUSOHJEITA
Tämä varoitusmerkki merkitsee vaaraa. Tilanne voi aiheuttaa ruumiillisia vammoja. Ennen kuin
käsittelet laitteistoa, huomioi sähköpiirien käsittelemiseen liittyvät riskit ja tutustu
onnettomuuksien yleisiin ehkäisytapoihin. Turvallisuusvaroitusten käännökset löytyvät laitteen
mukana toimitettujen käännettyjen turvallisuusvaroitusten joukosta varoitusten lopussa näkyvien
lausuntonumeroiden avulla.
SÄILYTÄ NÄMÄ OHJEET
Attention
IMPORTANTES INFORMATIONS DE SÉCURITÉ
Ce symbole d'avertissement indique un danger. Vous vous trouvez dans une situation pouvant
entraîner des blessures ou des dommages corporels. Avant de travailler sur un équipement, soyez
conscient des dangers liés aux circuits électriques et familiarisez-vous avec les procédures
couramment utilisées pour éviter les accidents. Pour prendre connaissance des traductions des
avertissements figurant dans les consignes de sécurité traduites qui accompagnent cet appareil,
référez-vous au numéro de l'instruction situé à la fin de chaque avertissement.
CONSERVEZ CES INFORMATIONS
Warnung
WICHTIGE SICHERHEITSHINWEISE
Dieses Warnsymbol bedeutet Gefahr. Sie befinden sich in einer Situation, die zu Verletzungen führen
kann. Machen Sie sich vor der Arbeit mit Geräten mit den Gefahren elektrischer Schaltungen und
den üblichen Verfahren zur Vorbeugung vor Unfällen vertraut. Suchen Sie mit der am Ende jeder
Warnung angegebenen Anweisungsnummer nach der jeweiligen Übersetzung in den übersetzten
Sicherheitshinweisen, die zusammen mit diesem Gerät ausgeliefert wurden.
BEWAHREN SIE DIESE HINWEISE GUT AUF.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
7
Preface
Conventions
Avvertenza
IMPORTANTI ISTRUZIONI SULLA SICUREZZA
Questo simbolo di avvertenza indica un pericolo. La situazione potrebbe causare infortuni alle
persone. Prima di intervenire su qualsiasi apparecchiatura, occorre essere al corrente dei pericoli
relativi ai circuiti elettrici e conoscere le procedure standard per la prevenzione di incidenti.
Utilizzare il numero di istruzione presente alla fine di ciascuna avvertenza per individuare le
traduzioni delle avvertenze riportate in questo documento.
CONSERVARE QUESTE ISTRUZIONI
Advarsel
VIKTIGE SIKKERHETSINSTRUKSJONER
Dette advarselssymbolet betyr fare. Du er i en situasjon som kan føre til skade på person. Før du
begynner å arbeide med noe av utstyret, må du være oppmerksom på farene forbundet med
elektriske kretser, og kjenne til standardprosedyrer for å forhindre ulykker. Bruk nummeret i slutten
av hver advarsel for å finne oversettelsen i de oversatte sikkerhetsadvarslene som fulgte med denne
enheten.
TA VARE PÅ DISSE INSTRUKSJONENE
Aviso
INSTRUÇÕES IMPORTANTES DE SEGURANÇA
Este símbolo de aviso significa perigo. Você está em uma situação que poderá ser causadora de
lesões corporais. Antes de iniciar a utilização de qualquer equipamento, tenha conhecimento dos
perigos envolvidos no manuseio de circuitos elétricos e familiarize-se com as práticas habituais de
prevenção de acidentes. Utilize o número da instrução fornecido ao final de cada aviso para
localizar sua tradução nos avisos de segurança traduzidos que acompanham este dispositivo.
GUARDE ESTAS INSTRUÇÕES
¡Advertencia!
INSTRUCCIONES IMPORTANTES DE SEGURIDAD
Este símbolo de aviso indica peligro. Existe riesgo para su integridad física. Antes de manipular
cualquier equipo, considere los riesgos de la corriente eléctrica y familiarícese con los
procedimientos estándar de prevención de accidentes. Al final de cada advertencia encontrará el
número que le ayudará a encontrar el texto traducido en el apartado de traducciones que acompaña
a este dispositivo.
GUARDE ESTAS INSTRUCCIONES
Varning!
VIKTIGA SÄKERHETSANVISNINGAR
Denna varningssignal signalerar fara. Du befinner dig i en situation som kan leda till personskada.
Innan du utför arbete på någon utrustning måste du vara medveten om farorna med elkretsar och
känna till vanliga förfaranden för att förebygga olyckor. Använd det nummer som finns i slutet av
varje varning för att hitta dess översättning i de översatta säkerhetsvarningar som medföljer denna
anordning.
SPARA DESSA ANVISNINGAR
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
8
OL-12622-02
Preface
Conventions
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
9
Preface
Conventions
Aviso
INSTRUÇÕES IMPORTANTES DE SEGURANÇA
Este símbolo de aviso significa perigo. Você se encontra em uma situação em que há risco de lesões
corporais. Antes de trabalhar com qualquer equipamento, esteja ciente dos riscos que envolvem os
circuitos elétricos e familiarize-se com as práticas padrão de prevenção de acidentes. Use o
número da declaração fornecido ao final de cada aviso para localizar sua tradução nos avisos de
segurança traduzidos que acompanham o dispositivo.
GUARDE ESTAS INSTRUÇÕES
Advarsel
VIGTIGE SIKKERHEDSANVISNINGER
Dette advarselssymbol betyder fare. Du befinder dig i en situation med risiko for
legemesbeskadigelse. Før du begynder arbejde på udstyr, skal du være opmærksom på de
involverede risici, der er ved elektriske kredsløb, og du skal sætte dig ind i standardprocedurer til
undgåelse af ulykker. Brug erklæringsnummeret efter hver advarsel for at finde oversættelsen i de
oversatte advarsler, der fulgte med denne enhed.
GEM DISSE ANVISNINGER
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
10
OL-12622-02
Preface
Conventions
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
11
Preface
Related Documentation
Related Documentation
For more information about working with Host Channel Adapters, refer to the related documentation at
http://www.cisco.com/en/US/products/ps6428/tsd_products_support_series_home.html:
•
Cisco SFS Boot over InfiniBand User Guide for Linux
•
Cisco SFS InfiniBand Host Drivers User Guide for Linux
•
Cisco OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for Linux
•
Release Notes for Cisco OFED
•
Release Notes for Linux Host Drivers
•
Release Notes for Windows Host Drivers
Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, submitting a service request, and gathering additional
information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and
revised Cisco technical documentation, at:
http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html
Subscribe to the What’s New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed
and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free
service and Cisco currently supports RSS version 2.0.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
12
OL-12622-02
C H A P T E R
1
About the Host Channel Adapter
The following sections appear in this chapter:
•
Introduction, page 1-1
•
Supported Protocols, page 1-2
•
Supported APIs, page 1-3
•
Supported Operating System Software, page 1-4
•
About Boot over InfiniBand Functionality, page 1-4
•
Hardware Specifications, page 1-4
Introduction
The Cisco InfiniBand Host Channel Adapter offers high-performance connectivity to PCI-X and PCIe
based servers: 4x IB SDR connectivity at 10-Gbps and 4x IB DDR connectivity at 20-Gbps. As an
integral part of the Cisco Server Fabric Switching, the Cisco SFS HCA allows you to create a unified
fabric for high-performance, consolidated clustering, networking, and storage communications. Host
servers can support one or two HCAs, depending on your host type.
The Cisco HCA can run bootable firmware, which allows you to use the Boot over InfiniBand feature.
For detailed information about Boot over InfiniBand (BoIB), refer to the Cisco SFS Boot over InfiniBand
User Guide for Linux, the Cisco SFS Boot over InfiniBand User Guide for Windows, or the Cisco
OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for Linux.
HCA Hardware
The HCA is a PCI-X or PCIe adapter card that provides 4x InfiniBand connections.
The InfiniBand copper connectors provide 10-Gbps SDR connections and 20-Gbps DDR connections
per port in each direction (20-Gbps aggregate bandwidth) and perform “network processing” in
hardware, freeing up the CPU. The HCA and associated protocol drivers are designed to run in
conjunction with server fabric switches. The HCA features a full suite of ULPs and APIs. APIs refer to
the software networking layer in the user space.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
1-1
Chapter 1
About the Host Channel Adapter
Supported Protocols
REVIEW DRAFT—CISCO CONFIDENTIAL
Figure 1-1 shows a Cisco HCA with InfiniBand ports and copper connectors that plug into a PCI slot.
Host Channel Adapter
182261
Figure 1-1
Note
Your HCA model may vary from this representation.
Firmware
The HCA capability can change based on the type of firmware installed on your HCA hardware. For
example, an HCA card shipped with standard firmware can be upgraded to bootable firmware if you have
purchased the appropriate license. Firmware upgrades occur automatically when drivers are installed,
but you can upgrade firmware independently. When you download new firmware, both the old and the
new versions remain on the card. The old firmware remains as a safety measure should something
interrupt the upgrade process. Use the tvflash command to update to the new firmware. (For detailed
information about the command line options, refer to the Cisco SFS InfiniBand Host Drivers User Guide
for Linux, the Cisco SFS InfiniBand Host Drivers User Guide for Windows, or the Cisco OpenFabrics
Enterprise Distribution InfiniBand Host Drivers User Guide for Linux.)
HCA Host Drivers
The HCA drivers are the software programs that allow applications to use the InfiniBand fabric. For
configuration details, refer to the Cisco SFS InfiniBand Host Drivers User Guide for Linux, the Cisco
SFS InfiniBand Host Drivers User Guide for Windows, or the Cisco OpenFabrics Enterprise Distribution
InfiniBand Host Drivers User Guide for Linux.
Supported Protocols
The term protocol refers to software in the networking layer in kernel space. The protocols documented
in this guide are described in this section. (For information about how these protocols are used with other
Cisco products, refer to the documents listed in the “Related Documentation” section on page -12.)
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
1-2
OL-12622-02
Chapter 1
About the Host Channel Adapter
Supported APIs
REVIEW DRAFT—CISCO CONFIDENTIAL
IPoiB
IPoIB protocol passes IP traffic over the IB network. Configuring IPoIB requires similar steps to
configuring IP on an Ethernet network. SDP relies on IPoIB to resolve IP addresses.
To configure IPoIB, you assign an IP address and subnet mask to each IB port. IPoIB automatically adds
IB interface names to the IP network configuration. To configure “IP over IP Protocol,” refer to the Cisco
SFS InfiniBand Host Drivers User Guide for Linux, the Cisco SFS InfiniBand Host Drivers User Guide
for Windows, or the Cisco OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for
Linux.
SDP (Linux only)
SDP is an IB-specific upper-layer protocol. It defines a standard wire protocol to support stream sockets
networking over IB. SDP enables a sockets-based application to take advantage of the enhanced
performance features provided by IB and achieves lower latency and higher bandwidth than IPoIB
running sockets-based applications. It provides a high-performance data transfer protocol for
stream-socket networking over an IB fabric. You can configure the driver to automatically translate TCP
to SDP based on a source IP, a destination, or an application name. For configuration instructions, refer
to the Cisco SFS InfiniBand Host Drivers User Guide for Linux.
SRP
SRP runs SCSI commands across RDMA-capable networks so that InfiniBand hosts can communicate
with Fibre Channel storage devices and IB-attached storage devices. SRP requires a server fabric switch
with a Fibre Channel gateway to connect the host to Fibre Channel storage. In conjunction with a server
switch, SRP disguises IB-attached hosts as Fibre Channel-attached hosts. The topology transparency
feature lets Fibre Channel storage communicate seamlessly with IB-attached hosts (known as SRP
hosts). For configuration instructions, refer to the Cisco SFS InfiniBand Host Drivers User Guide for
Linux or the Cisco SFS InfiniBand Host Drivers User Guide for Windows.
Supported APIs
The term API refers to software in the networking layer in user space. The APIs documented in this guide
are described in this section.
uDAPL (Linux only)
uDAPL defines a single set of user-level APIs for all RDMA-capable transports. The uDAPL mission is
to define a transport-independent and platform-standard set of APIs that takes advantage of RDMA
capabilities, such as those present in InfiniBand. For more information, refer to the Cisco SFS InfiniBand
Host Drivers User Guide for Linux.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
1-3
Chapter 1
About the Host Channel Adapter
Supported Operating System Software
REVIEW DRAFT—CISCO CONFIDENTIAL
MPI
MPI is a standard library functionality in C, C++, and Fortran that can be used to implement a message
passing program. MPI allows the coordination of a program running as multiple processes in a
distributed memory environment. For more information about setting up and configuring MPI, refer to
the Cisco SFS InfiniBand Host Drivers User Guide for Linux or the Cisco SFS InfiniBand Host Drivers
User Guide for Windows.
Supported Operating System Software
Cisco InfiniBand Host Channel Adapters can be use with either Linux or Windows operating system
software.
Linux Kernels
For the latest list of supported kernels and system architecture, go to the Cisco software center at this
URL:
http://www.cisco.com/cgi-bin/tablebuild.pl/sfs-linux
Refer to the Release Notes for Linux Host Drivers for more information.
Windows
Windows support includes Windows 2003, Windows XP, and Windows 2000, all in 32-bit and 64-bit
operation. For Windows host driver software packages, go to the Cisco software center at this URL:
http://www.cisco.com/cgi-bin/tablebuild.pl/sfs-win2K
Refer to the Release Notes for Windows Host Drivers for more information.
About Boot over InfiniBand Functionality
The HCA can run bootable firmware, which allows you to use the Boot over InfiniBand feature. For
detailed information about Boot over InfiniBand, refer to the Cisco Boot over InfiniBand User Guide for
Linux or the Cisco Boot over InfiniBand User Guide for Windows.
Hardware Specifications
For information about hardware specifcations, including general, physical, and environmental
specifications, see the Host Channel Adapter data sheet at this URL:
http://www.cisco.com/en/US/products/ps6428/products_data_sheets_list.html
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
1-4
OL-12622-02
C H A P T E R
2
Installing the Host Channel Adapter
This chapter explains how to select the slot for hardware installation, install the hardware, and download
and install the host driver software:
•
Selecting the PCI Slot, page 2-1
•
Installing an HCA, page 2-4
•
Installing the Host Drivers, page 2-7
Selecting the PCI Slot
This installation supports two types of connectors: PCI-X and PCIe.
•
PCI-X, page 2-1
•
PCI Express, page 2-2
PCI-X
When determining which PCI-X slot to use, inspect the server chassis and follow the recommendations
in this section.
Speed of the Slot
For maximum performance, 133 MHz PCI-X is required. While 100 MHz and 66 MHz can be used, they
are not recommended.
Locate the 133 MHz PCI-X (64-bit, 3.3 V) or 100 MHz PCI-X (64-bit, 3.3 V) slots. A conventional PCI
64-bit connector is not recommended as the first option, but it is supported.
Other Devices on the Bus
We recommend that you select a connector that is the only connector on that particular PCI-X bus, which
is most often the case for the 133 MHz connectors.
Refer to the motherboard (server) documentation to obtain a block diagram of all available PCI-X buses.
This diagram will help you determine which connectors belong to which buses. Contact your server
vendor technical support for additional assistance.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
2-1
Chapter 2
Installing the Host Channel Adapter
Selecting the PCI Slot
REVIEW DRAFT—CISCO CONFIDENTIAL
If two connectors (or more) are on the same PCI-X bus, be sure to remove all other devices from this
bus. Having another device on the same PCI-X bus can adversely affect performance. However, if
performance is not a concern, and the frequency of the PCI-X bus is 100 MHz, it is permissible to have
two devices (for example, an InfiniBand HCA and GE NIC) on the same bus.
Note
If the bus operates at 133 MHz, you must remove any other devices so that the InfiniBand HCA is the
only device on that bus.
Cooling
Most HCAs have passive cooling only, with no extra fans installed on the board. You must arrange for
suitable airflow to go around the HCA heat sink, which may require that you choose PCI-X slots that
place the HCA away from another card. Some server chassis vendors provide extra fan assemblies.
Install them if they are available.
Physical Stability of the Installation
When selecting the PCI-X slot, consider whether or not the HCA can be installed securely. Movement
from a poorly secured HCA or from improperly secured cables can damage the PCI-X connectors.
PCI-X Frequency Configuration
You should verify the PCI-X frequency configuration. Some motherboards have jumper configurations
for the PCI-X frequency. Check the motherboard documentation to verify that the frequencies are set to
133 MHz or 100 MHz. Some motherboards are PCI-X frequency-configurable through the CMOS BIOS
setup, and some provide jumpers and CMOS configuration.
Dual HCA Installation Requirements
Consider these points when installing dual HCAs:
•
For dual HCA installation in a single host, you must have two completely isolated PCI-X buses.
•
If the host has only one PCI-X 100 MHz or 133 MHz bus (regardless of the number of connectors),
then this motherboard should not be used for a dual HCA installation.
•
You may have one of the PCI-X slots operate at 133 MHz and the other at 100 MHz; however, the
best case is to have two individual 133 MHz connectors on two isolated PCI-X buses.
•
Systems with one 133 MHz connector and one 66 MHz connector are suitable for dual HCA
installations.
PCI Express
PCIe x8, x16, and x4 connectors can be used to install an HCA. PCIe x1 connectors should not be used,
even if it is possible mechanically.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
2-2
OL-12622-02
Chapter 2
Installing the Host Channel Adapter
Selecting the PCI Slot
REVIEW DRAFT—CISCO CONFIDENTIAL
Type of PCIe Connector
Using an x8 Connector for the HCA
Note
An x8 connector is recommended when you use a DDR HCA.
The HCA can be used at the maximum bandwidth when plugged into an x8 connector. If available
bandwidth is an issue, you should use the motherboard documentation to verify that the connector is
actually x8 and is supported by the BIOS as x8. This verification is important because some vendors use
x8 connectors for x4.
If your server has three x8 PCIe connectors, one of them is likely an x4 connector. The HCA can operate
with the x4 connector, but it functions with less of the possible bandwidth.
Some systems route only 4 lanes to a slot with an x8 connector. If an HCA with an x8 connector is
plugged into such a slot, then the PCIe link will “down-shift” and negotiate an x4 link.
Some early versions of the PCIe motherboards had problems with one of the PCIe connectors. If you
encounter problems when using the HCA in one of the PCIe connectors, it might help to move the HCA
to a different PCIe connector.
Using an x16 Connector for the HCA
The HCA will operate at the maximum bandwidth when plugged into an x16 PCIe connector.
Note
The x16 PCIe connector on some motherboards is only x16 in one direction; the other direction could
be x1 or x4. This asymmetrical configuration is not suitable for the InfiniBand HCA and should be
avoided.
Using an x4 Connector for the HCA
The HCA provides less bandwidth when plugged into an x4 connector.
If you are installing a single HCA in a system with dual PCIe connectors of x8 and x4, you should
differentiate the connectors and use the x8 PCIe connector for maximum bandwidth. However, you may
use the second x4 PCIe connector for the second InfiniBand HCA, provided that bandwidth is not an
issue.
Cooling
Most HCAs have passive cooling only, with no extra fans installed on the board. You must arrange for
suitable airflow to go around the HCA heat sink, which may require that you choose PCIe slots that place
the HCA away from another card. Some server chassis vendors provide extra fan assemblies. Install them
if they are available.
Physical Stability of the Installation
When selecting the PCIe slot, consider whether or not the HCA can be installed securely. Movement
from a poorly secured HCA or from improperly secured cables can damage the PCIe connectors.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
2-3
Chapter 2
Installing the Host Channel Adapter
Installing an HCA
REVIEW DRAFT—CISCO CONFIDENTIAL
Installing an HCA
Note
The HCA comes preconfigured. You do not need to set jumpers or connectors.
To install the HCA, follow these steps:
Step 1
Locate and write down the Global Unique ID (GUID) numbers from the underside of the HCA hardware.
You will need this number when performing configurations.
Your GUID follows this format: 00:05:ad:00:00:00:02:40
Step 2
Log on to the host system as the root user (Linux) or administrator user (Windows).
Step 3
Power-down the host system.
Step 4
Disconnect the power cord.
Caution
Step 5
Disconnecting the power cord is an important step because serious damage could result if the standby
power is accidently powered on during the HCA installation.
Ground yourself to an approved ground location by wearing an ESD-preventive wrist strap.
Caution
The wrist strap and clip must be used correctly to avoid ESD damage to the card. Verify that the strap
makes firm contact with your skin and that the resistance value of the strap is between 1 and 10 Mohm.
Step 6
Remove the host-system cover to access the PCI-X or PCIe slots. Figure 2-1 shows the host chassis with
the cover removed and the PCI slot exposed and circled.
PCI Slots in Host Chassis
182229
Figure 2-1
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
2-4
OL-12622-02
Chapter 2
Installing the Host Channel Adapter
Installing an HCA
REVIEW DRAFT—CISCO CONFIDENTIAL
Step 7
Find a PCI slot in which to insert the HCA. (See the “Selecting the PCI Slot” section on page 2-1.)
Step 8
Slide the HCA edge connector into the PCI slot, as shown in Figure 2-2.
Note
Depending upon your host type, you might need to unscrew and lift a riser in the host chassis to gain
easy access to the PCI slot.
Slide the IB Ports into the PCI Slot
182227
Figure 2-2
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
2-5
Chapter 2
Installing the Host Channel Adapter
Installing an HCA
REVIEW DRAFT—CISCO CONFIDENTIAL
Step 9
Make sure that the card is fully seated by pushing the card gently into the slot until the connectors are
no longer visible, as shown in Figure 2-3.
Push the Edge Connector of the HCA into the PCI-X Connector
182228
Figure 2-3
Step 10
Warning
Replace the host-system access cover.
Blank faceplates and cover panels serve three important functions: they prevent exposure to
hazardous voltages and currents inside the chassis; they contain electromagnetic interference (EMI)
that might disrupt other equipment; and they direct the flow of cooling air through the chassis. Do not
operate the system unless all cards, faceplates, front covers, and rear covers are in place.
Statement 1029
Step 11
Power-up the host system.
Step 12
Install the host drivers as described in the “Installing the Host Drivers” section on page 2-7.
Step 13
Connect the InfiniBand host to the InfiniBand switch by attaching a standard 4X InfiniBand cable to the
InfiniBand port. Push the connector into the interface until you hear or feel a click.
Copper passive DDR cables are limited to 8 meters in length. If you want to use a longer SDR cable in
place of a DDR cable, the port must be configured manually to run at the SDR rate.
Note
If your host does not provide ample free space around a given InfiniBand port, double-check that your
InfiniBand cable connector engages fully. Move your connector back and forth to be sure that both sides
of the connector have locked firmly into place.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
2-6
OL-12622-02
Chapter 2
Installing the Host Channel Adapter
Installing the Host Drivers
REVIEW DRAFT—CISCO CONFIDENTIAL
Installing the Host Drivers
To use the HCA adapter, you need InfiniBand drivers and documentation that provides instructions about
how to install and use the drivers. You can download the drivers from cisco.com.
Downloading Software from Cisco.com
The Cisco website has the latest Linux and Windows host drivers, Boot over InfiniBand firmware, and
technical documentation. After receiving the Cisco InfiniBand Host Channel Adapter, download the
latest required software and related documentation from the Cisco Systems website.
To download software and obtain documentation, follow these steps:
Step 1
Register at the Cisco.com registration site at this URL:
http://tools.cisco.com/RPF/register/register.do
By registering, you obtain a username and Cisco.com password.
Step 2
Download the software, based upon your specific requirement, as follows:
•
For Linux host driver software packages, go to this URL:
http://www.cisco.com/cgi-bin/tablebuild.pl/sfs-linux
•
For Windows host driver software packages, go to this URL:
http://www.cisco.com/cgi-bin/tablebuild.pl/sfs-win2K
Use your username and Cisco.com password to access the above URLs.
Step 3
Obtain related documentation at this URL:
http://www.cisco.com/en/US/products/ps6428/tsd_products_support_series_home.html
The following documents are available:
•
Cisco SFS Boot over InfiniBand User Guide for Linux
•
Cisco Host Drivers User Guide for Linux
•
Cisco Host Drivers User Guide for Windows
•
Cisco OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for Linux
•
Release Notes for Cisco OFED
•
Release Notes for Linux Host Drivers
•
Release Notes for Windows Host Drivers
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
2-7
Chapter 2
Installing the Host Channel Adapter
Installing the Host Drivers
REVIEW DRAFT—CISCO CONFIDENTIAL
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
2-8
OL-12622-02
CH A P T E R
3
Replacing a Tall Bracket with a Short Bracket
This chapter shows how to remove a tall bracket from an HCA expansion card and replace it with a short
bracket (see Figure 3-1), and it contains these sections:
Figure 3-1
•
Required Tools, page 3-2
•
Removing a Tall Bracket, page 3-2
•
Attaching a Short Bracket, page 3-4
HCA Expansion Card Brackets
HCA Expansion Card—Short Bracket
182511
182505
HCA Expansion Card—Tall Bracket
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12662-02
3-1
Chapter 3
Replacing a Tall Bracket with a Short Bracket
Required Tools
REVIEW DRAFT—CISCO CONFIDENTIAL
Required Tools
You will need these tools to perform the procedures in this chapter:
•
grounding wrist strap
•
anti-static mat or anti-static foam
•
small, flat-blade screw driver
•
torque screwdriver
Removing a Tall Bracket
Caution
Before performing this procedure, ground yourself appropriately to an approved grounding location by
wearing an ESD-preventive wrist strap. The wrist strap and clip must be used correctly to avoid ESD
damage to the card. Verify that the strap makes firm contact with your skin and that the resistance value
of the strap is between 1 and 10 megohms (Mohm).
To remove a tall bracket from an HCA expansion card, follow these steps:
Step 1
Place the HCA expansion card on an anti-static mat or on anti-static foam.
Step 2
With the front of the bracket facing you, locate the connector retention clip that embraces the top and
sides of the connector. (SeeFigure 3-2.)
Figure 3-2
Connector Retention Clip
Clip top
Clip Hook - grips onto the connector
Step 3
182506
Clip Center - embraces the connector
Using a small flat-blade screwdriver, gently push up one hook of the connector retention clip toward the
connector’s top side. (See Figure 3-3.)
Figure 3-3
Extract the Connector Clip
182507
3-2
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12662-02
Chapter 3
Replacing a Tall Bracket with a Short Bracket
Removing a Tall Bracket
REVIEW DRAFT—CISCO CONFIDENTIAL
Step 4
Gently push up the other hook on the other side of the connector, and lift the clip away from the
connector, being careful not to bend or twist the clip.
Step 5
Using a torque screwdriver, gently loosen the two screws that connect the bracket to the HCA card.
Remove the screws and washers, and retain them for installing the short bracket. (See Figure 3-4.)
Figure 3-4
Remove Bracket Screws and Washers
182508
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12662-02
3-3
Chapter 3
Replacing a Tall Bracket with a Short Bracket
Attaching a Short Bracket
REVIEW DRAFT—CISCO CONFIDENTIAL
Step 6
Grip the bracket firmly, and gently rotate it toward the component-side of the HCA, sliding the bracket
out of the connectors. (See Figure 3-5.)
Detach the Bracket
182509
Figure 3-5
Note
Be sure to use gentle force when extracting the bracket, taking care to protect the LEDs to the left of the
connector and all other components on the HCA card.
Attaching a Short Bracket
After you remove the tall bracket from the HCA card, you can attach the short bracket.
Caution
3-4
Before performing this procedure, ground yourself appropriately to an approved grounding location by
wearing an ESD-preventive wrist strap. The wrist strap and clip must be used correctly to avoid ESD
damage to the card. Verify that the strap makes firm contact with your skin and that the resistance value
of the strap is between 1 and 10 megohms (Mohm).
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12662-02
Chapter 3
Replacing a Tall Bracket with a Short Bracket
Attaching a Short Bracket
REVIEW DRAFT—CISCO CONFIDENTIAL
To attach the short bracket to the HCA card, follow these steps:
Step 1
Place the HCA expansion card on an anti-static mat or on anti-static foam.
Step 2
Gently place the bracket onto the HCA card, fitting the connectors through the bracket connector holes.
(See Figure 3-6.) Make sure that the LEDs to the left of the connector are aligned into their intended
bracket holes.
Place Short Bracket onto HCA Card
182578
Figure 3-6
Step 3
Place a washer over each of the two screws, and insert the screws into the holes to attach the bracket to
the card. (SeeFigure 3-7.) Use a torque screwdriver to apply up to 2 inch-pounds of torque to each screw.
Figure 3-7
Insert Washers and Screws
182508
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12662-02
3-5
Chapter 3
Replacing a Tall Bracket with a Short Bracket
Attaching a Short Bracket
REVIEW DRAFT—CISCO CONFIDENTIAL
Step 4
Gently push the clip over the top of the connector, and slide both clip hooks around the connector evenly.
(See Figure 3-8.) Take care not to bend or twist the clip.
Figure 3-8
Install the Connector Clip
182510
Step 5
3-6
Use a small flat-blade screwdriver to gently slide each clip hook into place on either side of the connector
base, making sure that each hook is firmly seated.
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12662-02
C H A P T E R
4
Troubleshooting the HCA Installation
This chapter describes how to troubleshoot the HCA installation and contains these sections:
•
Interpret HCA LEDs, page 4-1
•
Check the InfiniBand Cable, page 4-2
Interpret HCA LEDs
The HCA adapter card has two types of LEDs:
•
The yellow LED on the top indicates a logical link.
•
The green LED on the bottom indicates a physical link.
Figure 4-1 shows the two types of LEDs, located to the left of the InfiniBand port.
HCA LEDs
182232
Figure 4-1
Table 4-1 shows how to interpret the HCA LEDs.
Host Channel Adapter Hardware Installation Guide
OL-12622-02
4-1
Chapter 4
Troubleshooting the HCA Installation
Check the InfiniBand Cable
REVIEW DRAFT—CISCO CONFIDENTIAL
Table 4-1
HCA LED Indications
LED
Indication
Yellow LED
Indicates the current state of the link-state machine and traffic activity.
Green LED
•
The yellow LED is on only after the link is in the active state.
•
The yellow LED is off if the link-state machine cannot reach the active
state. A yellow LED that never comes up usually indicates that there is
no active Subnet Manager, which is the entity that recognizes the
presence of this node.
•
The yellow LED will blink if traffic is detected. There is some
dependency on the frequency and the traffic intensity.
Indicates the physical link status.
•
The green LED is on as soon as the cable is connected and there is a
functional InfiniBand device on the other end of the cable. This LED
should remain solid on as long as the physical connection is intact and
the device on the other end is functional.
•
The green LED is off to indicate that no physical link is detected.
•
The green LED will blink if a serious physical link problem is indicated,
such as a malfunctioning cable, an incomplete cable connection, or a
problematic HCA.
Check the InfiniBand Cable
•
Ensure that an InfiniBand cable is connected to a port on the HCA and a port on the InfiniBand
switch card. We recommend that you tug slightly on the cable to verify that it is tightly connected,
as poorly connected InfiniBand cables can cause errors that are difficult to detect.
•
If you are running the Element Manager, click the Refresh button and note if the corresponding
InfiniBand port on the Element Manager turns green. If it is green, you have a physical connection
and a logical link.
•
Check the port LEDs on the HCA. The bottom LED should be green.
•
Check the port LEDs on the InfiniBand switch. One should turn green, indicating that a physical
connection is established.
Host Channel Adapter Hardware Installation Guide
4-2
OL-12622-02
A P P E N D I X
A
Acronyms and Abbreviations
Table A-1 defines the acronyms and abbreviations that are used in this publication.
Table A-1
List of Acronyms and Abbreviations
Acronym
Expansion
API
Application Program Interface
BIOS
basic input/output system
BoIB
Boot over InfiniBand
CLI
command-line interface
CMOS
configuration memory operating system
DDR
Double Data Rate
GUI
graphical user interface
GUID
globally unique identifier
HCA
Host Channel Adapter
IB
InfiniBand
IPoIB
Internet Protocol over InfiniBand
ITL
Initiator/Target/LUN
LED
light emitting diode
LFM
linear feet per minute
LU
logical unit
LUN
logical unit number
MPI
Message Passing Interface
OFED
OpenFabrics Enterprise Distribution
PCI
protocol control information
PCIe
PCI Express
SAN
Storage Area Network
SCSI
Small Computer System Interface
SDP
Sockets Direct Protocol
SDR
Single Data Rate
Cisco OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for Linux
OL-10778-02
A-1
Appendix A
Table A-1
Acronyms and Abbreviations
List of Acronyms and Abbreviations (continued)
Acronym
Expansion
SFS
Server Fabric Solution
SRP
SCSI RDMA Protocol
RDMA
Remote Direct Memory Access
uDAPL
User Direct Access Programming Library
ULP
upper-layer protocol
Cisco OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for Linux
A-2
OL-10778-02
I N D EX
Numerics
conventions
1-6
organization
1-5
100 MHz PCI-X
2-1
related
133 MHz PCI-X
2-1
documentation
1-12
obtaining
1-12
dual HCA installation
A
acronyms and abbreviations table
APIs
supported
1-3
A-1
H
hardware specifications
audience, document
1-5
1-1
data sheet
B
Boot over InfiniBand
bracket
tall, removing
firmware
1-2
hardware
1-1
1-2
HCA, installing
2-4
HCA, LEDs
3-4
1-4
host drivers
1-4
short, attaching
1-4
HCA
about
functionality
2-2
4-1
host drivers, installing
3-2
C
2-7
I
cable, InfiniBand
installation stability
4-2
connector retention clip
extracting
installing
IPoIB
2-2, 2-3
1-3
3-2
3-6
conventions, document
cooling requirements
1-6
2-2
L
LEDs
4-1
Linux kernels
1-4
list of supported protocols
D
1-2
document
audience
1-5
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
OL-12622-02
IN-1
Index
M
U
MPI
uDAPL
1-4
O
W
operating system software
warning
supported
1-4
organization, document
1-5
1-3
definition
1-7
statement
1-7
statement,translations
Windows
P
1-7
1-4
PCIe
connector types
2-3
PCI Express
selecting the connector
PCI slot
2-2
2-1
PCI-X
speed
2-1
protocols
supported
1-2
R
requirements
dual HCA install
2-2
S
SDP
1-3
service requests
specifications
SRP
1-12
1-4
1-3
T
troubleshooting
4-1
Cisco InfiniBand Host Channel Adapter Hardware Installation Guide
IN-2
OL-12622-02