IBM SAN Volume Controller Physical
Installation Racking Guidelines V1.8
April 2013
Author/Owner: Bill Turnau - Sr IT Architect/PMP®/IBM Certified Sr PM, IBM GTS
DCMO Specialty Services Area
Contributing Author: Chuck Laing – Sr IT Architect/Open Group Master Certified
IT Specialist/IBM Sr Technical Staff Member, IBM GTS SO Delivery
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
SVC Node Physical Structure 101
SVC clusters must have redundancy and are used in critical environments,
therefore being treated with care.
An SVC node actually consists of up to 3 physical 1U hardware components
depending upon SVC model.
– A 2145-xxx SVC node with one or 2 onboard power supplies.
– An 8115-UPS (Uninterruptible Power Supply) with the main purpose is
to protect in-cache data for up to a maximum of 10 seconds in cases of
brown-outs or power interrupts to the individual SVC node ans also
allow storing cache and system state data to disk if extended power
failure. There is a special power cord/serial cord set that runs from the
SVC node power supplies & serial port to same on the 8115-UPS.
– An FC 8300 Redundant ac-Power Switch* (RPS) with a main power
cord input on the left side (labeled ‘Main’) and a 2nd power cord input on
the right (labeled ‘Backup’). Depending upon whether the SVC node is
the A or B node in an IO Group will dictate whether the power cord to
the ‘main’ hole will go to the left or right in-Cabinet PDU. There is in
turn an output(s) on the opposite end of the switch from which a power
cord gets run up to the UPS unit power inlet.
SVC IO Group Structure
– SVC nodes are installed in pairs (SVC Node A & SVC Node B)
– Each A & B node as a high availability pair together is considered an IO
Group
– A SAN cluster can range from 2 to 8 SVC nodes and therefore from 1 to
4 IO Groups.
– An IO Group will continue to operate if one of the nodes loses power,
therefore it is important to connect the different nodes in the IO group to
different power sources if possible.
– Nodes are not assigned identity as nodes in an IO Group until the
system is logically configured so it is critical that the physical
installer provide the precise layout info (RU location, 2145 serial #
& node ID) to the person performing the logical configuration.
This is so that the logical configuration will match the physical
layout.
* Note: The Redundant ac-Power Switch (RPS) aka Feature Code 8300
(FC 8300) was not available on earlier models of SVC nodes. In the rare
case where you may be re-using/installing a much older SVC model then
you may not have this component. A cabinet elevation having only the
2145 SVC nodes and UPS units has been included for install guidance.
2
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
Planning for SVC Cluster Installation - Single Cabinet vs.
Dual Cabinets
Choosing whether to install all 8 nodes of an SVC cluster in a single cabinet or to split
them across 2 separate cabinets (A Nodes in one and B Nodes in another) requires
consideration of your overall Storage environment you are implementing as well as the
facilities resiliency of the data center(s) you are installing your Storage environment into.
These guidelines are for a system installed in a single machine room (You can also further
increase availability by splitting the system between rooms, but that requires careful design of the
data system as a whole, including host configuration, storage system configuration, fabric design
and SVC configuration - which is beyond the scope of this document).
Data Center Facilities Aspects
–
Upstream redundancy of the power to your cabinet (power circuit panels and on-floor PDUs)
and within cabinet power redundancy (dual power strips or in-cabinet PDUs) as well as
upstream high availability structures (UPS, generators, etc.) will feed into your decision
–
Many data centers today are at an Uptime Tier 3 or higher level so power redundancy
concerns that would require a dual cabinet SVC implementation are not there
–
Fire Protection Systems Type (such as overhead wet pipe sprinkler systems, etc.) however
should be taken into consideration
Physical Separation and location of other key Storage environment components
–
Backend Disk Storage
• If you are implementing a single SVC cluster attached to a single backend storage device or
string (XIV frame or DS8K string) then separating your SVC cluster across 2 cabinets is not
any more advantageous to your overall Storage environment than installing it in a single
cabinet in case of a facilities problem such as breakage of an overhead wet pipe sprinkler
head
3
–
SAN Directors (aka switches)
• If you have your SAN director pair that all your front-end devices (servers, etc.) and your
back-end devices (SVCs, disk stores, VTLs, etc.)access installed together in the same
cabinet then again separating your SVC cluster across 2 cabinets is not any more
advantageous to your overall Storage environment than installing it in a single cabinet in
case of a facilities problem such as breakage of an overhead wetpipe sprinkler head.
–
Physical Separation Distance
• If you are implementing your entire Storage environment with multiple redundant devices
physically separated across multiple cabinets and strings you need to ensure sufficient
enough physical distance to ensure that your redundant components are in different fire
protection zones or power sourced zones. Otherwise your end to end Storage environment
can be compromised in cases of a zonal facilities failure.
In summary, when making the choice between a single or dual cabinet SVC cluster installation
approach you need to factor in all aspects of the data center your are installing into as well as the
high availability design characteristics of your overall Storage environment
–
If the data center you are boarding into has the proper power redundancy attributes & your
Storage environment design strategy does not have fully redundant components placed at
sufficient distances apart then a single cabinet implementation will save you the costs
associated with the 2nd cabinet.
–
If the data center does not have a robust enough power redundancy infrastructure and/or your
Storage environment design strategy does have fully redundant components place at
sufficient distances apart then the investment in a dual cabinet implementation is certainly
justified in furthering the level of high availability/redundancy of your overall Storage
environment.
–
Another consideration would be that if you anticipate that you will be adding another SVC
cluster to your Storage environment in the future, by going with a dual cabinet approach from
the start and reserving remaining space in each cabinet for nodes from the 2nd cluster you
accomplish both objectives. In the dual cabinet elevations shown in later pages you will see
that we have included the space allocations for a future 2nd cluster (CL2) to be added to the
cabinets.
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
Planning for SVC Cluster Installation – Choosing the
Right Cabinet
Choosing the right cabinet to install your SVC cluster into is
an equally important aspect.
IBM SVC nodes are specific purpose built devices using the
IBM x3550 server as the base hardware platform
– These SVC nodes along with their associated UPS unit and
Redundant ac-Power Switch (RPS) can be installed into any
IBM or 3rd party cabinet that servers can be installed in
Consult with the staff of the datacenter you will be installing
your SVC cluster into to see if they have a cabinet type that
they have set as the standard for their datacenter
– If they have a cabinet standard you will want to follow that
standard
– If they don’t have a standard cabinet type in use you can
source a cabinet of your choice that is agreeable to them (IBM
or 3rd party) as long as the cabinet heights at a minimum are
homogenous with other cabinets already in their data center
where possible
Default height for cabinets used to house SVC clusters
should be at least 42U unless height clearances in the data
center you are boarding into dictate a shorter cabinet be used
– Remember that any cabinet used needs to not only factor in
the RUs needed for the actual hardware (including spacing
between SVC nodes) but also room for structured data wiring
patch panels and wire managers, etc.
– If an IBM cabinet is OK with the staff in the data center where
the SVCs are to be installed then an IBM 7014-T42 might be
an acceptable 42U high choice
4
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
SVC Cluster Single Rack Elevation (with redundant ac-power switch)
5
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
SVC Cluster Rack Elevations (for older models without the redundant
ac-power switch)
6
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
SVC Cluster Dual Rack Elevation (with redundant ac-power switch)
7
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
Rack Elevation Order Rationale
The physical hardware is to be racked with the components associated with
the lowest numbered SVC node starting at the bottom of the rack/cabinet
(Note: The SVC GUI logical depiction shows a top to bottom order but the
best practices physical racking order to be followed is bottom to top)
While the 1U UPS units are the heaviest devices associated with an SVC
node, where the redundant ac-power switch is used it was chosen to put
these switches in the bottom most position in the rack for the following
reasons:
– There are power cords required to be plugged into both the front and the
back of the redundant ac-power switch so having them down at bottom
of the rack where they are more protected is the wiser choice
– The UPS units are still within the bottom half of the rack (to prevent risk
of tipping) with the bonus of them being at a level where their displays
can be more easily seen
– Ensures that there is no issue with the length of the special power/serial
cable between the UPS units and their respective 2145 SVC units
The components in either bright green or light green are the first SVC node
set in an IO Group node pair.
– Their UPS unit or redundant ac-power switch unit (‘main’ outlet) are to
be connected to left side (when facing back of rack) power ePDU.
– Their ‘backup’ outlet on the ac-power switch gets connected to the right
side ePDU.
The components in either red or pink are the second SVC node set in an IO
Group node pair.
– Their UPS unit or redundant ac-power switch unit (‘main’ outlet) are to
be connected to right side (when facing back of rack) power ePDU.
– Their ‘backup’ outlet on the ac-power switch gets connected to the left
side ePDU.
As SVC cluster environments can be scaled from 2 to 8 nodes, a default of 4
nodes minimum were assumed for the color schema. The purpose being that
even though you may only be starting with a 2 or 4 node SVC cluster, it is
very highly recommended that you plan your space for a full 8 node cluster
to prevent ugliness of having to find space later when additional nodes and
their components have to be added.
8
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
Examples of a 4 node cluster SVC Cluster Power Cabling Flow
Diagrams (with mandatory Redundant ac-Power Switches)
Please ensure that the power connection from each rack PDU goes to the Main receptacle on the acSwitch for one node in the IO Group pair and the Backup receptacle on the ac-Switch of the 2nd node in
the IO Group pair. And vice versa for the 2nd rack PDU source.
The excerpted examples below unfortunately don’t clearly depict the alternating connections. Please refer
to the SVC Cluster Power Cabling Physical Diagrams a few pages further for a better depiction.
For use with 2 x 60-amp Rack PDU
Implementations
Excerpted
from
WP102010
SAN
Volume
Controller –
Power
Cabling
Options &
Connector
Types for
Node
models CF8
& CG8
Author/Own
er R B
Smith
9
For use with 4 x 30-amp Rack PDU
Implementations
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
Examples of a 4 node cluster SVC Cluster Power Cabling Flow
Diagrams (older models w/o Redundant ac-Power Switches
1
0
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
SVC Cluster Power
Cabling Physical
Diagrams (single
cabinet)
IBM RPS Unit Info
The picture below is of the RPS (Redundant ac-Power Switch),
IBM Feature Code 8300, that is included as part of newer SVC
clusters. In more general terms this RPS unit is actually an
Automatic Transfer Switch (ATS).
All SVC Cluster Nodes in a Single Cabinet
10U Reserved (or 8U if 42U cabinet) for data
wiring patch panels
or filler panels
The picture orientation is looking at an actual installed RPS
from the front of the cabinet.
You can see that the RPS is attached to its brackets and
mounted on the cabinet back rails to ensure the RPS ‘main’ &
‘backup’ receptacles are facing the back of the cabinet. The
‘power out’ receptacles that are for connection to the UPS unit
should be facing the front of the cabinet.
SVC #8 IOGroup 3 Node B
AC
DC
Power cord from
secondary ePDU (right
side for odd nodes / left
side for even nodes)
gets plugged into Main
power inlet on back
side of RPS unit
Power cord from primary
ePDU (left side for odd
nodes / right side for
even nodes) gets
plugged into Main power
inlet on back side of this
RPS unit
AC
DC
1U Filler Panel
SVC #7 IOGroup 3 Node A
AC
AC
DC
DC
1U Filler Panel
SVC #6 IOGroup 2 Node B
AC
AC
DC
DC
1U Filler Panel
SVC #5 IOGroup 2 Node A
AC
DC
AC
DC
1U Filler Panel
SVC #4 IOGroup 1 Node B
AC
AC
DC
DC
AC
AC
DC
DC
1U Filler Panel
SVC #3 IOGroup 1 Node A
1U Filler Panel
SVC #2 IOGroup 0 Node B
Power Cord from UPS gets plugged
into Power Out 1 on front side of this
RPS unit
AC
AC
DC
DC
AC
AC
DC
DC
1U Filler Panel
SVC #1 IOGroup 0 Node A
1U Filler Panel
ATT
ENTI
CONNECT ONLY IBM SAN VOLUME
CONTROLLERS TO THESE OUTLETS.
ON
SEE SAN VOLUME CONTROLLER
INSTALLATION GUIDE.
12
ATT
ENTI
CONNECT ONLY IBM SAN VOLUME
CONTROLLERS TO THESE OUTLETS.
ON
SEE SAN VOLUME CONTROLLER
INSTALLATION GUIDE.
12
ATT
ENTI
CONNECT ONLY IBM SAN VOLUME
CONTROLLERS TO THESE OUTLETS.
ON
SEE SAN VOLUME CONTROLLER
INSTALLATION GUIDE.
12
ATT
ENTI
CONNECT ONLY IBM SAN VOLUME
CONTROLLERS TO THESE OUTLETS.
ON
SEE SAN VOLUME CONTROLLER
INSTALLATION GUIDE.
12
ATT
ENTI
CONNECT ONLY IBM SAN VOLUME
CONTROLLERS TO THESE OUTLETS.
ON
SEE SAN VOLUME CONTROLLER
INSTALLATION GUIDE.
12
ATT
ENTI
CONNECT ONLY IBM SAN VOLUME
CONTROLLERS TO THESE OUTLETS.
ON
SEE SAN VOLUME CONTROLLER
INSTALLATION GUIDE.
12
ATT
ENTI
CONNECT ONLY IBM SAN VOLUME
CONTROLLERS TO THESE OUTLETS.
ON
SEE SAN VOLUME CONTROLLER
INSTALLATION GUIDE.
12
ATT
ENTI
CONNECT ONLY IBM SAN VOLUME
CONTROLLERS TO THESE OUTLETS.
ON
SEE SAN VOLUME CONTROLLER
INSTALLATION GUIDE.
12
1U Filler Panel
1U Filler Panel
1
1
Left Side
ePDU
Main
Input
Backup
Input
Right Side
ePDU
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
SVC Cluster Power Cabling Physical Diagrams (dual
cabinet)
Node B Cabinet
10U Reserved (or 8U if 42U cabinet) for data
wiring patch panels
or filler panels
CL2 SVC#8 IOGroup 3 Node B
AC
DC
AC
DC
1U Filler Panel
CL2 SVC #6 IOGroup 2 Node B
AC
AC
DC
DC
1U Filler Panel
CL2 SVC #4 IOGroup 1 Node B
AC
AC
DC
DC
AC
AC
1U Filler Panel
CL2 SVC #2 IOGroup 0 Node B
DC
DC
1U Filler Panel
SVC#8 IOGroup 3 Node B
AC
AC
DC
DC
1U Filler Panel
SVC #6 IOGroup 2 Node B
AC
DC
AC
DC
1U Filler Panel
SVC #4IOGroup 1 Node B
AC
DC
AC
DC
1U Filler Panel
SVC #2 IOGroup 0 Node B
AC
AC
DC
DC
1U Filler Panel
ATT
ENTI
CON NEC T ONL Y IBM SAN VOL UME
CONTROL LER S TO TH ESE OUTLETS.
ON
SEE SAN VOLU ME CONTROL LER
INS TAL LATION GU IDE.
12
ATT
ENTI
CON NECT ONL Y IBM SAN VOL UME
C ONTROLL ERS TO TH ESE OU TLETS.
ON
SE E SA N VOLUM E C ONTROLL ER
INSTAL LATION GU ID E.
12
ATT
ENTI
CON NECT ONL Y IBM SAN VOL UME
C ONTROLL ERS TO TH ESE OU TLETS.
ON
SE E SA N VOLUM E C ONTROLL ER
INSTAL LATION GU ID E.
12
ATT
ENTI
CON NECT ONL Y IBM SAN VOL UME
C ONTROLL ERS TO TH ESE OU TLETS.
ON
SE E SA N VOLUM E C ONTROLL ER
INSTAL LATION GU ID E.
12
ATT
ENTI
CON NECT ONL Y IBM SAN VOL UME
C ONTROLL ERS TO TH ESE OU TLETS.
ON
SE E SA N VOLUM E C ONTROLL ER
INSTAL LATION GU ID E.
12
ATT
ENTI
CONN ECT ONL Y IBM SAN VOL UME
C ONTROLL ERS TO TH ESE OU TL ETS.
ON
SEE SAN VOL UME C ON TROLL ER
INSTAL LATION GU ID E.
12
ATT
ENTI
CON NEC T ONL Y IBM SAN VOL UME
CONTROL LER S TO TH ESE OUTLETS.
ON
S EE SAN VOLU ME C ONTROL LER
INSTAL LATION GU IDE.
12
ATT
ENTI
CON NEC T ONL Y IBM SAN V OL UME
C ONTROLL ERS TO TH ESE OU TLETS.
ON
S EE S AN VOLU ME C ONTROLL ER
INSTAL LATION GU IDE.
12
1U Filler Panel
1U Filler Panel
Left Side
ePDU
1
2
Main
Input
Backup
Input
Right Side
ePDU
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
Other SVC Node Racking Pearls of Wisdom
Hold an SVC Assurance & Installation Prep Meeting in advance of any install activities
– Have the cabinet elevation template and data cabling & power cord layouts ready to
be reviewed as part of this meeting
– Participants should be CEs, Data Wiring vendor, Storage Tower lead or architect,
Physical Architect (aka PADSMRT PM/Arch or comparable) and optionally the local
data center hardware planner as needed.
Use IBM CEs for initial installations of SVC equipment
– SVCs are classified as ‘CE installable’ and therefore you are entitled (and paid for)
installation by an IBM CE when you bought the units. So use them vs. a Data wiring
vendor to rack & stack SVCs.
– Educate the CEs on any cable dressing best practices to ensure tidy installation of
power cords, etc. Make sure that they also install the cable retainers on the nodes
and UPSs"
– As redundant ac-power switches ‘main’ and ‘backup’ labels are on top of device
please place additional labels on front plate of switches for these inlets and also to
indicate what SVC Node this ac-power switch is for.
– Also take care to ensure that when attaching the mounting brackets to ac-power
switch that the ‘Main’ & ‘Backup’ power inlet side to face the back of the cabinet and
the outlet to UPS unit side to be facing the front of the cabinet/rack.
– Ensure the UPS units are labeled with the SVC node they connect to.
Ensure that the CE follows the provided cabinet elevation template and racks the units
starting from the bottom of their respective sections in the rack (i.e. SVC Node #1
components all in their lowest respective RU locations) in the following order. The IBM
Install Guide for SVCs should also provide CE with instructions on which device
component to power up/test first etc.
1. Redundant AC-Power Switches (if applicable)
2. UPS Units
3. 2145 SVC nodes
Data Wiring vendor QA review post-CE install
– Have data wiring vendor assure and then fasten down all power cabling to best
practices standards or coordinate to have Data Wiring vendor assure neatness of CE
equipment install and then work with CE to install and wrap down both power cords
and data wiring.
– Ensure CE has installed all supplied power cord clips and then reinforce with
additional tie-downs to prevent any accidently disconnect of power cords from PDUs
or SVC components.
– Ensure enough slack is left when tying down power cord to eliminate power cord
coming loose/unplugged over time due to being tied down too tight.
– As logical SVC installers are at a remote location record the serial numbers and
physical locations of all SVC components installed by CE and provide them to
Storage Architect or team lead. In today’s IT business environment of remotely
performed configurations, this information is critical to ensuring that the locations of
the logically defined SVC nodes match to the installed physical locations.
1
3
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
Wrong Way Example Rear View
2145 SVC
Node
UPS Unit
Redundant acpower switch
2145 SVC node then
UPS then redundant
AC-power switch.
Note that the UPS is
more recessed
between the SVC node
and the ac-power
switch and is virtually
un-serviceable in case
of a problem as a
result.
1
4
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
Wrong Way Example Front View
2145 SVC Node
UPS Unit
Redundant acpower switch
(behind this 1U filler
panel racked on
back rails)
6 SVC nodes racked
and stacked incorrectly
as 2145 SVC node
then UPS then
redundant AC-power
switch.
1
5
© 2013 IBM
Corporation
IBM SAN Volume Controller Physical Installation Racking
Guidelines V1.8
A Right Way Example
You will see that the RPS units are all racked at the bottom with the UPS units in the middle
and the 2145 SVCs at the top.
- The cords feeding into the right hand side of the UPS are the 2 power cords from the SVC
unit.
- The center cable is the serial cable that runs between the SVC and the UPS to keep the
units in sync for caching
- The cable on the left hand side of the UPS is the power cord that is plugged into the front of
the RPS.
- The cables running from the back of the RPS units to the in-cabinet PDUs (odds to the left
and evens to the right in this case) are the last connection.
- Note how there is 1U of space between the SVC nodes, just as shown in the elevation
diagrams, to allow for better heat dissipation and to help prevent harm to any loose or dangling
fiber cables.
Front
View
Rear
View
1
6
© 2013 IBM
Corporation