Power Systems ATS
Live Partition Mobility Update
Ron Barker
Power Advanced Technical Sales Support
Dallas, TX
© 2013 IBM Corporation
Power ATS
Agenda
 Why you should be planning for partition mobility
 What are your options? Which is best for you?
 How to set up for LPM using vSCSI
 How to set up for LPM using NPIV
 Tips that will make setting it up easier
 The mobility process – start to finish
 Questions?
2
© 2013 IBM Corporation
Power ATS
The value of partition mobility
•
Flexibility: move workloads to servers that are more ideally suited
to your needs
– Application requirements can change
– The infrastructure should adapt quickly
– Service levels must be maintained
•
Availability: move workloads off machines that are scheduled for
planned maintenance
– Hardware migrations made easier if workloads can temporarily be run
elsewhere without interruption
– Remove the chance of disruption when a system firmware upgrade or
concurrent hardware maintenance is planned
3
© 2013 IBM Corporation
Power ATS
What options are there?
•
Use redundant systems and PowerHA, bringing one server down
and moving its workload to another
– Additional hardware and software costs, plus configuration and testing
•
Schedule a maintenance window and bring down all partitions and
applications
– Very disruptive; hard to get users to agree on when a maintenance
window should be scheduled
•
Use Live Partition Mobility to migrate a partition from one server to
another without an interruption in service
– Works on any Power server with PowerVM Enterprise Edition, but
requires complete I/O virtualization
4
© 2013 IBM Corporation
Power ATS
What are your options?
•
All storage virtualization technologies support migration
– Virtual SCSI: One or a pair of Virtual I/O Servers supply the client
partition with generic hdisks virtualized from SAN storage owned by the
VIOS lpars
– N-Port ID Virtualization (NPIV): VIO server HBAs act in pass-thru mode
to connect Virtual Fibre Channel adapters to the SAN environment,
where they are zoned and mapped to LUNs owned by each client
– Shared Storage Pools: Cluster of up to 16 VIO servers share
management of a large pool (up to 512 TB) of SAN storage that is
carved into file-backed Logical Units and allocated as hdisks to clients
5
© 2013 IBM Corporation
Power ATS
Readiness Checklist
 PowerVM Enterprise Edition license on source and target
servers
 Source and target servers are POWER6 or POWER7
• If processor architectures are different, the mobile
partition must use compatibility mode
 Check matrix for firmware level compatibility
http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/index.jsp?to
pic=/p7hc3/p7hc3firmwaresupportmatrix.htm
 Both servers are HMC managed
 Logical memory block size the same on source and target
 If Active Memory Sharing is used, make sure a shared
memory pool exists on the target server
6
© 2013 IBM Corporation
Power ATS
Readiness Checklist (cont’d)
 If Active Memory™ Expansion is used, make sure it’s
supported on the destination server
 If the mobile partition is suspend-resume capable, make sure
the target has a reserved storage pool greater than or equal
to 110 percent of the lpar size
 If you are moving an IBM i mobile partition, verify that the
destination server supports the migration of IBM i mobile
partitions and the restricted I/O mode
• Also, verify that the IBM i mobile partition is in the
restricted I/O mode
7
© 2013 IBM Corporation
Power ATS
Readiness Checklist (cont’d)
 If using a shared processor pool with an entitlement below 0.1
processing units and greater than or equal to 0.05
(POWER7+), make sure the destination server also shares
that capability
 If the mobile partition is using a virtual Ethernet adapter which
is using a virtual switch in the VEPA (Virtual Ethernet Port
Aggregator) mode, or the mobile partition is using a virtual
Ethernet adapter with a VSI profile, then verify that the
destination server also supports virtual server network (VSN)
• The legacy virtual switch mode on Power is VEB, or
Virtual Ethernet Bridge, and is most common today
8
© 2013 IBM Corporation
Power ATS
Readiness Checklist (cont’d)
▪
▪
▪
▪
▪
▪
▪
9
The VIO servers must provide clients access to the same
subnet
Virtual SCSI disks must be on shared LUNs with the disk
reserve policy set to no_reserve
LUNs using NPIV need to be mapped to both world wide
port names on each client Virtual Fibre Channel adapter
All I/O resources must be shared or virtualized prior to
migration – dedicated devices need to be removed
Processors and memory may be dedicated or shared
The SAN switch must support NPIV, at least on the port to
which the VIOS physical adapter connects
Make sure the SAN switch software is at the latest
supported level
© 2013 IBM Corporation
Power ATS
Setting up a VIOS and VIO clients using vSCSI
• Size VIOS using Workload Estimator http://www912.ibm.com/estimator
• Or use the rule-of-thumb: 1.0-2.0 processing units from the
shared pool, uncapped, with 2-4 GB of memory (4 GB
Shared Storage Pool)
• 1-10 Gbps Ethernet or Integrated Multi-function Card (in
promiscuous mode for Shared Ethernet Adapter usage)
• One or more 8 Gbps dual-port Fibre Channel adapters
• Current VIOS operating system, e.g., VIOS 2.2.2.2-FP26 SP
01
• VIOS can boot from SAN, but the best practice is to boot
from internal disks
10
© 2013 IBM Corporation
Power ATS
Setting up a VIOS and VIOC using vSCSI
• Designate the VIOS as a Mover Service Partition (MSP)
• Make sure no virtual adapter is “Required”
• Enable Time reference on Settings tab, under Service and
Support
• Zone and map storage from the SAN to the VIOS world wide
port name for virtualization
• When creating the VIOC. make sure Virtual Adapters are
listed as “Desired”
• Using the HMC, carefully map slot numbers for each Server
SCSI adapter (vhost) to the slot of a corresponding Client
SCSI adapter
Go to IBM Redbooks to download IBM PowerVM Virtualization
Introduction and Overview
http://www.redbooks.ibm.com/
11
© 2013 IBM Corporation
Power ATS
Setting up a VIOS and VIOC for NPIV
• VIOS creation is the same as for vSCSI, except the physical
adapter's world wide port name is not mapped to the client's
LUNs
• The VIOS HBA is mapped to the switch so it can perform its
pass-thru function for the client VFCs
• Both world wide port names on each client VFC are zoned
and mapped to storage (one's active and one's standby)
• There's a one-to-one relationship between a Server VFC
and a Client VFC
• A Server VFC is mapped to one of the ports on a physical
HBA (64 Client VFC hosts are supported on each HBA port);
only one Client VFC is supported per physical server port
12
© 2013 IBM Corporation
Power ATS
Tips to make setup easier
 The Hypervisor will automatically manage migration of CPU
and memory, but resources must be available on the target
server
 The administrator should monitor the validation and MSP
pairing to make sure the desired VIO servers are used and
the correct MPIO mappings are maintained
 Dedicated I/O adapters, if any, must be de-allocated before
migration using DLPAR, but available dedicated I/O
adapters may be re-added on the target after the migration
13
© 2013 IBM Corporation
Power ATS
Tips to make setup easier
 For vSCSI, the reserve attribute on the physical storage
must be set to no_reserve
 For NPIV, you can set the reserve attribute to pr_shared,
but only IF you at the following levels:

HMC version 7 release 3.5.0, or later

VIOS version 2.1.2.0, or later

The physical adapters support the SCSI-3 Persistent
Reserve standard
 The reserve attribute must be the same on the source and
destination VIOS partitions for successful Partition Mobility
14
© 2013 IBM Corporation
Power ATS
Tips to make setup easier
• Where do you find the wwpns?
– On the HMC, look at the client profile and view the
details of all client VFC adapters
15
© 2013 IBM Corporation
Power ATS
Exposing all World Wide Port Names to the Switch
•
The chnportlogin and lsnportlogin commands can be used to
expose the world wide port names from the Client Virtual Fibre
Channel to the SAN administrator
$ chnportlogin -o login -m Doc -p mob76
$ lsnportlogin -m Doc --filter “lpar_names=mob76” -F
lpar_name,wwpn,wwpn_status
Where wwpn status values are:
0=wwpn is not activated
1=wwpn is activated
2=wwpn status unknown
Best practice: run
chnportlogin -o logout after
wwpns are zoned and mapped;
active wwpns will stay logged in
16
barker@mdhmc1:~> lsnportlogin -m
Doc -d 3 --filter "lpar_names=mob76" F lpar_name,wwpn,wwpn_status
mob76,c050760003ec0048,1
mob76,c050760003ec0049,1
mob76,c050760003ec0046,1
mob76,c050760003ec0047,1
barker@mdhmc1:~>
© 2013 IBM Corporation
Power ATS
Both GUI and Command Line Interfaces
17
© 2013 IBM Corporation
Power ATS
Command Line Interface -- migrlpar
 Migrations can be scripted using the migrlpar command
 Run migrlpar --help or man migrlpar to see the syntax (it’s
updated as features are added)
 You can be very specific in selecting the source and
destination MSP servers, the mapping of virtual SCSI and
virtual FC servers and slots, and other configuration details
18
© 2013 IBM Corporation
Power ATS
Command line interface – migrlpar (cont’d)
 When using virtual_fc_mappings and virtual_scsi_mappings
attribute pairings, enclose in double quotes preceded by a
backslash (\)
 New migrlpar flags:
 -- mpio [1|2] Require MPIO pathing from source be
duplicated on target; 1 (default) means require multipathing; 2 means to try, but allow regardless
 -- vlanbridge [1|2] Require target to have VLAN access to
the same external network; 1 (default) means yes; 2 says
to try but allow regardless
 -- protectstorage [1|2] When migrating a suspended
partition, 1 requires the storage be activated for the move
to prevent accidental reassignment to another partition; 2
means you assume responsibility for the safety of the
suspended partition’s storage
© 2013 IBM Corporation
Power ATS
Validation – High Level
 Capability and compatibility check
 Resource Monitoring and Control (RMC) check
 Partition readiness
 System resource availability
 Virtual adapter mapping
 Operating system and application readiness check
20
© 2013 IBM Corporation
Power ATS
Validation
Please Note: there are other validation steps, this
presentation simply shows some of the sequence.
 System Properties support Partition Mobility
•Inactive and Active Partition Mobility Capable = True
 Mover Service Partitions on both Systems
•VIO Servers with VASI device defined, and MSP enabled
21
© 2013 IBM Corporation
Power ATS
Validation Output
 Looking at a validation output screen can be confusing
because warnings are listed as well as errors
 Warnings won’t derail a migration, but errors do
 Some pervasive issue – such as incorrect mapping of virtual
adapters -- will show up multiple times, usually as the same
failure on a different object
22
© 2013 IBM Corporation
Power ATS
Sample Validation Errors
This indicates SSH has not been set up between a local
and a remote HMC in order to do a Remote HMC migration
$ mkauthkeys -g --ip <HMC_ip_address> -user
<HMC_user_for_both HMCs> --passwd <password> -t rsa
23
© 2013 IBM Corporation
Power ATS
Sample Validation Errors
This refers to RMC communication and probably can be
fixed by stopping and restarting rsct daemons on the
client partition
24
© 2013 IBM Corporation
Power ATS
Migration
 If validation passes, migration can begin
 From this point, all state changes are rolled back if an error
occurs
Mobile
Partition
1
MSP
VASI
2
POWER Hypervisor
Source System
Network
3
Mobile
Partition
MSP
VASI
4
5
POWER Hypervisor
Target System
Partition State Transfer Flow
25
© 2013 IBM Corporation
Power ATS
Migration Steps (1 of 6)
 The HMC creates a shell partition on the destination system
 The HMC configures the source and destination Mover
Service Partitions (MSP)
• MSPs connect to PHYP thru the Virtual Asynchronous
Serial Interface (VASI)
 The MSPs set up a private, full-duplex channel to transfer
partition state data
26
© 2013 IBM Corporation
Power ATS
Migration Steps (2 of 6)
 The HMC sends a Resource Monitoring and Control (RMC)
event to the mobile partition so it can prepare for migration
 The HMC creates the virtual target devices and virtual SCSI
adapters in the destination MSP
• You do NOT do this manually; it must happen under
Hypervisor control
 The MSP on the source system starts sending the partition
state to the MSP on the destination server
27
© 2013 IBM Corporation
Power ATS
Migration Steps (3 of 6)
 The source MSP keeps copying memory pages to the target
in successive phases until modified pages have been
reduced to near zero
 The MSP on the source instructs the PHYP to suspend the
mobile partition
 The mobile partition confirms the suspension by suspending
threads
28
© 2013 IBM Corporation
Power ATS
Migration Steps (4 of 6)
 The source MSP copies the latest modified memory pages
and state data
 Execution is resumed on the destination server and the
partition re-establishes the operating environment
 The mobile partition recovers I/O on the destination server
and retries all uncompleted I/O operations that were going on
during the suspension
• It also sends gratuitous ARP requests to all VLAN
adapters
29
© 2013 IBM Corporation
Power ATS
Migration Steps (5 of 6)
 When the destination server receives the last modified pages,
the migration is complete
 In the final steps, all resources are returned to the source and
destination systems and the mobile partition is restored to its
fully functional state
 The network channel between MSPs is closed
 The VASI channel between MSP and PHYP is closed
 VSCSI and VFC adapters on the source MSP are removed
30
© 2013 IBM Corporation
Power ATS
Migration Steps (6 of 6)
 The HMC informs the MSPs that the migration is complete
and all migration data can be removed from their memory
tables
 The mobile partition and all its profiles are deleted from the
source server
 You can now add dedicated adapters to the mobile partition
via DLPAR as needed, or put it in an LPAR workload group
31
© 2013 IBM Corporation
Power ATS
Effects
 Server properties
• The affinity characteristics of the logical memory blocks may change
• The maximum number of potential and installed physical processors
may change
• The L1 and/or L2 cache size and association may change
This is not a functional issue, but may affect performance
characteristics
 Console
• Any active (serial) console sessions will be lost when the partition is
migrated
• Console sessions must be re-established on the target system by the
user after migration
32
© 2013 IBM Corporation
Power ATS
Effects
 Network
• A temporary network outage of seconds is expected to occur as part
of suspending the partition
Temporary network outages may be visible to application clients,
but it is assumed that these are inherently recoverable
 VSCSI Server and Server Virtual Fibre Channel Adapters
• Adapters that are configured with the remote partition set to the
migrating partition will be removed
Adapters that are configured to allow “any” partition to connect will
be left configured after the migration
Any I/O operations that were in progress at time of the migration will
be retried once the partition is resumed
• As long as unused virtual slots exist on the target VIO server, the
necessary controllers and target devices will be automatically created
33
© 2013 IBM Corporation
Power ATS
Effects
 Error logs
• When a partition migrates, all of the error messages that the partition
received will appear on the target system
• Error logs contain the machine type, model and serial number so it is
possible to correlate the error with the system that detected it
 Partition time
• When a partition is migrated the Time of Day and timebase values of
the partition are migrated.
• The Time of Day of the partition is recalculated ensuring partition
timebase value increases monotonically and accounting for any
delays in migration.
34
© 2013 IBM Corporation
Power ATS
If slot numbers cannot be maintained…
…or server/HMC capabilities differ
35
© 2013 IBM Corporation
Power ATS
Make selections carefully for redundancy
The default
selected by PHYP
was ce51: vio1,
which was found
first searching by
lowest slot and then
partition name;
ec01 and ec02
were chosen
manually
36
© 2013 IBM Corporation
Power ATS
Watch which Mover Service Partition was selected
Notice that the
MSPs may not be
ordered the way
you want or
expect; they
apparently are
sorted by partition
name
37
© 2013 IBM Corporation
Power ATS
You can make selection manually
38
© 2013 IBM Corporation
Power ATS
Review Validation Errors/Warnings
This appears when different
HMC capabilities or server
capabilities exist on the two
servers
39
© 2013 IBM Corporation
Power ATS
Questions?
© 2013 IBM Corporation
Power ATS
Helpful References
Redbooks
IBM PowerVM Live Partition Mobility, SG24-7460
PowerVM Migration Physical to Virtual Storage, SG24-7825
IBM PowerVM Virtualization Introduction and Configuration, SG24-7940-05
IBM PowerVM Virtualization Managing and Monitoring, SG24-7590-04
IBM Redbooks home page:
http://www.redbooks.ibm.com
InfoCenter Help
http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/p7hc3
/iphc3hmcprepservers.htm
© 2013 IBM Corporation