PowerHA HyperSwap 日常维护 tips
目录
1
前期准备................................................................................................................................... 3
1.1
检查是否可以启动集群的前期准备 ........................................................................... 3
1.2
与 HyperSwap 相关的系统和磁盘设置 ...................................................................... 3
1.3
配置 Oracle RAC over HyperSwap 的特别注意事项.................................................... 4
1.3.1
尽量安排先安装 Oracle Grid............................................................................ 4
1.3.2
尽量将用于存放用户数据的盘的 PVID 清除掉 ............................................. 4
1.3.3
DG（Disk Group）规划 .................................................................................... 4
1.3.4
如何迁移 OCR 信息 .......................................................................................... 4
1.3.5
如何迁移 ASM spfile 信息 ................................................................................ 5
1.3.6
如何创建用于 Oracle 的 Quorum Vote Disk 的 NFS Server 和 Client ............. 5
1.3.7
如何创建包含有基于 NFS 的 Quorum Vote Disk 的 Disk Group .................... 6
1.3.8
如何替换 Disk Group ........................................................................................ 6
1.4
如何充分利用 FC 多路径的性能 ................................................................................. 6
1.5
补丁............................................................................................................................... 7
1.5.1
AIX ..................................................................................................................... 7
1.5.2
PowerHA............................................................................................................ 7
1.5.3
1.5.4
2
Storage（模拟站点间 FC 链路故障对 LUN 的 IO 影响） .............................. 7
Tips to A-A environment ................................................................................... 9
1.5.5
A-S 环境配置注意 .......................................................................................... 10
1.5.6
CAA 参数示例 ................................................................................................. 10
创建/配置期间 ....................................................................................................................... 11
2.1
利用 PowerHA 配置 CAA 之前建议手工创建 CAA 来检测环境是否正常............... 11
2.2
停 CAA 服务 ................................................................................................................ 11
2.3
Clcomd 服务不能启动................................................................................................ 12
2.4
创建 CAA Cluster 时报错的一般处理方法 1 ............................................................. 12
2.5
创建 CAA Cluster 时报错的一般处理方法 2 ............................................................. 13
2.6
创建 CAA Cluster 时报错的一般处理方法 3 ............................................................. 17
2.7
影子盘存在会导致创建 CAA Cluster 失败 ................................................................ 17
2.8
Clcomd 服务起来后马上又掉下去了 ........................................................................ 19
2.9
创建资源组时的站点间管理策略的选择： ............................................................. 19
2.10
配置 DS8K 的 LUN 注意 LSS 划分 .......................................................................... 20
2.11
创建 MG（Mirror Group）报 22 错误 .................................................................. 20
2.12
配置 Repository MG 时，报错 ERROR: Failed to run 'chrepos'. ............................ 20
2.13
配置 MG 时 Unplanned HyperSwap Timeout 不能超过 120 秒 ........................... 21
2.14
出现 Halt 状态的 PPRC 如何解决 .......................................................................... 21
2.15
增加 clxd phake 的 trace 日志................................................................................ 21
2.16
如何配制 dsh .......................................................................................................... 21
2.17
如果想让某些网卡不作为心跳网络 ..................................................................... 22
2.18
配置 LVM Mirror + HyperSwap 注意事项 .............................................................. 22
1 / 67
3
4
2.19
删除 Cluster 信息失败 ........................................................................................... 23
2.20
同步返回错误码列表 ............................................................................................. 24
运行期间................................................................................................................................. 25
3.1
重要日志..................................................................................................................... 25
3.2
PowerHA tie-breaker 的监控和管理 .......................................................................... 26
3.3
如何判断包含 Mirror Group 配置的一个 PowerHA 服务启动成功 ........................ 26
3.4
备份存储故障导致 PPRC 的 meta 信息丢失后，如何修复？ ................................ 27
3.5
脑裂后 CAA 状态的恢复 ............................................................................................ 28
3.6
删除 PowerHA 配置是遇到删除 cluster 信息失败 ................................................... 28
3.7
如何诊断 mirror group 是否 active，以及不写 syslog.phake 文件的问题 ............. 29
3.8
Voting disk 的恢复 ...................................................................................................... 30
3.9
在 7.1.3 中在线从 Multi-cast 转为 Uni-cast .............................................................. 35
其他......................................................................................................................................... 38
4.1
与 PowerHA HyperSwap 有关的一些 ODM 组件： .................................................. 38
4.2
Active-Active 环境下脑裂后存储的恢复................................................................... 48
4.3
Actice-Standby 的脑裂场景 ....................................................................................... 52
4.4
常用的脚本................................................................................................................. 62
4.5
当出现存储切换未达到期望结果时，需要进一步采集 trace 与 dev 沟通的方法66
2 / 67
1
前期准备
1.1
检查是否可以启动集群的前期准备
1． lssrc -a|egrep "StorageRM|cthags|ctrmc|clcomd|clconfd|clxd" 检查服务是否都已经启动
2． lscluster -m 查看 CAA 集群是否正常
3． 确认集群配置之前都同步过
4． Reboot，one by one (从 7.1.2 开始，就可以并行处理了)
5． Start powerha service, one by one(从 7.1.2 开始，就可以并行处理了)
如果上述有服务没有启动，需要尝试用手工启动（startsrc -s ..）
，如果手工启动失败，需要
先解决这些问题。
1.2
与 HyperSwap 相关的系统和磁盘设置
To configure a PowerHA® SystemMirror® cluster to use in-band communication for traditional
Metro Mirror PPRC, complete the following steps on each node in the cluster:
1. Enter the following command to configure all disks that are part of the storage system to use
the AIX Path Control Module (PCM) driver:
manage_disk_drivers –d device -o AIX_AAPCM
2. Restart the node by entering the shutdown -r command.
3. SCSI reservations are not supported for disks that are used in a HyperSwap® mirror group.
Enter the following command to verify that no disk reservations are set:
devrsrv -c query -l hdisk_name
The command returns the following information:
ODM Reservation Policy
: NO RESERVE
Device Reservation State
: NO RESERVE
4. Enter the following command to change the disk reservation policy to no_reserve:
chdev –a reserve_policy=no_reserve –l hdisk_number
5. Enter the following command to change the attributes of the primary disk for all nodes in
the primary site:
chdev –a san_rep_cfg=migrate_disk –l hdisk_name -U
Where hdisk_name is the name of the primary disk.
6. Enter the following command to change the attributes of the auxiliary disk for all nodes in
the auxiliary site:
chdev –a san_rep_cfg=migrate_disk –l hdisk_name -U
Where hdisk_name is the name of the auxiliary disk.
7. Enter the following command to verify that the disks in the storage system are configured
correctly:
lsattr –EO –a san_rep_cfg,san_rep_device,reserve_policy –l hdisk_number
The command returns the following information:
#san_rep_cfg:san_rep_device:reserve_policy
migrate_disk:detected:no_reserve
3 / 67
Note: The value for san_rep_cfg might display none, which is acceptable. However, the
san_rep_device value must be detected or yes.
1.3
配置 Oracle RAC over HyperSwap 的特别注意事项
1.3.1
尽量安排先安装 Oracle Grid
如果先安装了 PowerHA 软件，在安装 Oracle Grid 时，需要安装 Oracle 补丁，参考 metalink
的文档：1384060.1
1.3.2
尽量将用于存放用户数据的盘的 PVID 清除掉
如果这些盘的 PVID 存在，对于 GRID 的安装、数据库的创建没有影响，但是 PowerHA 在配
置基于 Raw Disk 的 Mirror Pool 时，
是列举不出带有 PVID 的盘的；只能通过命令行进行增加，
例如下面例子中就是创建一个 user mg(racmg)，包含 4 块裸盘(hdisk1-4)，以及其他的一些参
数。
/usr/es/sbin/cluster/xd_generic/xd_cli/clxd_create_mg_smit -z 'ds8k_inband_mm' -y 'user' -n
'racmg'
-a
'hdisk1:ec2607c3-1421-f180-744f-f2a7a19891a5
hdisk2:76d585c6-7e43-9491-a266-4155b28677b6
hdisk3:7852cad3-b2b2-c342-2f34-99d5fced5120
hdisk4:c12e2798-6bfd-4898-4131-c4994a16917f' -x 'enabled' -g 'yes' -t '60' -p 'Medium' -r
'automatic' -R 'manual'
1.3.3
DG（Disk Group）规划
三个 DG：一个是 tempDG，一个是 voteDG，一个是 dataDG
1．在初始安装 Grid 期间，利用一块临时共享盘，创建 tempDG，里面放置 OCR、Vote、ASM
spfile 等信息；
2．等 Grid 创建后，先创建用户存放用户数据的 dataDG
3．然后将 OCR、ASM spfile 迁移到该 DG 内
4．之后创建包含 3 个 vote 设备的 voteDG（每个存储各一块单独的 LUN，再加上来自 NFS
的设备）
5．将 voteDG 设置为主 vote disk group
1.3.4
如何迁移 OCR 信息
##replace ocr, from tempDG to dataDG
su - grid
ocrcheck
ocrconfig -add +dataDG
ocrconfig -delete +tempDG
4 / 67
(add new ocr on dataDG )
(remove old ocr)
ocrcheck
1.3.5
如何迁移 ASM spfile 信息
##move asm spfile, from tempDG to dataDG
su - grid
asmcmd
spget
(查当前 spfile)
spcopy 'old_spfile' 'new_spfile'
(复制 spfile，例如：spcopy +TEMPDG/rac-cluster/ASMPARAMETERFILE/REGISTRY.253.866325101
+DATADG/rac-cluster/ASMPARAMETERFILE/spfile)
spset 'new_spfile'
+DATADG/rac-cluster/ASMPARAMETERFILE/spfile)
shutdown --abort
startup
spget
1.3.6
( 设 置 新 spfile 位 置 ， spset
(每个节点分别执行)
(每个节点分别执行)
(确认当前 spfile)
如何创建用于 Oracle 的 Quorum Vote Disk 的 NFS Server 和 Client
在 NFS Server 创建一个文件系统，并将它 export 出去，这样，在/etc/exports 中会生成类似
下面的一行：
/tftpboot -sec=sys:krb5p:krb5i:krb5:dh,rw
在 NFS Server 上需要创建一些用户和组，UID 和 PID 需要和 RAC 环境一致，脚本示例如下：
mkgroup -A id=501 oinstall
mkgroup -A id=502 dba
mkgroup -A id=504 asmadmin
mkgroup -A id=506 asmdba
mkgroup -A id=507 asmoper
mkuser id='500' pgrp='oinstall' groups='asmadmin,asmdba,asmoper' home='/home/grid' grid
mkuser id='501' pgrp='oinstall' groups='dba,asmdba' home='/home/oracle' oracle
然后在 NFS Server 上对 NFS 进行修改权限
chown grid:asmadmin /tftpboot
chmod 660 /tftpboot
别忘了在 NFS Server 的/etc/hosts 中增加 RAC 的所有节点的主机描述，以用于对客户端的识
别
随后在 RAC 节点上创建 NFS 的客户端，或直接在/etc/filesystems 配置文件中增加一项，注意
mount 的参数，其中 noac 表示取消缓存。
5 / 67
/nfsvote3:
dev
= "/tftpboot"
vfs
= nfs
nodename
= nfsserver
mount
= false
options
rw,bg,hard,intr,rsize=32768,wsize=32768,timeo=600,vers=3,proto=tcp,noac,sec=sys
这样，在 RAC 节点就可以 mount 远程的文件系统了（mount /nfsvote3）
1.3.7
如何创建包含有基于 NFS 的 Quorum Vote Disk 的 Disk Group
##create vote file on nfs
su - grid
dd if=/dev/zero of=/nfsvote3/nfs_vote bs=100M count=1
chmod 660 /nfsvote3/nfs_vote
chown grid:asmadmin /nfsvote3/nfs_vote
##create votedg
su - grid
sqlplus / as sysasm
create diskgroup voteDG normal redundancy
failgroup fg1 disk '/dev/rhdisk6'
failgroup fg2 disk '/dev/rhdisk18'
quorum failgroup fg3 disk '/nfsvote3/nfs_vote'
attribute 'compatible.asm' = '11.2.0.0.0';
1.3.8
如何替换 Disk Group
##replace votedisk, from votetmpdg to votedg
su - grid
crsctl replace votedisk +voteDG
crsctl query css votedisk
1.4 如何充分利用 FC 多路径的性能
修改磁盘的 algorithm 从 fail_over 为 round_robin.
6 / 67
=
1.5 补丁
推荐采用 6.1TL9SP3(7.1.3SP3) + PowerHA 7.1.3 SP2.
如果采用较低版本（AIX6.1TL8SP2 and PowerHA 7.1.2SP2），则对补丁有要求：
1.5.1 AIX
filename
related files
ABSTRACT
1
aix_872056.140211.epkg.Z
/usr/sbin/rsct/lib/libhb_client.a
efix for pmr20816
2
dping1_61X.140522.epkg.Z
/usr/lib/drivers/dpcomdd
Fix for disk ping for stale node c
3
oracle_61X.130607.epkg.Z
/usr/lib/libcfgscsi.a
/usr/lib/methods/chgdisk
Fixes for hyperswap disk config
/usr/lib/drivers/cluster
4
CAA_140527.140527.epkg.Z
/usr/ccs/lib/libcluster.a
61X CAA fixes for PowerHA HyperSwa
/usr/sbin/clcomd
/usr/lib/drivers/storfwork
/usr/lib/drivers/aixdiskpcmke
5
hdrop7_61X.140513.epkg.Z
/usr/lib/drivers/scsidisk
Fixes for hypeswap on 61TL8
/usr/lib/drivers/scsidiskpin
/usr/lib/ras/autoload/aixdiskpcmke64.kdb
1.5.2 PowerHA
filename
directory
checks
1
def872286a.130725.epkg.Z
2
process_resources
/usr/es/sbin/cluster/events
62984
3
cluster
/usr/lib/drivers/
29934
446 cluster
4
clhaver
/usr/es/sbin/cluster/utilities
48067
256 clhaver
5
phake
/usr/lib/drivers/
20485
532 phake
6
libclxd_ds8k_mm.so
/usr/es/sbin/cluster/xd_generic/xd_ds8k_mm/
08649
554 libclxd_ds8k_mm.so
7
cldare
/usr/es/sbin/cluster/utilities/
08262
166 cldare
8
genxd_predisk_available
/usr/es/sbin/cluster/xd_generic/rresmethods/
27158
88 process_resources
12 genxd_predisk_availab
1.5.3 Storage（模拟站点间 FC 链路故障对 LUN 的 IO 影响）
例如下图，如果两个站点之间的光纤链路出现故障（可以是真实故障，或者是采用 disable 交
换机端口的方式等）
，对 LUN 的 IO 影响如下：
1.) 如果 LUN 已经配置成 Metro Mirror，并配置到 PowerHA Mirror Group 中，这时候由
7 / 67
PowerHA 发起 freezepprc 的命令；因为处于对数据安全的考虑，对这些 LUN 对应的
pprcpath 会设置 CG(Consistent Group)，这时候 external long busy 参数会生效（默认是 60s）
，
也就是 PowerHA 有 60 秒的时间来处理是否需要切换的动作
2.) 如果 LUN 已经配置 Metro Mirror，并未配置进 PowerHA Mirror Pool，并且 enable CG，这
样的话，发生 FC 链路故障，触发 DS8K freeze 的动作，并等待 external long busy 时间后
才能恢复
3.) 如果 LUN 已经配置 Metro Mirror，并未配置进 PowerHA Mirror Pool，也并未 enable CG，
如果采用这种方式模拟后，这些 LUN 的 IO 会暂停最多 30S
Site1
Site2
PowerHA Cluster
App1
RG1
App3
App2
Tie-breaker
from DS5020
SAN_switch1
SAN_switch3
SAN_switch4
SAN_switch2
10KM
Metro Mirror
DS8800(Primary)
DS8870
基于上述原理，建议配置如下：
1．
）PowerHA HyperSwap 方案的环境中，配置到 MG 里的盘都需要 enable CG
chlss -pprcconsistgrp enable C5
mkpprcpath -remotedev IBM.2107-75LY981 -remotewwnn 5005076308FFC6D4 -srclss
c5 -tgtlss c5 -consistgrp I0231:I0130
2．
）在 A-A 环境中，用于 NFS Vote 的 LUN 不能 enable CG，否则会有问题；如果 enable CG，
在 pprcpath 链路故障时，该 LUN 会自动被 freeze，并持续 60s，在这期间，Oracle 不能访问
第三个 vote 设备，再加上另外一个 vote disk 的不能访问，并且 Oracle CRS 的 misscount 时间
只设置了 45s，这样会造成 Oracle CRS 的全面瘫痪，引起不应有的故障。如果不 enable CG，
IO hang 的时间只有 30s，这样在 misscount 超时到达之前，IO 就可以恢复了，CRS 就不会被
杀掉。
8 / 67
Site1
Site2
PowerHA Cluster
NFS
RAC1
internal
LVM
Mirror
RAC4
NFS
Backup
RAC3
RAC2
B40-1
F48-1
F48-2
B40-2
Metro Mirror
Vote3
DS8800(Primary)
Vote1
Vote2
Vote3
Metro Mirror
DS8870
1.5.4 Tips to A-A environment
OS parameter
Remark
1. node_down_delay=20s
Don’t set them via clctrl command, need to via PowerHA smit
node_timeout=30s
menu:
Custom
Cluster
Configuration->Cluster
Nodes
and
Networks->Manage the Cluster->Cluster heartbeat settings->
*Node Failure Detection Timeout
*Failure Indication Grace Time
2. repos_mode=a
Set it via clctrl command
clctrl -tune -o repos_mode=a
Default is e(event), set it to a(assert)后, this setting avoids
Oracle’s uncertain character runder cluster partition scenarios.
3. Not set heatbeating on SAN
4. Not set PowerHA tie-breaker
Using Oracle 3 voting device mechanism
configure
5. Configure network interface for
accessing NFS into
Make sure the interface will not be used in CAA heartbeating.
private
network in PowerHA
6. Recovery voting configure after
some scenarios by manual
fault partition
7. Need to syncvg NFSVG after
primary
storage
fault
Including storage fault scenario, cluster partition scenario, site
and
Check if there is stale lv in nfsvg if has, need to do syncvg or
varyonvg
repaired
8. Set Oracle’s misscount =45s
9 / 67
crsctl set css miscount=45s
After primary storage fault, it needs >30s for LVM mirror to
recover IO, so need to increase misscount parameter.
9. Disable crs auto start after
crsctl disable crs
reboot OS
10. DS8K consistent group enable
showlss
chlss
mkpprcpath
1.5.5 A-S 环境配置注意
OS parameter
Remark
1. node_down_delay=20s
Don’t set them via clctrl command, need to via PowerHA smit
node_timeout=30s
menu:
Custom
Cluster
Configuration->Cluster
Nodes
and
Networks->Manage the Cluster->Cluster heartbeat settings->
*Node Failure Detection Timeout
*Failure Indication Grace Time
2. Configure PowerHA tie-breaker
配置 tie-breaker 后，对于脑裂场景非常有帮助
disk
3. Set
Action=’Automatic’
Recovery
for
user
如果设置成默认值’Manual’，当发生主站点故障，如果当时
的 RG 不是在备站点，PowerHA 不会做 swap 的动作，需要等
人确认后手工 swap，设计上就是这样的；
MG
设置成’Automatic’之后，在上述场景中，磁盘就可以自动
swap；
这个配置在 Active-Standby 场景中是建议设上的
备注 1：如果设置 tie-breaker，还是否需要设置? 这个参数
和仲裁的设置没有冲突
备注 2：如果设置成’Manual’，在模拟主站点宕时，如果通
过’halt –q’来模拟节点宕，可能磁盘也会 swap；但是’halt –q’
和 实 际 的 主 机 宕 是 有 区 别 的 ； 最 好 采 用 ’shutdown
immediately via HMC’来操作
备注 3：当发生脑裂脑裂场景，需要设置’tie-breaker’来保证
PowerHA 只会选择一个站点存活，再加上’Automatic’的选项，
保证磁盘在存活站点继续可以访问
1.5.6 CAA 参数示例
# clctrl -tune -L
NAME
DEF
MIN
MAX
UNIT
SCOPE
ENTITY_NAME(UUID)
CUR
-------------------------------------------------------------------------------communication_mode
10 / 67
m
c
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
u
-------------------------------------------------------------------------------config_timeout
240
0
2G-1
seconds
cn
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
240
-------------------------------------------------------------------------------deadman_mode
a
cn
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
a
-------------------------------------------------------------------------------link_timeout
30000
0
1171K milliseconds
cn
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
30000
-------------------------------------------------------------------------------node_down_delay
10000
5000
600000 milliseconds
cn
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
10000
-------------------------------------------------------------------------------node_timeout
20000
10000 600000 milliseconds
cn
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
20000
-------------------------------------------------------------------------------packet_ttl
32
1
64
cn
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
32
-------------------------------------------------------------------------------remote_hb_factor
10
1
100
c
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
10
-------------------------------------------------------------------------------repos_mode
e
cn
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
e
-------------------------------------------------------------------------------site_merge_policy
p
NewOS1_cluster(db789b62-7930-11e4-a41c-2a5c2f2ddc02)
c
h
--------------------------------------------------------------------------------
2
2.1
创建/配置期间
利用 PowerHA 配置 CAA 之前建议手工创建 CAA 来检测环境是否正常
如果不正常，先解决后再用 PowerHA 来配置。
如果手工创建发现没有创建 CAA cluster license，用工具 clprodunlock
2.2
停 CAA 服务
To upgrade the MPIO software or third-party multipath software, you must stop CAA cluster
services by entering the clmgr offline cluster STOP_CAA=yes command
11 / 67
clmgr stop node (your node_name) STOP_CAA=yes
该命令可以将某个节点的 PowerHA service和CAA service都停掉，停掉之后，就可以在剩余节
点对repository mg进行swap了
Note: 如需启动 CAA service，用 clmgr start node (your node_name) START_CAA=yes
如果需要对整个 Cluster 进行操作，可以采用：
clmgr stop cluster STOP_CAA=yes or
clmgr start cluster START_CAA=yes
2.3
Clcomd 服务不能启动
检查/etc/cluster/rhosts 的格式，只需要 IP 或主机名，不能和/etc/hosts 的格式一样，例如：
# cat /etc/cluster/rhosts
172.16.51.101
172.16.51.102
172.16.51.103
2.4
创建 CAA Cluster 时报错的一般处理方法 1
root@node3[/mnt/AIXTools/CAA_unlock]#mkcluster -r hdisk7 -m node2,node3 -n testcluster
ERROR: *** POWERHA SCRIPT /usr/es/sbin/cluster/sbin/smcaactrl RETURNED ERROR, FAILING ADD_NODE CHECK
***
ERROR: *** SEE POWERHA SCRIPT LOGS FOR ERRORS ***
ERROR:
********************************************************************************************
**********
WARNING: trans_id=2, client config FAIL: ADD_NODE CHECK node3 rc=1
ERROR: trans_id=2 Timeout occurred on initiator node3; fail the request.
ERROR: rc= -1 timeout_or_error_effect
ERROR: CHECK failed
mkcluster: ADD_NODE operation failed for node3.
WARNING: Force continue.
mkcluster: Could not create cluster.
The file access permissions do not allow the specified action.
需要检查
1． 需要检查 Node3 上的 ha 配置是否清除，
如果没有清除，
odm 还保留相关信息，
所以需要通过 smit hacmp
删除当前节点的 ha 的配置，然后再进行测试
检查命令：odmget HACMPsircol
这种情况一般出现在第一次同步 HA 配置时失败了，但其实 HA 已经在该节点的 HACMPsircol 里写入了
本节点的信息，所以如果再想同步的话，就需要先删除 HA 的配置，再进行重新配置及同步的操作。
12 / 67
如果还不行，则尝试以下命令
2． export CAA_FORCE_ENABLED=1
3． rmcluster –r hdisk*
4． dd if=/dev/zero of=/dev/rhdisk* bs=1m count=1024
然后再进行尝试
2.5
创建 CAA Cluster 时报错的一般处理方法 2
# mkcluster -r hdisk6 -m p7502901,p7502902,p7502903 -n testcluster
ERROR:
*******************************************************************************
***********************
ERROR: *** ADD_NODE CHECK POWERHA SCRIPT /usr/es/sbin/cluster/sbin/smcaactrl RETURNED
ERROR ***
ERROR: *** SEE POWERHA SCRIPT LOGS FOR ERRORS ***
ERROR:
*******************************************************************************
***********************
ERROR: ADD_NODE CHECK Script timed out or returned failure on p7502901
1035-260 mkcluster: ADD_NODE operation failed for p7502901.
1035-305 mkcluster: Could not create cluster.
The file access permissions do not allow the specified action.
经过分析，该出错的原因在于之前在删除 PowerHA Cluster 配置时未成功执行，并且还导致
p7502901 节点 crash：
Crash 的原因是触发了 Dead Man Switch:
# errpt -aj 225E3B63|more
--------------------------------------------------------------------------LABEL:
KERNEL_PANIC
IDENTIFIER:
225E3B63
Date/Time:
Fri Jul 10 09:50:57 CST 2015
Sequence Number: 4722294
Machine Id:
00F8BA814C00
Node Id:
p7502901
Class:
S
13 / 67
Type:
WPAR:
Resource Name:
TEMP
Global
PANIC
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
ASSERT STRING
PANIC STRING
Deadman timer triggered.
重起之后，发现 CAA Cluster 的配置并没有清理干净：
# lspv
hdisk0
hdisk1
hdisk4
hdisk5
hdisk6
hdisk2
hdisk3
00f8ba8174713d1a
00f8ba812f5f5a5b
00f8ba813257e573
00f8ba813257e3cf
00f8ba816c218165
00f8ba816c23bdab
00f8ba816c23be6d
# lsvg -l caavg_private
caavg_private:
LV NAME
TYPE
caalv_private1
boot
caalv_private2
boot
caalv_private3
boot
powerha_crlv
boot
rootvg
oravg
testVG
testVG
caavg_private
testVG
testVG
LPs
1
1
4
PPs
1
1
4
1
1
active
active
active
PVs LV STATE
MOUNT POINT
1
closed/syncd N/A
1
closed/syncd N/A
1
open/syncd
N/A
1
closed/syncd N/A
这时候可以采取以下方法：
1． 在该节点直接清理 Repository disk
# dd if=/dev/zero of=/dev/rhdisk6 bs=1m count=1024
1024+0 records in.
1024+0 records out.
2． 重新起动所有节点
3． 重起之后，给 hdisk6 生成 pvid，因为这时候 hdisk6 的 pvid 是没有的，如果直接这么创
建 CAA Cluster，会报错
14 / 67
# lspv
hdisk0
hdisk1
hdisk4
hdisk5
hdisk6
hdisk2
hdisk3
00f8ba8174713d1a
00f8ba812f5f5a5b
00f8ba813257e573
00f8ba813257e3cf
none
00f8ba816c23bdab
00f8ba816c23be6d
rootvg
oravg
testVG
testVG
None
testVG
testVG
active
active
# mkcluster -r hdisk6 -m p7502901,p7502902,p7502903 -n testcluster
ERROR: return=-1, create_cvg failed
1035-253 mkcluster: Could not create repository.
ERROR: Cannot varyonvg caavg_private. It does not exist.
WARNING: LVM operation importvg or reducevg for caavg_private failed.
WARNING: Force continue.
ERROR: Cluster repository device /dev/rhdisk6 does not have valid boot record.
1035-305 mkcluster: Could not create cluster.
A system call received a parameter that is not valid.
# chdev -l hdisk6 -a pv=yes
hdisk6 changed
# lspv
hdisk0
00f8ba8174713d1a
hdisk1
00f8ba812f5f5a5b
hdisk4
00f8ba813257e573
hdisk5
00f8ba813257e3cf
hdisk6
00f8ba8175aff7c2
hdisk2
00f8ba816c23bdab
hdisk3
00f8ba816c23be6d
rootvg
oravg
testVG
testVG
None
testVG
testVG
active
active
4． 在 p7502901 上产生 PVID 之后，需要其他节点更新 PVID，这就需要在其他节点单独删除
repository disk，并进行相关参数修改
# hostname
p7502902
# lspv
hdisk0
hdisk1
hdisk2
hdisk4
hdisk5
hdisk6
hdisk3
15 / 67
00f8ba8177e43708
00f8ba812f5f9bcb
00f8ba816c23bdab
00f8ba813257e573
00f8ba813257e3cf
00f8ba816c218165
00f8ba816c23be6d
rootvg
oravg
testVG
testVG
testVG
None
testVG
active
active
# rmdev -dl hdisk6 -R
hdisk6 deleted
# cfgmgr
# lspv
hdisk0
00f8ba8177e43708
hdisk1
00f8ba812f5f9bcb
hdisk2
00f8ba816c23bdab
hdisk4
00f8ba813257e573
hdisk5
00f8ba813257e3cf
hdisk6
00f8ba8175aff7c2
hdisk3
00f8ba816c23be6d
hdisk11
none
# chdev -l hdisk6 -a reserve_policy=no_reserve
hdisk6 changed
# chdev -l hdisk6 -a algorithm=round_robin
hdisk6 changed
# chdev -l hdisk6 -a san_rep_cfg=migrate_disk
hdisk6 changed
# lspprc -Ao
hdisk#
PPRC
Primary
Secondary
state
path group path group
ID
ID
hdisk4
Active 0(s)
1
hdisk5
Active 0(s)
1
hdisk6
Active 0(s)
1
rootvg
oravg
testVG
testVG
testVG
None
testVG
None
Primary Storage
WWNN
active
active
Secondary Storage
WWNN
5005076305ffd4a4 5005076305ffd4ac
5005076305ffd4a4 5005076305ffd4ac
5005076305ffd4a4 5005076305ffd4ac
5． 之后，再重新创建 CAA Cluster，发现报错：
# mkcluster -r hdisk6 -m p7502901,p7502902,p7502903 -n testcluster
ERROR:
*******************************************************************************
***********************
ERROR: *** ADD_NODE CHECK POWERHA SCRIPT /usr/es/sbin/cluster/sbin/smcaactrl RETURNED
ERROR ***
ERROR: *** SEE POWERHA SCRIPT LOGS FOR ERRORS ***
ERROR:
*******************************************************************************
***********************
ERROR: ADD_NODE CHECK Script timed out or returned failure on p7502901
1035-260 mkcluster: ADD_NODE operation failed for p7502901.
1035-305 mkcluster: Could not create cluster.
The file access permissions do not allow the specified action.
6． 判断是在 P7502901 上的 PowerHA 信息没有删除干净，smitty cm_manage_cluster
16 / 67
7． 删除之后，再创建 CAA Service，成功
# mkcluster -r hdisk6 -m p7502901,p7502902,p7502903 -n testcluster
mkcluster: Successfully created cluster.
# rmcluster -n testcluster
rmcluster: Successfully removed testcluster.
2.6
创建 CAA Cluster 时报错的一般处理方法 3
# hostname
p7502902
# mkcluster -r hdisk6 -m p7502901,p7502902,p7502903 -n testcluster
ERROR: Cluster configuration detected on p7502903.
ERROR: Remove any cluster configuration from p7502903 and retry command.
ERROR: ADD_NODE failed for p7502903
1035-264 mkcluster: Could not add all new entities.
1035-305 mkcluster: Could not create cluster.
The file access permissions do not allow the specified action.
# rmcluster -n testcluster
rmcluster: Successfully removed testcluster
如果在某个节点创建 CAA Cluster 时报告说其他节点的对应盘已经有 Cluster 信息了，那就直
接在那个节点进行创建，创建的时候会做部分清理的工作，就省去了手工清理的工作。
# mkcluster -r hdisk6 -m p7502901,p7502902,p7502903 -n testcluster
mkcluster: Successfully created cluster.
# hostname
p7502903
2.7
影子盘存在会导致创建 CAA Cluster 失败
有 时 候 在 刚 创 建 hacmp 的 配 置 ， 配 置 完 repository disk 后 同 步 会 失 败 ， 查 看
/var/adm/ras/syslog.caa 有如下报错：
ul 25 15:05:53 CARD1 caa:info unix: kcluster_event.c
17 / 67
ahafs_cluster_init
2699
nodeState event registered with
AHAFS_NODE_ID = 1
Jul 25 15:05:53 CARD1 caa:info unix: kcluster_event.c
ahafs_cluster_init
2722
nodeAddress event registered with
AHAFS_ADDR_ID = 2
Jul 25 15:05:53 CARD1 caa:info unix: kcluster_event.c
ahafs_cluster_init
2744
networkAdapterState event
registered with AHAFS_INTERFACE_ID = 3
…
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: clusterconf_lib.c
_find_and_load_repos
1279
got nothing from
ODM
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: clusterconf_lib.c
_find_and_load_repos
1296
return = -1 No
repository candidate found.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: cluster_utils.c
local_node_in_repos
10846
Failed to load the
repository.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_topology.c cl_incluster
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_query.c
1011
cl_query
Local node is not in the repository.
2376
Query failed: line 2338: The
system call does not exist on this system.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: clusterconf_lib.c
_find_and_load_repos
1279
got nothing from
ODM
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: clusterconf_lib.c
_find_and_load_repos
1296
return = -1 No
repository candidate found.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: cluster_utils.c
local_node_in_repos
10846
Failed to load the
repository.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_topology.c cl_incluster
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: clusterconf_lib.c
1011
Local node is not in the repository.
_find_and_load_repos
1279
got nothing from
ODM
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: clusterconf_lib.c
_find_and_load_repos
1296
return = -1 No
repository candidate found.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: cluster_utils.c
local_node_in_repos
10846
Failed to load the
repository.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_topology.c cl_incluster
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_query.c
1011
Local node is not in the repository.
cl_query
2376
Query failed: line 2338: The
cl_query
2376
Query failed: line 1973: The
system call does not exist on this system.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_query.c
system call does not exist on this system.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_main.c
_validate_and_fill
476
Could not get disk UUID
from disk name.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: clusterconf_lib.c
_find_and_load_repos
1279
got nothing from
ODM
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: clusterconf_lib.c
_find_and_load_repos
1296
return = -1 No
repository candidate found.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: cluster_utils.c
local_node_in_repos
10846
Failed to load the
repository.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_topology.c cl_incluster
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_query.c
system call does not exist on this system.
18 / 67
cl_query
1011
Local node is not in the repository.
2376
Query failed: line 2338: The
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: cluster_bootutils.c
is_clvdisk
394
return = 0, hdisk42: Cannot
open disk.
Jul 25 15:05:53 CARD1 caa:err|error cluster[5899094]: get_other_disk_uuid: Device hdisk42 is not configured in Available state.
Jul 25 15:05:53 CARD1 caa:err|error cluster[5899094]: register_other_disk: Could not create a disk UUID for device hdisk42.
Jul 25 15:05:53 CARD1 caa:err|error cluster[5899094]: cluster_utils.c
cl_validate_device
6362
Specified device,
hdisk42, is not a supported type.
Jul 25 15:05:53 CARD1 caa:info cluster[5899094]: caa_main.c
cl_create
1829
FINISH return -1
环境中 hdisk42 其实是一块影子盘，是 defined 状态，这时需要删除该盘，然后再重新创建。
2.8
Clcomd 服务起来后马上又掉下去了
检查网络配置，特别是/etc/cluster/rhosts 文件和/etc/hosts 的对应关系；修正后重新启动
clcomd 服务
2.9
创建资源组时的站点间管理策略的选择：
这里还需要修改资源组的 ‘Inter-Site Management Policy’，从’ignore’修改为’Online On Either
Site’，说明如下：
1. 设置inter-site management policy为'Prefer primary site'：
在启动HA的时候，RG会在primary site激活；
当主站点节点故障后，RG会切换到备站点
当主站点恢复后，RG会自动回切到主站点（此时，Fallback Policy是不生效的）
2. 设置inter-site management policy为'Online on either site'：
在启动HA的时候，如果先启动主站点服务，那RG在该站点激活；如果先启动备站点服务，
那RG就会在备站点激活
如果RG在主站点激活，当主站点节点故障后，RG切换到备站点，然后主站点节点恢复，
RG不会回切
所以，如果不想让 RG 在主站点回复的时候自动回切，需要设置'Online on either site'；注意
在启动服务的时候，先启动主站点的节点，然后再启动备站点的节点
19 / 67
2.10 配置 DS8K 的 LUN 注意 LSS 划分
根据 DS8K 的技术特点，当某些原因（光纤链路、磁盘本身或其他）导致某块盘 Suspend 时，
与该 LUN 相同 LSS ID 的所有 LUN 都会 Suspend。
基于这个特性，在配置 MG（Mirror Group in PowerHA）时，相同 LSS ID 的 LUN 不能规划在
不同的 MG 里面；但一个 MG 可以包含多个 LSS ID 的盘（测试过６４个）
；否则在同步的时
候会报错。
但是，
如果含有相同 LSS ID 的盘分别配置给不同 PowerHA Cluster 内的 MG，那就判断不出来，
但是在运行中会相互影响。
2.11 创建 MG（Mirror Group）报 22 错误
错误类似如下：
Failed to changed XD Mirror Group 'User_RD_MG2'. errno=22 msg=No Storage System name
added
OR PPRC path not established properly
OR Hyperswap is not enabled for disks. Check clxd_debug.log for details
This message means that you did not correctly configure the disks that you are using for the
mirror group that uses the HyperSwap function.检查该盘的 HyperSwap 是否正常配置。
另外，需要检查 User MG 的 Recovery Action 是否设置了 Automatic，如果还是保留 Manual
选项，在当前版本会出现 errno=22 的问题。（PHA 7.1.3SP3）
2.12 配置 Repository MG 时，报错 ERROR: Failed to run 'chrepos'.
ERROR: Failed to run 'chrepos'.
Follow recommended user actions to recover from here
(if needed in case caavg_private has not been seen in some or all nodes).
20 / 67
这时候可以采用手工的方式替换，来检查问题在哪里，例如：
Examples
1
2
To replace the hdiskY disk with the hdiskX disk in a cluster named cl1:
chrepos -n cl1 -r +hdiskX,-hdiskY
To replace the existing repository disk with the hdiskX disk in the cluster called cl1:
chrepos -n cl1 -r +hdiskX
2.13 配置 MG 时 Unplanned HyperSwap Timeout 不能超过 120 秒
如果>=120s，同步没问题，但是运行的时候会出问题。
2.14 出现 Halt 状态的 PPRC 如何解决
例如：
# lspprc -Ao
# hdisk#
PPRC
state
hdisk104
hdisk14
hdisk15
hdisk16
hdisk17
hdisk18
Active
Halt
Halt
Active
Active
Active
Primary
Secondary Primary Storage Secondary Storage
path group path group WWNN
WWNN
ID
ID
-1
0
500507630affc16b 5005076308ffc6d4
0(s)
1
500507630affc16b 5005076308ffc6d4
0(s)
1
500507630affc16b 5005076308ffc6d4
0(s)
-1
500507630affc16b 5005076308ffc6d4
0(s)
-1
500507630affc16b 5005076308ffc6d4
0(s)
1
500507630affc16b 5005076308ffc6d4
To move a disk from the halt state to an active state, you must reboot the node or bring the
resource group for the associated disk online.
2.15 增加 clxd phake 的 trace 日志
a.) 在/etc/syslog.conf 最后增加一行
kern.debug /tmp/syslog.phake rotate size 10M files 10
b.)
>/tmp/syslog.phake
c.)
stopsrc -s syslogd; sleep 2; startsrc -s syslogd
2.16 如何配制 dsh
1. 安装 ssh,ssl
21 / 67
2. /usr/bin/ssh-keygen 生成公钥，私钥
3. 将所有节点的公钥合到一个文本文件中，文件名为 authorized_keys
4. 将 authorized_keys 文件复制到所有节点的/.ssh 目录中
5. 这样各节点之间的访问就不需要输入密码了，不过需要用 ssh 先访问一下，同意自动连
接
6. 在/etc/environment 文件中加入两个环境变量:
DSH_NODE_RSH=/usr/bin/ssh
WCOLL=/home/dshlist
7. 需要在 WCOLL 环境变量所指的文件中加入节点名，例如:
P77003N3
P77003N4
8. 重新登录后就可以使用 dsh 命令了
2.17 如果想让某些网卡不作为心跳网络
CAA cluster 创建后，自动将所有网卡都作为心跳传输的设备；如果想让某些网卡不作为心跳
设备，建议如下操作：
手工在 PowerHA 将这些网卡配置到 private network 中，重新同步后，
会在/etc/cluster/ifrestrict
文件中增加这些设备名。
可以通过 lscluster -i 或者 lscluster -m 来检查。
2.18 配置 LVM Mirror + HyperSwap 注意事项
1.
2.
3.
4.
5.
Extend the existing vg to the Hyperswap disks, disable quorum for VG
Create mirror pool Mp1 and add non hyperswap (storage 1 disks) to MP1
Create mirror pool Mp2 and add Hyperswap disks to Mp2
Keep Mirror pool First copy of LV on Mp1 and second copy on Mp2
Create LVM RG and include the vg in this RG, set "Use forced varyon of volume groups, if
necessary" True.
6. Create RAW Disk mg with the hyperswap disks using following command (MG cannot be
created using smit since the disks will not be visible in the Create MG pannel)
/usr/es/sbin/cluster/xd_generic/xd_cli/clxd_create_mg_smit -z 'ds8k_inband_mm' -y 'user' -n
'<NAME>' -a '<RD1> <RD2> <RD3>' -x 'enabled' -g 'yes' -t '60' -p 'Medium' -r
'<manual/automatic>' -R '<manual/automatic>'
例如：
/usr/es/sbin/cluster/xd_generic/xd_cli/clxd_create_mg_smit -z 'ds8k_inband_mm' -y 'user' -n
'test1mg'
-a
'hdisk3:e962bae5-b568-b917-5872-4d953dfe885a
hdisk4:4bf2729e-2247-759b-443c-a9c546b14b2b' -x 'enabled' -g 'yes' -t '60' -p 'Medium' -r
'automatic' -R 'manual'
7. Set MG's 'Recovery Action' to Automatic if want to swap MG automatically in case of
primary site down
8. Create a RG For Hyperswap disks and include MG in the RG (along with RAW disks).
22 / 67
9.
Configure "Parent Child Dependency" between LVM RG and Hyperswap RG(Hyperswap RG
as parent and LVM RG as Child).
10. Configure "Online on the Same Node Dependency" between LVM RG and Hyperswap RG.
11. Do V&S
另外，如果重新开始配置，参考步骤如下：
1． 物理分区、线路、盘等环境就绪
2． 手工创建 CAA Cluster，以验证环境是否满足要求
3． 配置 PowerHA Cluster，包括节点、repository disk
4． 在 C-SPOC 菜单中添加 VG(testvg)，并配置两个 mirror pool，方便创建镜像的 LV，disable
testvg 的 quorum 属性，修改 userVGRG 属性，包括设置”"Use forced varyon of volume
groups, if necessary"为 true
5． 创建 Resource，包括存储、Repository MG
6． 用脚本创建 User MG，基于 RAW disks
7． 创建新的 RG(userMGRG)，包括 User MG 和 RAW disks
8． 定义 RG 之间的父子关系，userMGRG 是父、userVGRG 是子
9． 定义 RG 之间的依赖关系，都在同一个节点起停“Online on the Same Node Dependency”
10．
同步并起 PowerHA service
11．
起来后在 C-SPOC 菜单中创建文件系统
2.19 删除 Cluster 信息失败
在 HyperSwap 环境中，如果想删除 PowerHA Cluster 信息，会遇到删除操作 hang 在删除本
节点信息的步骤上，例如在 p7502903 节点上删除整个 cluster 信息，最后会 hang 在删除该
节点信息的过程中：
Ensuring that the following nodes are offline: p7502901, p7502902, p7502903
Warning: cluster services are already offline on node "p7502901" (state is "ST_INIT"). Removing
that node from the shutdown list.
Warning: cluster services are already offline on node "p7502902" (state is "ST_INIT"). Removing
that node from the shutdown list.
Warning: cluster services are already offline on node "p7502903" (state is "ST_INIT"). Removing
that node from the shutdown list.
Attempting to delete node "p7502901" from the cluster...
Attempting to delete node "p7502902" from the cluster...
Attempting to remove the CAA cluster from "p7502903"...
解决方案：
1． 在 hang 的时候，执行’stopsrc -s clxd’，这样删除工作可以继续
2． 如果手工中断删除作业，也可以执行’stopsrc -s clxd’，然后重新删除一次
注意事项：
如果采用 rmcluster –r 的方式强制清理盘上的 repository 信息，ODM 库里的信息不会删除，
那下次在创建 CAA Cluster 的时候就会出错，通过以下命令检查 ODM 库里信息是否清空：
23 / 67
# odmget HACMPsircol
HACMPsircol:
name = "p7502901_cluster_sircol"
id = 0
uuid = "0"
ip_address = ""
repository = "00f8ba817adb1c11"
backup_repository =
2.20 同步返回错误码列表
PPRC_RS_NO_ERROR = 0,
#/usr/include/errno.h
#define EPERM
1
/* Operation not permitted
#define ENOENT 2
/* No such file or directory
#define
ESRCH
3
/*
*/
PPRC_RS_TYPE_MISMATCH = 2,
*/
No
such
process
#define EINTR
4
/* interrupted system call
#define EIO
5
/* I/O error
#define ENXIO
6
/* No such device or address
#define E2BIG
7
/* Arg list too long
*/
9
/* Bad file descriptor
#define ECHILD 10
/* No child processes
#define EAGAIN 11
/* Resource temporarily unavailable
#define EACCES 13
EFAULT
PPRC_RS_NOT_FOUND = 8,
*/
#define EBADF
*/
*/
*/
PPRC_RS_NO_DEV_SUPPORT = 13,
*/
PPRC_RS_COMMAND_FAILURE = 14,
/* Permission denied
Bad
address
PPRC_RS_NOT_USER_CALL = 15,
PPRC_RS_COPYIN_FAILURE = 16,
*/
#define ENOTBLK 15
EBUSY
/* Block device required
16
/*
Resource
*/
PPRC_RS_COPYOUT_FAILURE = 17,
busy
PPRC_RS_ALLOC_FAILURE = 18,
PPRC_RS_COUNT_INVALID = 19,
*/
#define EEXIST 17
#define EXDEV
#define
PPRC_RS_INCOMPLETE = 11,
PPRC_RS_NO_TARGET_ACCESS = 12,
*/
/* Not enough space
/*
PPRC_RS_INVALID_KEYS = 9,
PPRC_RS_INVALID_ACTION = 10,
*/
14
PPRC_RS_INVALID_LIST = 6,
PPRC_RS_OTHER_INVALID = 7,
*/
/* Exec format error
#define ENOMEM 12
PPRC_RS_DUP_NAME = 5,
*/
#define ENOEXEC 8
#define
PPRC_RS_ROLE_MISMATCH = 3,
PPRC_RS_INVALID_NAME = 4,
*/
#define
PPRC_RS_SYNC_MISMATCH = 1,
18
ENODEV
/* File exists
/* Improper link
19
PPRC_RS_INVALID_STORTYPE = 20,
*/
*/
/*
No
such
device
PPRC_RS_IS_GROUP_MEMBER = 21,
PPRC_RS_GROUP_LSS_OVERLAP = 22
有可能是 pprcpath 的报错，需要在存储上查
*/
#define ENOTDIR 20
/* Not a directory
*/
#define EISDIR 21
/* Is a directory
#define EINVAL 22
/* Invalid argument
#define ENFILE 23
/* Too many open files in system
#define EMFILE 24
/* Too many open files
#define ENOTTY 25
/* Inappropriate I/O control operation */
看，所有 pprcpath 都不能有 failed。,
PPRC_RS_DISK_CLOSED = 23,
*/
*/
PPRC_RS_DISK_RESERVED = 24,
PPRC_RS_STRUCT_VERSION = 25,
*/
*/
PPRC_RS_INTERRUPTED = 26,
PPRC_RS_TRANSPORT_ERROR = 27,
#define ETXTBSY 26
/* Text file busy
*/
PPRC_RS_NO_SOURCE_ACCESS = 28,
#define EFBIG
/* File too large
*/
PPRC_RS_INVALID_HANDLE = 29,
24 / 67
27
#define ENOSPC 28
/* No space left on device
#define ESPIPE 29
#define EROFS
30
/* Read only file system
PPRC_RS_INTR_CALL = 31,
33
/* Domain error within math function
#define ECHRNG 37
PPRC_RS_NO_GROUP_INFO = 33,
*/
PPRC_RS_INVALID_REPTYPE = 34,
PPRC_RS_NOCHG_REPTYPE = 35,
PPRC_RS_NOCHG_STORTYPE = 36,
*/
*/
PPRC_RS_NO_PCM = 37,
/* Identifier removed
*/
PPRC_RS_GROUP_INACTIVE = 38,
/* Channel number out of range
*/
PPRC_RS_INVALID_STATE = 39,
/* No message of desired type
36
*/
*/
/* Result too large
#define ENOMSG 35
PPRC_RS_ODM_ERROR = 32,
*/
/* Broken pipe
#define ERANGE 34
#define EIDRM
*/
/* Too many links
32
#define EDOM
PPRC_RS_OP_IN_PROG = 30,
/* Invalid seek
#define EMLINK 31
#define EPIPE
*/
PPRC_RS_INVALID_RESYNC_POL = 40,
#define EL2NSYNC 38
/* Level 2 not synchronized
*/
#define EL3HLT 39
/* Level 3 halted
*/
PPRC_RS_INVALID_SWAP_POL = 41,
#define EL3RST 40
/* Level 3 reset
*/
PPRC_RS_NULL_ARG = 42,
*/
PPRC_RS_ALREADY_IN_STATE = 43
#define ELNRNG 41
/* Link number out of range
#define EUNATCH 42
/* Protocol driver not attached
*/
#define ENOCSI 43
/* No CSI structure available
*/
#define EL2HLT 44
/* Level 2 halted
#define EDEADLK 45
/* Resource deadlock avoided
#define ENOTREADY
46
*/
*/
/* Device not ready
*/
#define EWRPROTECT
47
/* Write-protected media
*/
#define EFORMAT
48
/* Unformatted media
*/
#define ENOLCK
49
#define ENOCONNECT
/* No locks available
50
*/
/* no connection
*/
#define ESTALE
52
/* no filesystem
#define EDIST
53
/* old, currently unused AIX errno*
3
3.1
*/
运行期间
重要日志
Name
hacmp.out
Location
/var/hacmp/log/hacmp.out
Description
Displays messages that are related
to detection, migration,
termination, and execution of a
PowerHA® SystemMirror® cluster for
25 / 67
Name
Location
Description
an application.
clutils.log
/var/hacmp/log/clutils.log
Displays the results of the
automatic verification that runs on
a specified PowerHA SystemMirror
cluster node every 24 hours.
clxd_debug.log /var/hacmp/xd/log/clxd_debug.log Displays information about the clxd
daemon.
另外再加上 clxd 服务的日志（同 2.7）
：
You can also configure kernel extension to create debug logs in the /etc/syslog.conf file by
completing the following steps:
1. In the /etc/syslog.conf file, add the following line:
kern.debug /var/hacmp/xd/log/syslog.phake rotate size 500k files 7
2. Create a file that is called syslog.phake in the /var/hacmp/xd/log directory.
3. Refresh the syslogd daemon.
Note: The debug logs are also logged in the console.
3.2
PowerHA tie-breaker 的监控和管理
当配置了 tie-breaker 盘之后，当 PowerHA 启动之后，需要检查 tie-breaker 盘是否被某个节
点锁住；如果出现这种情况，需要手工将锁信息清除，以免在真正发生脑裂故障时被误判断。
例如，正常运行期间，tie-breaker 的锁被 SiteB 的节点保留着，实际 tie-breaker 的盘部署在
SiteA（例如阜新环境）
，这时候如果发生脑裂现象，由于 SiteB 的节点保留了 tie-breaker 锁，
因此会尝试活下来，但其实并没有真正拿到锁（因为这时候已经访问不到 tie-breaker 盘了）
，
所以 SiteB 的节点会 crash；SiteA 的节点由于还是未锁住 tie-breaker 盘的状态，所以也 crash
了。
这个问题还在跟进。
命令：
devrsrv -c query -l hdisk*
devrsrv -c release -l hdisk*
devrsrv -f -l hdisk*
3.3
如何判断包含 Mirror Group 配置的一个 PowerHA 服务启动成功
1. Smit 菜单显示 ok 后，不代表服务已经启动
2. lssrc -ls clstrmgrES|grep stat 显示 stable 代表启动完全
3. 启动集群需要一个接着一个启动
另外， 确定 mirror group 是否启动正确，采用以下方法：
# tail -f /var/hacmp/xd/log/clxd.log，看到类似下面信息并不再增加内容，才说明启动正确
Refresh Mirror Group Config 'oramg' completed.
26 / 67
3.4
备份存储故障导致 PPRC 的 meta 信息丢失后，如何修复？
当备份存储出现异常宕机，可能会丢失 PPRC 的 meta 信息，就是 lspprc 的信息看不到了，
pprcpath 的信息也没有了，这时候，如何不间断应用来恢复环境.
注释：由于存储的 meta 信息出现故障，不能简单采用 resumepprc 来恢复数据，需要重新
mkpprc. Mkpprc 会导致 uuid 的变化，PowerHA 的 CAA repository disk 对 UUID 敏感，因此在
恢复过程中对该盘先采取 migrate 到另外一块盘，最后在 migrate 回来。
1． 存储
增加一个 LUN，并配置成 hyperswap 盘，分配给相关主机
1. AIX
a.) 不中断应用，将 PowerHA service 停下
b.) 将盘的属性 san_rep_cfg 修改为 revert_disk, 注意增加-U 参数
c.) cfgmgr 认出新加的盘，修改 reserve_policy=no_reserve 和 san_rep_cfg=migrate_disk,
并用 lspprc –Ao 确认
d.) smitty cm_add_repository_disk 把这块盘增加为 resposity 盘
e.) smitty cl_replace_repository_nm 用这块盘替换原先的盘
f.) 这样，就把 caa 的 repository 盘切换到新的盘上了
2. 存储
a.) 先在备份存储创建反向的 pprcpath, 例如
mkpprcpath -remotedev IBM.2107-75XP411 -remotewwnn 500507630BFFC4C8 -srclss
24 -tgtlss 34 I0003:I0030
注：主存储为 IBM.2107-75XP411
b.) 在主存储上创建正向 pprcpath, 例如
mkpprcpath -remotedev IBM.2107-75VF451 -remotewwnn 500507630AFFC454 -srclss 34
-tgtlss 24 I0030:I0003
注：备份存储为 IBM.2107-75VF451
c.) 在主存储删除 pprc
rmpprc -remotedev IBM.2107-75VF451 -at src -unconditional 3401
d.) 在主存储创建 pprc
mkpprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 3401:2401
3. AIX
a.) cfgmgr 认出一组新盘
b.) 修改原先盘的属性 san_rep_cfg 修改为 migrate_disk, 注意增加-U 参数
c.) cfgmgr
d.) 用 lspprc –Ao 确认
e.) 在一个节点修改 PowerHA 里 mirror group 的配置，包括 repository, system 和 user
mirror group, 以及 resource group 的属性
f.) 删除其他节点的 PowerHA 配置
g.) 从这个点进行重新同步
h.) 同步成功后启动服务
27 / 67
3.5
脑裂后 CAA 状态的恢复
当 Stretched cluster 脑裂场景发生后，网络和 FC 链路再次恢复，CAA 是不会自动恢复，
也就是说 mping 是失败的，两个站点之间是没有心跳信息传递的，这时候，如果想让 CAA
集群恢复正常，需要重起部分节点
a.) 如果两个站点的节点数一致，则需要重起节点号小的所在站点的所有节点
b.) 如果两个站点的节点数不一致，则需要重新启动节点数量少的站点所在站点的所有节点
3.6
删除 PowerHA 配置是遇到删除 cluster 信息失败
删除 HA 信息时，出现类似下面的错误：
这时候，重启机器后，
28 / 67
先 varyoffvg & exportvg
# varyoffvg caavg_private
# exportvg caavg_private
然后进行强制删除磁盘上的 cluster 信息，以下步骤在每个节点都需要操作
# export CAA_FORCE_ENABLED=1
# rmcluster -r hdisk6
rmcluster: This operation will scrub hdisk6, removing any volume groups and clearing cluster
identifiers.
If another cluster is using this disk, that cluster will be destroyed.
Are you sure? (y/[n]) y
WARNING: Force continue.
rmcluster: Successfully removed hdisk6.
最后在一个节点做清理磁盘的动作
# dd if=/dev/zero of=/dev/rhdisk6 bs=1m count=1000
100+0 records in.
100+0 records out.
3.7
如何诊断 mirror group 是否 active，以及不写 syslog.phake 文件的问题
1． 服务起来后，用 fv 工具测试下
2． 如果结果显示 mirror group inactive
3． genkex | grep phake
4．
#sum /usr/lib/drivers/phake
28675 407
/usr/lib/drivers/phake
5． 检查 phake 的文件是否正确（通过 filezilla 从 ausgsa.ibm.com 下载的就不行）
root@G2_VIOC2[/tmp/testbinaries]#getrepgrplst -c 10
29 / 67
sfwGetRepGroups API calL Done
----------Start PPRC RETUERN INFO Struct Detail-----------PPRC Group Version
:1
Return Flag
:0x0
Return Status
:0
PPRC Name
:
pprc_scsi_status.status
:0x0
pprc_scsi_status.key
:0x0
pprc_scsi_status.asc
:0x0
pprc_scsi_status.ascq
:0x0
-------End PPRC RETUERN INFO Struct Detail------------ Start PPRC REP DISK GRP LIST Details-----Version
:1
count
:2
Group Name:
pha_1365332521rdg0
UUID:0x420e7cb8c06f11fb572328fff276c92c Group State:
2:PPRC_GS_ACTIVE
Group Name:
pha_1465333531rdg0
UUID:0x694209594b6365ba3741608307666678 Group State:
2:PPRC_GS_ACTIVE
--- End PPRC REP DISK GRP LIST Details------------------Start Testcase Result -------RC
:0
Flags
:0
Status
:0
:SUCCESS
:PPRC_RS_NO_ERROR
--------------End Testcase Result ----------rc=0
flags=0
status=0
root@G2_VIOC2[/tmp/testbinaries]#genkex|grep phake
f1000000c04d5000
f1000000c045d000
3.8
5000 /usr/lib/drivers/phakernmgr
4b000 /usr/lib/drivers/phake
Voting disk 的恢复
在 RAC(A-A)测试模型中，如果主存储故障导致 voting disk 盘丢失，当主存储恢复后，如果再
把丢失的 voting 盘 online。
1. beforer primary storage
ASMCMD> lsdsk -G data
30 / 67
Path
/dev/rhdisk50
/dev/rhdisk53
/votedisk/nfs_vote
sqlplus "/as sysasm"
SQL> select disk_number,repair_timer,state,name,path from v$asm_disk order by name;
DISK_NUMBER REPAIR_TIMER STATE
NAME
PATH
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------0
0 NORMAL DATADG_0000
/dev/rhdisk0
1
0 NORMAL DATADG_0001
/dev/rhdisk1
2
0 NORMAL DATADG_0002
/dev/rhdisk2
3
0 NORMAL DATADG_0003
/dev/rhdisk3
4
0 NORMAL DATADG_0004
/dev/rhdisk15
6
0 NORMAL DATADG_0006
/dev/rhdisk11
7
0 NORMAL DATADG_0007
/dev/rhdisk12
8
0 NORMAL DATADG_0008
/dev/rhdisk9
10
0 NORMAL DATADG_0010
/dev/rhdisk10
11
0 NORMAL DATADG_0011
/dev/rhdisk13
12
0 NORMAL DATADG_0012
/dev/rhdisk14
DISK_NUMBER REPAIR_TIMER STATE
NAME
PATH
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------17
0 NORMAL DATADG_0017
/dev/rhdisk4
18
0 NORMAL DATADG_0018
/dev/rhdisk5
19
0 NORMAL DATADG_0019
/dev/rhdisk6
20
0 NORMAL DATADG_0020
/dev/rhdisk7
21
0 NORMAL DATADG_0021
/dev/rhdisk8
0
0 NORMAL DATA_0000
/dev/rhdisk50
1
0 NORMAL DATA_0001
/dev/rhdisk53
2
0 NORMAL DATA_0002
/votedisk/nfs_vote
8
0 NORMAL
/dev/rhdisk16
9
0 NORMAL
/dev/rhdisk17
10
0 NORMAL
/dev/rhdisk18
DISK_NUMBER REPAIR_TIMER STATE
NAME
PATH
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------31 / 67
11
0 NORMAL
13
0 NORMAL
14
0 NORMAL
16
0 NORMAL
17
0 NORMAL
15
0 NORMAL
2. After primary storage fails
ASMCMD> lsdsk -G data
Path
/dev/rhdisk19
/dev/rhdisk20
/dev/rhdisk22
/dev/rhdisk24
/dev/rhdisk25
/dev/rhdisk23
/dev/rhdisk50
/votedisk/nfs_vote
set linesize 1000
Col name format a25
select disk_number,repair_timer,state,name,path from v$asm_disk order by name;
SQL> select disk_number,repair_timer,state,name,path from v$asm_disk order by name;
DISK_NUMBER REPAIR_TIMER STATE
NAME
PATH
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------0
0 NORMAL DATADG_0000
/dev/rhdisk0
1
0 NORMAL DATADG_0001
/dev/rhdisk1
2
0 NORMAL DATADG_0002
/dev/rhdisk2
3
0 NORMAL DATADG_0003
/dev/rhdisk3
4
0 NORMAL DATADG_0004
/dev/rhdisk15
6
0 NORMAL DATADG_0006
/dev/rhdisk11
7
0 NORMAL DATADG_0007
/dev/rhdisk12
8
0 NORMAL DATADG_0008
/dev/rhdisk9
10
0 NORMAL DATADG_0010
/dev/rhdisk10
11
0 NORMAL DATADG_0011
/dev/rhdisk13
12
0 NORMAL DATADG_0012
/dev/rhdisk14
DISK_NUMBER REPAIR_TIMER STATE
NAME
PATH
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------17
0 NORMAL DATADG_0017
/dev/rhdisk4
18
0 NORMAL DATADG_0018
/dev/rhdisk5
32 / 67
19
20
21
0
1
2
0
8
9
0 NORMAL
0 NORMAL
0 NORMAL
0 NORMAL
12758 NORMAL
0 NORMAL
0 NORMAL
0 NORMAL
0 NORMAL
DATADG_0019
DATADG_0020
DATADG_0021
DATA_0000
DATA_0001
DATA_0002
/dev/rhdisk6
/dev/rhdisk7
/dev/rhdisk8
/dev/rhdisk50
/votedisk/nfs_vote
/dev/rhdisk53
/dev/rhdisk16
/dev/rhdisk17
DISK_NUMBER REPAIR_TIMER STATE
NAME
PATH
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------10
0 NORMAL
/dev/rhdisk18
11
0 NORMAL
/dev/rhdisk19
14
0 NORMAL
/dev/rhdisk22
13
0 NORMAL
/dev/rhdisk20
15
0 NORMAL
/dev/rhdisk23
16
0 NORMAL
/dev/rhdisk24
17
0 NORMAL
/dev/rhdisk25
29 rows selected.
3. Online votedisk
SQL> alter diskgroup data online disk DATA_0001;
Diskgroup altered.
4. Result
SQL> select disk_number,repair_timer,state,name,path from v$asm_disk order by name;
DISK_NUMBER REPAIR_TIMER STATE
NAME
PATH
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------0
0 NORMAL DATADG_0000
/dev/rhdisk0
1
0 NORMAL DATADG_0001
/dev/rhdisk1
2
0 NORMAL DATADG_0002
/dev/rhdisk2
3
0 NORMAL DATADG_0003
/dev/rhdisk3
4
0 NORMAL DATADG_0004
/dev/rhdisk15
6
0 NORMAL DATADG_0006
/dev/rhdisk11
7
0 NORMAL DATADG_0007
/dev/rhdisk12
33 / 67
8
10
11
12
0 NORMAL
0 NORMAL
0 NORMAL
0 NORMAL
DATADG_0008
DATADG_0010
DATADG_0011
DATADG_0012
/dev/rhdisk9
/dev/rhdisk10
/dev/rhdisk13
/dev/rhdisk14
DISK_NUMBER REPAIR_TIMER STATE
NAME
PATH
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------17
0 NORMAL DATADG_0017
/dev/rhdisk4
18
0 NORMAL DATADG_0018
/dev/rhdisk5
19
0 NORMAL DATADG_0019
/dev/rhdisk6
20
0 NORMAL DATADG_0020
/dev/rhdisk7
21
0 NORMAL DATADG_0021
/dev/rhdisk8
0
0 NORMAL DATA_0000
/dev/rhdisk50
1
0 NORMAL DATA_0001
/dev/rhdisk53
2
0 NORMAL DATA_0002
/votedisk/nfs_vote
8
0 NORMAL
/dev/rhdisk16
9
0 NORMAL
/dev/rhdisk17
10
0 NORMAL
/dev/rhdisk18
DISK_NUMBER REPAIR_TIMER STATE
NAME
PATH
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------11
0 NORMAL
/dev/rhdisk19
13
0 NORMAL
/dev/rhdisk20
14
0 NORMAL
/dev/rhdisk22
16
0 NORMAL
/dev/rhdisk24
17
0 NORMAL
/dev/rhdisk25
15
0 NORMAL
/dev/rhdisk23
28 rows selected.
ASMCMD> lsdsk -G data
Path
/dev/rhdisk50
/dev/rhdisk53
/votedisk/nfs_vote
Other comments:
If the record of data_0000 is missing in v$asm_disk, we can add it to DG by follow command.
alter diskgroup <vote diskgroup name> add disk '</path/name>'
34 / 67
force;
alter diskgroup data add disk '/dev/rhdisk53' force;
alter diskgroup data add disk '/dev/rhdisk50' force;
alter diskgroup vtdg add disk '/dev/rhdisk26' force;
alter diskgroup vtdg online disk VTDG_0004;
set linesize 1000
Col name format a25
select disk_number,repair_timer,state,name,path from v$asm_disk order by name;
alter diskgroup vtdg add disk '/dev/rhdisk26' force;
alter diskgroup vtdg add disk '/dev/rhdisk28' force;
0
3
/votedisk/quorum_vtdg
4
==
Xin Yu’s
0 NORMAL
VTDG_0000
0
0 NORMAL
/dev/rhdisk28
NORMAL
VTDG_0003
VTDG_0004
/dev/rhdisk26
一、给DG增加磁盘（如果磁盘之前曾经属于这个dg，需要dd if磁盘）
SQL> alter DISKGROUP data add failgroup FG2
disk
'/dev/rhdisk50';
Diskgroup altered.
二、给DG增加或者删除完磁盘以后，一定要等REBALANCE的进程结束，否则可能会造成库损坏。
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA STAT
POWER
ACTUAL
SOFAR
EST_WORK
EST_RATE
EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- -------------------- -------------------------------------------4 REBAL ERRS
1
ORA-15041
3.9
如何查看与 Tiebreaker 相关的日志
首先运行 ctnsap
cd /tmp/ctsupt/var/ct/IW/log/mc/IBM.ConfigRM
35 / 67
rpttr -odtic trace* > trace.output
trace.output 文件中包含了详细的 IBM.ConfigRM 服务的日志信息
还有一个命令：
# lsrpnode
Name OpState RSCTVersion
GSAS3 Online 3.2.0.0
GSAS2 Offline 3.2.0.0
GSAS1 Offline 3.2.0.0
3.10 在 7.1.3 中在线从 Multi-cast 转为 Uni-cast
Migrating from Multicast heartbeating to Unicast heartbeating
Even though Unicast heartbeating is default for new PowerHA v7.1.3 installs and for upgrades
from versions prior to v7.1, you will need to manually update the cluster if you wish to migrate
from Multicast heartbeating.
On the current primary node update the heartbeat type (we only have a single cluster in this
example)
p6lpar12:/#clmgr modify cluster HEARTBEAT_TYPE=unicast
p6lpar12:/#clmgr query cluster | grep HEARTBEAT_TYPE
HEARTBEAT_TYPE="UNICAST"
heartbeat type as Unicast
- a cluster query will show the
p6lpar12:/#lscluster -c | grep Mode
Communication Mode: multicast
not been performed its still using Multicast
- however as a synchronisation has
However on the current standby node
p6lpar11:/#clmgr query cluster | grep HEARTBEAT_TYPE
HEARTBEAT_TYPE="MULTICAST"
performed its still using Multicast
p6lpar11:/#lscluster -c | grep Mode
36 / 67
- as a synchronisation has not been
Communication Mode: multicast
performed its still using Multicast
- as a synchronisation has not been
NOTE: A synchronisation of the cluster must be performed from the current primary node
p6lpar12:/#clmgr sync cluster
Check the log to confirm the synchronisation completed successfully
p6lpar11:/#clmgr query cluster | grep HEARTBEAT_TYPE
HEARTBEAT_TYPE="UNICAST"
p6lpar11:/#lscluster -c | grep Mode
Communication Mode: unicast
now see Unicast heartbeating across the board
- following the synchronisation we can
To confirm check the full 'lscluster -c' output
p6lpar12:/#lscluster -c
Cluster Name: P6_Cluster
Cluster UUID: 0bca7144-73ee-11e3-9fc5-a6dd3f971c02
Number of nodes in cluster = 2
Cluster ID for node p6lpar11: 1
Primary IP address for node p6lpar11: x.y.148.80
Cluster ID for node p6lpar12: 2
Primary IP address for node p6lpar12: x.y.148.81
Number of disks in cluster = 1
Disk = hdisk3 UUID = 35c59abc-b294-b7ff-59f9-8b5932aa3806 cluster_major = 0
cluster_minor = 1
Multicast for site LOCAL: IPv4 228.y.148.80 IPv6 ff05::e4c4:9450
however its not used
37 / 67
- the Multicast IP is still listed,
Communication Mode: unicast
Unicast mode Smile
- confirmation you are now running in
Local node maximum capabilities: HNAME_CHG, UNICAST, IPV6, SITE
Effective cluster-wide capabilities: HNAME_CHG, UNICAST, IPV6, SITE
4
4.1
其他
与 PowerHA HyperSwap 有关的一些 ODM 组件：
hdisk162
00f74e9a069c8945
Name
HACMPsircol
caavg_private
active
注释
content
HACMPsircol:
root@BANCS1[/etc/objrepos]#lscluster -c
name = "BANCS1_cluster_sircol"
Cluster Name: BANCS1_cluster
id = 0
Cluster UUID: 77aa8904-f5fd-11e2-bfe7-e41f13fbddbd
uuid = "0"
Number of nodes in cluster = 2
ip_address = "228.16.58.155"
Cluster ID for node BANCS1: 1
repository = "00f74e9a069c8945"
Primary IP address for node BANCS1:
backup_repository = ""
172.16.58.155
Cluster ID for node BANCS2: 2
Primary IP address for node BANCS2:
172.16.58.156
Number of disks in cluster = 1
Disk
=
hdisk162
UUID
=
57634a1a-17ac-2f27-c981-e0430143e9a6
cluster_major = 0 cluster_minor = 1
Multicast for site LOCAL: IPv4 228.16.58.155 IPv6
ff05::e410:3a9
HACMPsite
root@BANCS1[/etc/objrepos]#odmget HACMPsite
HACMPsite:
id = 1
name = "Site1"
nodelist = "BANCS1"
38 / 67
dominance = "yes"
protect_type = "NONE"
on_merge = ""
merge_priority = 1
HACMPsite:
id = 2
name = "Site2"
nodelist = "BANCS2"
dominance = "no"
protect_type = "NONE"
on_merge = ""
merge_priority = 2
HACMPnode
HACMPnode:
name = "BANCS1"
object = "NAME_SERVER"
value = "FALSE"
node_id = 1
node_handle = 1
version = 14
…
HACMPnode:
name = "BANCS2"
object = "NAME_SERVER"
value = "FALSE"
node_id = 2
node_handle = 2
version = 14
HACMPsplitmerge
HACMPsplitmerge:
id = 0
policy = "action"
value = "Reboot"
HACMPsplitmerge:
id = 0
policy = "split"
value = "TieBreaker"
HACMPsplitmerge:
id = 0
policy = "merge"
value = "TieBreaker"
HACMPsplitmerge:
39 / 67
id = 0
policy = "tiebreaker"
value = "00f74e9a1aa3c3c5"
HACMPxd_ext_attr
HACMPxd_ext_attr:
identifier = "repmg"
field_type = "rdg_name"
value = "pha_9654001201rdg0"
HACMPxd_ext_attr:
identifier = "repmg"
field_type = "hypswp_disk"
value
=
"57634a1a-17ac-2f27-c981-e0430143e9a6"
HACMPxd_ext_attr:
identifier = "repmg"
field_type = "non_hypswp_disk"
value
"57634a1a-17ac-2f27-c981-e0430143e9a6"
HACMPxd_ext_attr:
identifier = "repmg"
field_type = "site"
value = "Site1"
HACMPxd_ext_attr:
identifier = "repmg"
field_type = "site"
value = "Site2"
HACMPxd_ext_attr:
identifier = "db2mg"
field_type = "VG"
value = "appvg"
HACMPxd_ext_attr:
identifier = "db2mg"
field_type = "VG"
value = "datavg"
HACMPxd_ext_attr:
identifier = "db2mg"
field_type = "rdg_name"
value = "pha_11653961161rdg0"
40 / 67
=
HACMPxd_ext_attr:
identifier = "db2mg"
field_type = "rdg_name"
value = "pha_11653971171rdg1"
HACMPxd_ext_attr:
identifier = "db2mg"
field_type = "rdg_name"
value = "pha_11653921121rdg2"
HACMPxd_ext_attr:
identifier = "db2mg"
field_type = "rdg_name"
value = "pha_11653931131rdg3"
HACMPxd_ext_attr:
identifier = "db2mg"
field_type = "rdg_name"
value = "pha_11653941141rdg4"
HACMPxd_ext_attr:
identifier = "db2mg"
field_type = "rdg_name"
value = "pha_11653951151rdg5"
HACMPxd_mirror_grou
HACMPxd_mirror_group:
p
xd_mirror_group_id = 9
xd_mirror_group_name = "repmg"
xd_mirror_group_mode = 1
xd_mirror_group_consistency = 1
xd_mirror_group_recovery_action = 2
xd_mirror_group_vendor_specific_identifier
= ""
xd_mirror_group_hyperswap_enabled = 1
xd_mirror_group_priority = 1
xd_mirror_group_type = 3
xd_mirror_group_hyperswap_unplanned_timeout
60
HACMPxd_mirror_group:
xd_mirror_group_id = 11
xd_mirror_group_name = "db2mg"
xd_mirror_group_mode = 1
41 / 67
=
xd_mirror_group_consistency = 1
xd_mirror_group_recovery_action = 1
xd_mirror_group_vendor_specific_identifier
= ""
xd_mirror_group_hyperswap_enabled = 1
xd_mirror_group_priority = 2
xd_mirror_group_type = 1
xd_mirror_group_hyperswap_unplanned_timeout
=
60
HACMPxd_mirror_grou
p_ss_relation
HACMPxd_mirror_group_ss_relation:
xd_mirror_group_id = 9
xd_storage_system_id = 7
HACMPxd_mirror_group_ss_relation:
xd_mirror_group_id = 9
xd_storage_system_id = 8
HACMPxd_mirror_group_ss_relation:
xd_mirror_group_id = 11
xd_storage_system_id = 7
HACMPxd_mirror_group_ss_relation:
xd_mirror_group_id = 11
xd_storage_system_id = 8
HACMPxd_storage_mo
dule
HACMPxd_storage_module:
xd_storage_tech_id = 5
xd_storage_module_id = 6
xd_module_path
=
"/usr/es/sbin/cluster/xd_generic/xd_ds8k_mm/libclxd
_ds8k_mm.so"
xd_module_entry_func
=
"clxd_ds8k_im_init_func"
xd_module_exit_func
=
"clxd_ds8k_im_exit_func"
xd_module_cfg_file
=
"/usr/es/sbin/cluster/xd_generic/xd_ds8k_mm/ds8k_i
nband_mm.cfg"
HACMPxd_storage_mo
dule_private
HACMPxd_storage_module_private:
dscli> lspprcpath 74
storage_tech_id = 5
Src Tgt State
storage_module_id = 6
============================================
type
=
"MG_PPRCPATHS_10_99_pha_1065301211rdg5"
value
42 / 67
=
"FF14,14
FF74,74
SS
Port Attached Port Tgt WWNN
=============
74
14
500507630BFFC1E2
Success
FF16
I0002
I0130
0030,0037,0130,0002,0002,0130,0037,0030"
74
14
Success
FF16
I0037
I0030
500507630BFFC1E2
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
dscli> lspprcpath 14
type
=
"MG_PPRCPATHS_10_99_pha_1065301211rdg5"
value
=
"FF15,15
Src Tgt State
SS
Port Attached Port Tgt WWNN
============================================
FF75,75
0030,0037,0130,0002,0002,0130,0037,0030"
=============
14
74
Success
FF76
I0030
I0037
FF76
I0130
I0002
500507630AFFC454
HACMPxd_storage_module_private:
14
storage_tech_id = 5
type
=
"MG_PPRCPATHS_10_99_pha_1065301211rdg5"
=
"FF13,13
FF73,73
0030,0037,0130,0002,0002,0130,0037,0030"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_10_99_pha_1065301211rdg5"
value
=
"FF10,10
FF70,70
0030,0037,0130,0002,0002,0130,0037,0030"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_10_99_pha_1065301211rdg5"
value
=
"FF11,11
FF71,71
0030,0037,0130,0002,0002,0130,0037,0030"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_10_99_pha_1065301211rdg5"
value
=
"FF12,12
0030,0037,0130,0002,0002,0130,0037,0030"
HACMPxd_storage_module_private:
storage_tech_id = 5
43 / 67
Success
500507630AFFC454
storage_module_id = 6
value
74
FF72,72
storage_module_id = 6
type
=
"MG_PPRCPATHS_10_99_pha_1065296161rdg0"
value
=
"
FF14,14
FF74,74
0030,0037,0130,0002,0002,0130,0037,0030"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_10_99_pha_1065297171rdg1"
value
=
"FF15,15
FF75,75
0030,0037,0130,0002,0002,0130,0037,0030"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_10_99_pha_10652981812rdg2"
value
=
"FF13,13
FF73,73
0030,0037,0130,0002,0002,0130,0037,0030"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_10_99_pha_1065299191rdg3"
value
=
"FF10,10
FF70,70
0030,0037,0130,0002,0002,0130,0037,0030"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_10_99_pha_1065300201rdg4"
value
=
"FF11,11
FF71,71
0030,0037,0130,0002,0002,0130,0037,0030"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_9_99_pha_9654001201rdg0"
value
44 / 67
=
"FF78,78
FF18,18
0002,0130,0037,0030,0030,0037,0130,0002"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_11_99_pha_11653961161rdg0"
value
=
"FF72,72
FF12,12
0002,0130,0037,0030,0030,0037,0130,0002"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_11_99_pha_11653971171rdg1"
value
=
"FF73,73
FF13,13
0002,0130,0037,0030,0030,0037,0130,0002"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_11_99_pha_11653921121rdg2"
value
=
"FF71,71
FF11,11
0002,0130,0037,0030,0030,0037,0130,0002"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_11_99_pha_11653931131rdg3"
value
=
"FF74,74
FF14,14
0002,0130,0037,0030,0030,0037,0130,0002"
HACMPxd_storage_module_private:
storage_tech_id = 5
storage_module_id = 6
type
=
"MG_PPRCPATHS_11_99_pha_11653941141rdg4"
value
=
"FF75,75
0002,0130,0037,0030,0030,0037,0130,0002"
HACMPxd_storage_module_private:
storage_tech_id = 5
45 / 67
FF15,15
storage_module_id = 6
type
=
"MG_PPRCPATHS_11_99_pha_11653951151rdg5"
value
=
"FF72,72
FF12,12
0002,0130,0037,0030,0030,0037,0130,0002"
HACMPxd_storage_syst
em
HACMPxd_storage_system:
xd_storage_tech_id = 5
xd_storage_system_id = 7
xd_storage_system_name = "DS8803"
xd_storage_vendor_unique_id
=
"IBM.2107-00000WT971"
xd_storage_system_site_affiliation = "Site1"
xd_storage_system_wwnn
=
"500507630BFFC1E2"
HACMPxd_storage_system:
xd_storage_tech_id = 5
xd_storage_system_id = 8
xd_storage_system_name = "DS8801"
xd_storage_vendor_unique_id
=
"IBM.2107-00000VF451"
xd_storage_system_site_affiliation = "Site2"
xd_storage_system_wwnn
=
"500507630AFFC454"
HACMPxd_storage_tec
hnology
HACMPxd_storage_technology:
st_id = 5
st_name = "ds8k_inband_mm"
st_description = "Metro Mirror Support
With In-band for DS8K"
st_supported_modes = 1
st_in_band = 1
st_authentication_model = 1
st_out_of_band_relationship = 0
st_supports_snmp = 0
st_requires_path_management = 1
st_supports_device_groups = 0
st_supports_composite_group = 0
st_supports_consistency_group = 1
st_supported_inter_connects = 0
st_version = "1.0"
st_hyperswap = 1
46 / 67
47 / 67
4.2
Active-Active 环境下脑裂后存储的恢复
DS8801
DS8805
dscli> mkpprcpath -remotedev IBM.2107-75VF451 -remotewwnn 500507630AFFC454 -srclss 34 -tgtlss 24 I0030:I0003
dscli> lspprcpath 34-36
Src Tgt State
SS
Port Attached Port Tgt WWNN
=========================================================
34 24 Success FF24 I0030 I0003
500507630AFFC454
35 25 Success FF25 I0030 I0003
500507630AFFC454
36 26 Success FF26 I0030 I0003
500507630AFFC454
dscli> resumepprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 3401:2401
CMUN03068E resumepprc: 3401:2401: Copy Services operation failure: not ready for resynchronization
dscli> resumepprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 3501:2501
48 / 67
CMUN03068E resumepprc: 3501:2501: Copy Services operation failure: not ready for resynchronization
dscli> resumepprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 3502:2502
CMUN03068E resumepprc: 3502:2502: Copy Services operation failure: not ready for resynchronization
dscli> failbackpprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 3401:2401
CMUC00197I failbackpprc: Remote Mirror and Copy pair 3401:2401 successfully failed back.
dscli> failbackpprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 3501:2501
CMUN03030E failbackpprc: 3501:2501: Copy Services operation failure: failback primary volume not suspended
dscli> pausepprc -remotedev IBM.2107-75VF451 -at src 3501:2501
CMUN03012E pausepprc: 3501:2501: A Remote Mirror and Copy operation has failed or a channel extender being used for a
copy operation is reporting an error, and the command cannot be completed.
dscli> failbackpprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 3501:2501
CMUC00197I failbackpprc: Remote Mirror and Copy pair 3501:2501 successfully failed back.
dscli> pausepprc -remotedev IBM.2107-75VF451 -at src 3502:2502
CMUN03012E pausepprc: 3502:2502: A Remote Mirror and Copy operation has failed or a channel extender being used for a
copy operation is reporting an error, and the command cannot be completed.
dscli> lspprc 3401-3603
ID
State
Reason
Type
SourceLSS Timeout (secs) Critical Mode First Pass Status
=======================================================================================================
3401:2401 Full Duplex -
Metro Mirror 34
60
Disabled
Invalid
3501:2501 Full Duplex -
Metro Mirror 35
60
Disabled
Invalid
3502:2502 Suspended
Host Source Metro Mirror 35
60
Disabled
3601:2601 Full Duplex -
Metro Mirror 36
60
Disabled
Invalid
3602:2602 Full Duplex -
Metro Mirror 36
60
Disabled
Invalid
dscli> failbackpprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 3502:2502
CMUC00197I failbackpprc: Remote Mirror and Copy pair 3502:2502 successfully failed back.
dscli> lspprc 3401-3603
49 / 67
Invalid
ID
State
Reason Type
SourceLSS Timeout (secs) Critical Mode First Pass Status
==================================================================================================
DS8801
3401:2401 Full Duplex -
Metro Mirror 34
60
Disabled
Invalid
3501:2501 Full Duplex -
Metro Mirror 35
60
Disabled
Invalid
3502:2502 Full Duplex -
Metro Mirror 35
60
Disabled
Invalid
3601:2601 Full Duplex -
Metro Mirror 36
60
Disabled
Invalid
3602:2602 Full Duplex -
Metro Mirror 36
60
Disabled
Invalid
DS8805
dscli> mkpprcpath -remotedev IBM.2107-75VF451 -remotewwnn 500507630AFFC454 -srclss 22 -tgtlss 12 I0030:I0003
CMUC00149I mkpprcpath: Remote Mirror and Copy path 22:12 successfully established.
dscli> resumepprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 2201:1201
CMUC00158I resumepprc: Remote Mirror and Copy volume pair 2201:1201 relationship successfully resumed. This message is being returned
before the copy completes.
50 / 67
dscli> resumepprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 2202:1202
CMUC00158I resumepprc: Remote Mirror and Copy volume pair 2202:1202 relationship successfully resumed. This message is being returned
before the copy completes.
dscli> pausepprc -remotedev IBM.2107-75VF451 -at src 2401:1401
CMUN03012E pausepprc: 2401:1401: A Remote Mirror and Copy operation has failed or a channel extender being used for a copy operation is
reporting an error, and the command cannot be completed.
dscli> lspprc 2001-2402
ID
State
Reason
Type
SourceLSS Timeout (secs) Critical Mode First Pass Status
=======================================================================================================
2001:1001 Full Duplex -
Metro Mirror 20
60
Disabled
Invalid
2101:1101 Full Duplex -
Metro Mirror 21
60
Disabled
Invalid
2102:1102 Full Duplex -
Metro Mirror 21
60
Disabled
Invalid
2201:1201 Full Duplex -
Metro Mirror 22
60
Disabled
Invalid
Metro Mirror 22
60
Disabled
Invalid
2202:1202 Full Duplex 2401:1401 Suspended
Host Source Metro Mirror 24
60
Disabled
Invalid
dscli> failbackpprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 2401:1401
CMUC00197I failbackpprc: Remote Mirror and Copy pair 2401:1401 successfully failed back.
dscli> lspprc 2001-2402
ID
State
Reason Type
SourceLSS Timeout (secs) Critical Mode First Pass Status
==================================================================================================
51 / 67
2001:1001 Full Duplex -
Metro Mirror 20
60
Disabled
Invalid
2101:1101 Full Duplex -
Metro Mirror 21
60
Disabled
Invalid
2102:1102 Full Duplex -
Metro Mirror 21
60
Disabled
Invalid
2201:1201 Full Duplex -
Metro Mirror 22
60
Disabled
Invalid
2202:1202 Full Duplex -
Metro Mirror 22
60
Disabled
Invalid
2401:1401 Full Duplex -
Metro Mirror 24
60
Disabled
Invalid
有时候遇到 mping 的错误，需要检查是否有某个节点多一个网卡。
4.3
Actice-Standby 的脑裂场景
Site1
Network switch1
Site2
Network link
Between sites
root@node2[/]#lspprc -p hdisk5
path
Network switch2
Node2
Node3
SAN-switch1
FC link
Between sites
WWNN
Metro Mirror
DS8801
DS8805
path
group status
=======================================================
500507630affc454 0x12 0x01
1(s)
500507630bffc4c8 0x22 0x01
path
path path
group id id
IBM.2107-75VF451
500507630AFFC454
VOL
group id
0
SAN-switch2
LSS
SECONDARY
PRIMARY
parent connection
status
=====================================================================
IBM.2107-75XP411
500507630BFFC4C8
0
0
Enabled
fscsi3 500507630a00c454,4012400100000000
1
1
Enabled
fscsi2 500507630b0844c8,4022400100000000
SVC storage
As tie-breaker
Hdisk5 1201
Hdisk5 2201(testvg)
root@node2[/]#lspprc -p hdisk6
Hdisk6 1202
Hdisk6 2202(testvg)
path
Hdisk7 1401
Hdisk7 2401 (caa)
52 / 67
group id
WWNN
LSS
VOL
path
group status
=======================================================
0
500507630affc454 0x12 0x02
1(s)
500507630bffc4c8 0x22 0x02
path
path path
group id id
SECONDARY
PRIMARY
parent connection
status
=====================================================================
0
0
Enabled
fscsi3 500507630a00c454,4012400200000000
1
1
Enabled
fscsi2 500507630b0844c8,4022400200000000
root@node2[/]#lspprc -p hdisk7
path
WWNN
LSS
VOL
group id
path
group status
=======================================================
0
500507630affc454 0x14 0x01
1(s)
500507630bffc4c8 0x24 0x01
path
path path
group id id
SECONDARY
PRIMARY
parent connection
status
=====================================================================
53 / 67
0
0
Enabled
fscsi3 500507630a00c454,4014400100000000
1
1
Enabled
fscsi2 500507630b0844c8,4024400100000000
Site1
Network switch1
Site2
Network link
Between sites
root@node3[/var/hacmp/adm]#lspprc -p hdisk5
lpath
Network switch2
WWNN
LSS
VOL
group id
path
group status
=======================================================
Node2
Node3
SAN-switch1
FC link
Between sites
SAN-switch2
0
Lock the 1(s)
tie-breaker
500507630affc454 0x12 0x01
500507630bffc4c8 0x22 0x01
path
Metro Mirror
path path
group id id
SECONDARY
PRIMARY
parent connection
status
=====================================================================
DS8801
DS8805
IBM.2107-75VF451
500507630AFFC454
IBM.2107-75XP411
500507630BFFC4C8
0
0
Failed
1
1
Enabled
fscsi3 500507630a00c454,4012400100000000
fscsi2 500507630b0844c8,4022400100000000
root@node3[/var/hacmp/adm]#spprc -p hdisk6
path
SVC storage
As tie-breaker
WWNN
LSS
VOL
group id
Hdisk5 1201
Hdisk5 2201(testvg)
Hdisk6 1202
Hdisk6 2202(testvg)
Hdisk7 1401
Hdisk7 2401 (caa)
path
group status
=======================================================
0
500507630affc454 0x12 0x02
1(s)
500507630bffc4c8 0x22 0x02
path
path path
group id id
SECONDARY
PRIMARY
parent connection
status
=====================================================================
0
0
Failed
1
1
Enabled
fscsi3 500507630a00c454,4012400200000000
fscsi2 500507630b0844c8,4022400200000000
root@node3[/var/hacmp/adm]#lspprc -p hdisk7
path
54 / 67
WWNN
LSS
VOL
path
group id
group status
=======================================================
0
500507630affc454 0x14 0x01
1(s)
500507630bffc4c8 0x24 0x01
SECONDARY
PRIMARY,
SUSPENDED,
OOS
path
path path
group id id
parent connection
status
=====================================================================
0
0
Failed
1
1
Enabled
fscsi3 500507630a00c454,4014400100000000
fscsi2 500507630b0844c8,4024400100000000
root@node3[/var/hacmp/adm]#lspprc -Ao
hdisk#
PPRC
state
Primary
path group
Secondary
path group
ID
Primary Storage
Secondary Storage
WWNN
ID
hdisk7
Active
1(s)
0
500507630bffc4c8 500507630affc454
hdisk5
Active
1(s)
0
500507630bffc4c8 500507630affc454
hdisk6
Active
1(s)
0
500507630bffc4c8 500507630affc454
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
root@node2[/]#lspprc -p hdisk5
path
WWNN
LSS
VOL
group id
path
group status
=======================================================
55 / 67
WWNN
0(s)
500507630affc454 0x12 0x01
-1
500507630bffc4c8 0x22 0x01
SECONDARY
path
path path
group id id
parent connection
status
=====================================================================
0
0
Enabled
fscsi3 500507630a00c454,4012400100000000
root@node2[/]#lspprc -p hdisk6
path
WWNN
LSS
VOL
group id
path
group status
=======================================================
0(s)
500507630affc454 0x12 0x02
-1
500507630bffc4c8 0x22 0x02
path
path path
group id id
SECONDARY
parent connection
status
=====================================================================
0
0
Enabled
fscsi3 500507630a00c454,4012400200000000
root@node2[/]#lspprc -p hdisk7
path
WWNN
LSS
VOL
group id
path
group status
=======================================================
0(s)
500507630affc454 0x14 0x01
PRIMARY,
SUSPENDED,
OOS
-1
path
56 / 67
500507630bffc4c8 0x24 0x01
path path
parent connection
group id id
status
=====================================================================
0
0
Enabled
fscsi3 500507630a00c454,4014400100000000
root@node2[/]#lspv
hdisk0
00cf8de69e362f55
rootvg
hdisk1
00cf8de6c7566825
None
hdisk5
00cf8de6057a5612
testvg
hdisk6
00cf8de6c7565919
testvg
hdisk7
00cf8de6058a8553
caavg_private
环境恢复
1. 网络，SAN 恢复
dscli> lspprcpath 12 14
dscli> lspprc 2201 2202 2401
Src Tgt State
ID
SS
Port Attached Port Tgt WWNN
State
Reason
Type
active
active
SourceLSS Timeout
=========================================================
(secs) Critical Mode First Pass Status
12 22 Success FF22 I0003 I0030
500507630BFFC4C8
================================================================================
14 24 Success FF24 I0003 I0030
500507630BFFC4C8
======================================
dscli> lspprc 1201 1202 1401
ID
State
2201:1201 Full Duplex Reason
Type
SourceLSS Timeout
Disabled
2202:1202 Full Duplex -
=========================================================================
Disabled
=====================================
2401:1401 Suspended
Disabled
57 / 67
Invalid
Disabled
Metro Mirror 22
60
Invalid
Internal Conditions Target Metro Mirror 24
Invalid
dscli> lspprcpath 22 24
Metro Mirror 22
unknown
Invalid
2202:1202 Target Full Duplex Disabled
unknown
Invalid
2201:1201 Target Full Duplex Disabled
Host Source Metro Mirror 14
60
Invalid
(secs) Critical Mode First Pass Status
1401:2401 Suspended
Metro Mirror 22
Src Tgt State
SS
Port Attached Port Tgt WWNN
=========================================================
Metro Mirror 22
unknown
22 12 Success FF12 I0030 I0003
24 14 Failed FF14 -
-
500507630AFFC454
500507630AFFC454
60
dscli> lssi
Name
dscli> lssi
ID
Storage Unit
Model WWNN
State
Name
ID
Storage Unit
Model WWNN
State ESSNet
ESSNet
================================================================================
=========================================================================
=
========
DS8800-05 IBM.2107-75XP411 IBM.2107-75XP410 951
DS8800-01 IBM.2107-75VF451 IBM.2107-75VF450 951
500507630BFFC4C8 Online Enabled
500507630AFFC454 Online
Enabled
dscli>
mkpprcpath
-remotedev
500507630AFFC454 -srclss 24 -tgtlss 14
IBM.2107-75VF451
-remotewwnn
I0030:I0003
CMUC00149I mkpprcpath: Remote Mirror and Copy path 24:14 successfully established.
dscli> failbackpprc -remotedev IBM.2107-75VF451 -type mmir -tgtse 2401:1401
CMUC00197I failbackpprc: Remote Mirror and Copy pair 2401:1401 successfully failed
back.
dscli> lspprc 2201 2202 2401
ID
State
Reason Type
SourceLSS Timeout (secs) Critical
Mode First Pass Status
=======================================================================
===========================
2201:1201 Full Duplex -
Metro Mirror 22
60
Disabled
Metro Mirror 22
60
Disabled
Metro Mirror 24
60
Disabled
Invalid
2202:1202 Full Duplex Invalid
2401:1401 Full Duplex Invalid
dscli> lspprcpath 22 24
Src Tgt State
58 / 67
SS
Port Attached Port Tgt WWNN
=========================================================
22
12 Success FF12 I0030 I0003
500507630AFFC454
24
14 Success FF14 I0030 I0003
500507630AFFC454
Reboot node2
root@node2[/]#lscluster -c
Cluster Name: testcluster
Cluster UUID: ed3706ce-f2ac-11e2-a334-2a5c29a0ae0b
Number of nodes in cluster = 2
Cluster ID for node node2: 1
Primary IP address for node node2: 172.16.25.92
Cluster ID for node node3: 2
Primary IP address for node node3: 172.16.25.95
Number of disks in cluster = 1
Disk
=
hdisk7
UUID
c3e12504-4075-4e3b-f19c-6ddf6be99a6e cluster_major =
cluster_minor = 1
Multicast for site LOCAL: IPv4 228.16.25.92 IPv6 ff05::e410:195c
=
0
root@node2[/]#lspprc -p hdisk5
path
WWNN
LSS VOL
path
group id
group status
=======================================================
0
500507630affc454 0x12 0x01
SECONDARY
1(s)
500507630bffc4c8 0x22 0x01
PRIMARY
path
path path
group id id
status
59 / 67
parent connection
root@node3[/var/hacmp/adm]#lscluster -c
Cluster Name: testcluster
Cluster UUID: ed3706ce-f2ac-11e2-a334-2a5c29a0ae0b
Number of nodes in cluster = 2
Cluster ID for node node2: 1
Primary IP address for node node2: 172.16.25.92
Cluster ID for node node3: 2
Primary IP address for node node3: 172.16.25.95
Number of disks in cluster = 1
Disk = hdisk7 UUID = c3e12504-4075-4e3b-f19c-6ddf6be99a6e
cluster_major = 0 cluster_minor = 1
Multicast for site LOCAL: IPv4 228.16.25.92 IPv6 ff05::e410:195c
=======================================================
==============
0
0
Enabled
fscsi3
500507630a00c454,4012400100000000
1
1
Enabled
fscsi2
500507630b0844c8,4022400100000000
root@node2[/]#lspprc -p hdisk6
path
WWNN
LSS VOL
path
group id
group status
=======================================================
0
500507630affc454 0x12 0x02
SECONDARY
1(s)
500507630bffc4c8 0x22 0x02
PRIMARY
path
path path
parent connection
group id id
status
=======================================================
==============
0
0
Enabled
fscsi3
500507630a00c454,4012400200000000
1
1
Enabled
fscsi2
500507630b0844c8,4022400200000000
root@node2[/]#lspprc -p hdisk7
path
WWNN
LSS VOL
path
group id
group status
=======================================================
0
500507630affc454 0x14 0x01
SECONDARY
60 / 67
1(s)
500507630bffc4c8 0x24 0x01
PRIMARY
path
path path
parent connection
group id id
status
=======================================================
==============
0
0
Enabled
fscsi3
500507630a00c454,4014400100000000
1
1
Enabled
fscsi2
500507630b0844c8,4024400100000000
root@node2[/]#lspprc -Ao
hdisk#
PPRC
Primary
Secondary
Primary
Storage Secondary Storage
state
path group
path group
WWNN
WWNN
ID
ID
hdisk5
Active
1(s)
0
500507630bffc4c8 500507630affc454
hdisk7
Active
1(s)
0
500507630bffc4c8 500507630affc454
hdisk6
Active
1(s)
0
500507630bffc4c8 500507630affc454
Start PowerHA service on node2
61 / 67
4.4
常用的脚本
HyperSwap 测试用的七个脚本:
cat readme.sh
#manage_disk_drivers -d 2107DS8K -o AIX_AAPCM
#echo "kern.debug????????????? /tmp/syslog.phake rotate size 400k files 10">>/etc/syslog.conf;>/tmp/syslog.phake; stopsrc -s syslogd; sleep 2; startsrc -s syslogd
#1.sh ? ? ?Change PV Propertits and combie hdisk path
#2.sh ? ? ?Check HA services/RG/RDG/CRS Voting device
#3.sh ? ? ?Check Logs location
#4.sh ? ? ?Swap MG Manually
#5.sh ? ? ?Check PV LUN_ID?
#6.sh ? ? ?Check MG planed_unplaned TM setting
#7.sh ? ? ?Continuous switching MG
脚本 1：
# cat 1.sh
## Change ownership to grid and oracle
for fc in `lsdev | grep -i fscsi | awk '{print $1}'`
do
chdev -l $fc -a dyntrk=yes -a fc_err_recov=fast_fail -P
done
for pv in `lsdev -Cc disk | grep -i mpio | awk '{print $1}'`
do
chdev -l $fc -a dyntrk=yes -a fc_err_recov=fast_fail -P
62 / 67
chdev -l $pv -a reserve_policy=no_reserve -a algorithm=round_robin
chdev -l $pv -a san_rep_cfg=migrate_disk -U
#chdev -l $pv -a san_rep_cfg=revert_disk -U
chdev -l $pv -a pv=yes
chown grid:oinstall /dev/r$pv
chmod g+rw /dev/r$pv
done
[email protected]
脚本 2：
# cat 2.sh
echo "=====================================Hdisk to RDG====================================================== "
/tmp/testb/getrepdisklst -c 100
echo "========================================================================================================="
echo "=====================================HA services state===================================================="
lssrc -a|egrep "StorageRM|cthags|ctrmc|clcomd|clconfd|clxd"
echo "========================================================================================================="
echo "======================================RG State=========================================================="
/usr/es/sbin/cluster/utilities/clRGinfo
echo "========================================================================================================="
echo "=======================================RDG State========================================================"
/tmp/testb/getrepgrplst -c 100
63 / 67
echo "========================================================================================================="
echo "========================================Ha State========================================================="
lssrc -ls clstrmgrES|grep stat
echo "========================================================================================================="
echo "=========================================CRS Voteing device=============================================="
crsctl query css votedisk?
#cd?/u01/app/11.2.0/grid/log/aa1/cssd;grep "master node" ocssd.log
#cd?/u01/app/11.2.0/grid/log/aa3/cssd;grep "master node" ocssd.log
脚本 3：
# cat 3.sh
tail -f /var/hacmp/log/hacmp.out
tail -f /var/hacmp/xd/log/clxd.log
tail -f /var/adm/ras/syslog.caa
tail -f /tmp/syslog.phake
tail -f /var/hacmp/xd/log/clxd_debug.log ?
tail -f /var/hacmp/adm/cluster.log???????
tail -f /u01/app/11.2.0/grid/log/aa1/alertaa1.log?
64 / 67
脚本 4
# cat 4.sh
date
lspprc -Ao
/usr/es/sbin/cluster/xd_generic/xd_cli/cl_clxd_manage_mg_smit -t 'user? '? -m 'mg1' -o 'swap'
/usr/es/sbin/cluster/xd_generic/xd_cli/cl_clxd_manage_mg_smit -t 'repository'? -m 'caamg' -o 'swap'
echo "Finish do the? swap operation"
date
banner Finish!
lspprc -Ao
脚本 5：
# cat 5.sh
for pv in `lsdev -Cc disk | grep -i mpio | awk '{print $1}'`
do
echo "$pv";lscfg -vpl $pv|egrep "Serial|Z7"
done
脚本 6
cat 6.sh
cat /usr/es/sbin/cluster/xd_generic/xd_ds8k_mm/ds8k_inband_mm.cfg|grep -i "PLANNED_HYPERSWAP_TIMEOUT"
cd /etc/objrepos/;odmget HACMPxd_mirror_group |grep -i "xd_mirror_group_hyperswap_unplanned_timeout"
脚本 7：
65 / 67
# cat swap.sh
i=1
while [ $i -lt 10000 ]
do
banner Begin!
echo "Begin do the $i times swap operation"
date
/usr/es/sbin/cluster/xd_generic/xd_cli/cl_clxd_manage_mg_smit -t 'user? '? -m 'mg1' -o 'swap'
/usr/es/sbin/cluster/xd_generic/xd_cli/cl_clxd_manage_mg_smit -t 'repository'? -m 'caamg' -o 'swap'
echo "Finish do the $i swap operation"
banner Finish!
date
sleep 18
i=`expr $i + 1`
done
4.5
当出现存储切换未达到期望结果时，需要进一步采集 trace 与 dev 沟通的方法
########### Before test #########
>/var/hacmp/xd/log/clxd.log
>/var/hacmp/xd/log/clxd_debug.log
>/var/hacmp/adm/cluster.log
>/var/hacmp/log/hacmp.out
>/tmp/syslog.phake
errclear 0
66 / 67
rm -rf /tmp/traces
ctctrl memtracebufsize=1000000 -c scsidiskdd -r -c aixdiskpcmke -c storfwork
########### After test #########
errpt > errpt.out
errpt -a > errpt_a.out
cp /var/hacmp/xd/log/clxd.log .
cp /var/hacmp/xd/log/clxd_debug.log .
cp /var/hacmp/adm/cluster.log .
cp /var/hacmp/log/hacmp.out .
cp /tmp/syslog.phake* .
ctctrl -r memtracemax memtracebufsize=1000000 -c scsidiskdd -c aixdiskpcmke -c storfwork -D
-d /tmp/traces
livedumpstart symptom=busted -c aixdiskpcmke -c storfwork -c scsidiskdd -r
mv /tmp/traces .
#将 livedumpstart 生成的文件也放入保存目录中，然后打包发给 Dev. Team.
67 / 67