http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 1 [Check]
20 ○
30 ○
40 ○
50 ○
60 ○
70 ○
80 ○
90 ○
100 ○
110 ○
120 ○
130 ○
140 ○
150 ○
160 ○
170 ○
180 ○
190 ○
200 ○
210 ○
220 ○
230 ○
240 ○
250 ○
260 ○
270 ○
280 ○
290 ○
300 ○
310 ○
320 ○
330 ○
340 ○
350 ○
360 ○
370 ○
380 ○
390 ○
400 ○
Ver.1.00
2016/11/15
【想定するサーバのスペックとネットワーク構成】
CPU: 2Core 以上
Memory: 2GB 以上
DVD ドライブ: 1台
HDD 1: 8GB 以上 （1st Disk, OS 用）
HDD 2: 1GB 以上 （swap 用）
HDD 3: 1GB 以上 （2nd Disk, データ用）
HDD 4: 1GB 以上 （3rd Disk, データ用）
HDD 5: 1GB 以上 （4th Disk, データ用）
HDD 6: 1GB 以上 （5th Disk, データ用）
NIC 1: 1Gbps 以上、サービス用セグメント
（IBM
NIC 2: 1Gbps 以上、インターコネクト用セグメント（IBM
NIC 3: 1Gbps 以上、サービス用セグメント
（IBM
NIC 4: 1Gbps 以上、インターコネクト用セグメント（IBM
(eth1,eth3)bond1
.2
iscsitgt01a.example.com
Bluemix(SoftLayer)
Bluemix(SoftLayer)
Bluemix(SoftLayer)
Bluemix(SoftLayer)
Swap Disk
Disk
Disk
Disk
Disk
LVM(vg0)
DRBD
LVM(vg1)
LVM(vg0)
DRBD
LVM(vg1)
iSCSI
lun0,1,2,3
Snapshot
.57
(eth0,eth2)bond0
Private
Public
Private
Public
VLAN）へ接続
VLAN）へ接続
VLAN）へ接続
VLAN）へ接続
bond1(eth1,eth3)
.3
iscsitgt01s.example.com
1st Disk: OS
2nd
3rd
4th
5th
では
では
では
では
[インターコネクト・セグメント]
192.168.1.0/24
MTU: 9000
1st Disk: OS
/dev/sda
Swap Disk
/dev/sdb
2nd
3rd
4th
5th
/dev/sdc
/dev/sdd
/dev/sde
/dev/sdf
iSCSI
lun0,1,2,3
Snapshot
.58
VIP
bond0(eth0,eth2)
←---.59 -------------------→
Disk
Disk
Disk
Disk
Router
.1
InitiatorName:
Initiator01
Initiator02
Initiator03
Initiator04
NTP1: 10.0.77.54
NTP2:
NTP3:
NTP4:
DNS1: 10.0.80.11
DNS2: 10.0.80.12
[サービス・セグメント]
10.110.88.0/26
MTU: 9000
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 2 410
420
430
440
450
460
470
480
490
500
510
520
530
540
550
560
570
580
590
600
610
620
630
640
650
660
670
○
○
※　当文書内で緑色にした部分は、環境に合わせて読み替えたり、カスタマイズ（名前を変えたり、実行するしないを選択）
する部分を表します。ただし、日付や注目していない UUID 等は除きます。
※　IBM Bluemix(SoftLayer) のベアメタルサーバで NIC を冗長化した場合、
「NIC 1 (eth0)」と「NIC 3 (eth2)」、「NIC 2 (eth1)」と「NIC 4 (eth3)」が LAG で束ねられています。
※　本手順書では、LAG の設定がない前提としますが、LAG 対応させるための設定方法は注記しておきます。
※　IBM Bluemix(SoftLayer) の仮想サーバは、ベアメタルサーバと比較して、主に以下の相違点があります。
・「NIC 3」と「NIC 4」を追加できません。
・MTU は 1500 までしかサポートされません。
・ローカルストレージのデバイス名が異なります。
○
【共有ストレージの構成】
○
○
○
○
○
○
○
○
○
○
○
○
/dev/sdc1
/dev/sdd1
/dev/sde1
/dev/sdf1
vg0
/dev/vg0/lv-drbd0
/dev/drbd0
vg1
/dev/vg1/lv-lun0
/dev/vg1/lv-lun1
/dev/vg1/lv-lun2
/dev/vg1/lv-lun3
LVM 物理ボリューム
LVM 物理ボリューム
LVM 物理ボリューム
LVM 物理ボリューム
LVM ボリュームグループ
LVM 論理ボリューム（DRBD 用ブロックデバイスとして使用）
DRBD リソース（LVM 物理ボリュームとして使用）
DRBD 上のボリュームグループ
DRBD 上の LVM 論理ボリューム（lun0 としてエクスポート）
DRBD 上の LVM 論理ボリューム（lun1 としてエクスポート）
DRBD 上の LVM 論理ボリューム（lun2 としてエクスポート）
DRBD 上の LVM 論理ボリューム（lun3 としてエクスポート）
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 3 680
690
700
710
720
730
740
750
760
770
780
790
800
810
820
830
840
850
860
870
880
890
900
910
920
930
940
950
960
970
980
990
1000
1010
1020
1030
1040
1050
1060
1070
○
【OS のインストールと初期設定】
○
インストーラを DVD ドライブにセットし、サーバを起動します。
a,s
○
a,s
○
a,s
a,s
a,s
# V834394-01.iso (Oracle Linux 7.3)
インストーラの起動メニューが表示されたら60秒以内に「Tab」キーを押下します。
# Tab
起動オプションを以下のように編集し、「Enter」キーを押下します。
# vmlinuz … rd.live.check quiet
# ↓
# vmlinuz … net.ifnames=0 biosdevname=0 selinux=0 vconsole.keymap=jp106
※　英語キーボードとして認識されている状態なので、「=」を入力するには「^」を押下します。
○
a,s
○
a,s
○
a,s
anaconda の「Welcome」画面が出てきたら「Ctrl + Alt + F3」キーを押下し、シェルに移行します。
# Ctrl + Alt + F3
[anaconda root@localhost /]#
HDD の情報を確認します。
fdisk -l | grep ^Disk | sort
Disk /dev/mapper/live-base: 2147 MB, 2147483648 bytes, 4194304 sectors
Disk /dev/mapper/live-rw: 2147 MB, 2147483648 bytes, 4194304 sectors
Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors
Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors
Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sdd: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sde: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sdf: 107.4 GB, 107374182400 bytes, 209715200 sectors
パーティションを作成します。
fdisk -H 64 -S 32 /dev/sda
Welcome to fdisk (util-linux 2.23.2).
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 4 1080
1090
1100
1110
1120
1130
1140
1150
1160
1170
1180
1190
1200
1210
1220
1230
1240
1250
1260
1270
1280
1290
1300
1310
1320
1330
1340
1350
1360
1370
1380
1390
1400
1410
1420
1430
1440
1450
1460
1470
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x2a058c02.
a,s
#Command
o
(m for help): o
Building a new DOS disklabel with disk identifier 0xc9c2368a.
a,s
#Command
n
(m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
#Select
[Enter]
(default p): [Enter]
Using default response p
#Partition
[Enter] number (1-4, default 1): [Enter]
#First
[Enter]
sector (2048-33554431, default 2048): [Enter]
Using default value 2048
#Last
+500M
sector, +sectors or +size{K,M,G} (2048-33554431, default 33554431): +500M
Partition 1 of type Linux and of size 500 MiB is set
a,s
a,s
a,s
a,s
a,s
#Command
a
(m for help): a
Selected partition 1
a,s
#Command
n
(m for help): n
Partition type:
p primary (1 primary, 0 extended, 3 free)
e extended
#Select
[Enter]
(default p): [Enter]
Using default response p
#Partition
[Enter] number (2-4, default 2): [Enter]
#First
[Enter]
sector (1026048-33554431, default 1026048): [Enter]
Using default value 1026048
#Last
[Enter]
sector, +sectors or +size{K,M,G} (1026048-33554431, default 33554431): [Enter]
Using default value 33554431
Partition 2 of type Linux and of size 15.5 GiB is set
a,s
a,s
a,s
a,s
a,s
#Command
p
(m for help): p
Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 5 1480
1490
1500
1510
1520
1530
1540
1550
1560
1570
1580
1590
1600
1610
1620
1630
1640
1650
1660
1670
1680
1690
1700
1710
1720
1730
1740
1750
1760
1770
1780
1790
1800
1810
1820
1830
1840
1850
1860
1870
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xc9c2368a
Device Boot
/dev/sda1
*
/dev/sda2
a,s
Start
2048
1026048
End
1026047
33554431
Blocks
512000
16264192
Id System
83 Linux
83 Linux
#Command
w
(m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
a,s
fdisk -H 64 -S 32 /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x2a058c02.
a,s
#Command
o
(m for help): o
Building a new DOS disklabel with disk identifier 0xb3afd860.
a,s
#Command
n
(m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
#Select
[Enter]
(default p): [Enter]
Using default response p
#Partition
[Enter] number (1-4, default 1): [Enter]
#First
[Enter]
sector (2048-2097151, default 2048): [Enter]
Using default value 2048
#Last
[Enter]
sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): [Enter]
Using default value 2097151
Partition 1 of type Linux and of size 1023 MiB is set
a,s
a,s
a,s
a,s
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 6 1880
1890
1900
1910
1920
1930
1940
1950
1960
1970
1980
1990
2000
2010
2020
2030
2040
2050
2060
2070
2080
2090
2100
2110
2120
2130
2140
2150
2160
2170
2180
2190
2200
2210
2220
2230
2240
2250
2260
2270
a,s
a,s
a,s
#Command
t
(m for help): t
Selected partition 1
#Hex
82code (type L to list all codes): 82
Changed type of partition 'Linux' to 'Linux swap / Solaris'
#Command
p
(m for help): p
Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xb3afd860
Device Boot
/dev/sdb1
a,s
Start
2048
End
2097151
Blocks
1047552
Id System
82 Linux swap / Solaris
#Command
w
(m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
a,s
○
a,s
fdisk -l | grep /dev/ | sort
/dev/sda1
*
2048
1026047
512000 83 Linux
/dev/sda2
1026048
33554431
16264192 83 Linux
/dev/sdb1
2048
2097151
1047552 82 Linux swap / Solaris
Disk /dev/mapper/live-base: 2147 MB, 2147483648 bytes, 4194304 sectors
Disk /dev/mapper/live-rw: 2147 MB, 2147483648 bytes, 4194304 sectors
Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors
Disk /dev/sdb: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sdd: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sde: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sdf: 107.4 GB, 107374182400 bytes, 209715200 sectors
「Ctrl + Alt + F6」キーを押下し、anaconda の「welcome」画面に戻ります。
# Ctrl + Alt + F6
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 7 2280
2290
2300
2310
2320
2330
2340
2350
2360
2370
2380
2390
2400
2410
2420
2430
2440
2450
2460
2470
2480
2490
2500
2510
2520
2530
2540
2550
2560
2570
2580
2590
2600
2610
2620
2630
2640
2650
2660
2670
○
a,s
○
a,s
○
a,s
○
a,s
a,s
a,s
○
a,s
○
a,s
○
a,s
○
a,s
○
「English (United States)」が選択されていることを確認し、「Continue」を選択します。
# Continue
「DATE & TIME」を選択し、「Asia / Tokyo」を選択します。
# DATE & TIME: Asia / Tokyo
「KEYBOARD」を選択し、「Japanese (OADG 109A)」のみ選ばれているように選択します。
# KEYBOARD: Japanese (OADG 109A)
「INSTALLATION DESTINATION」を選択し、以下のように設定します。
デバイス名 FS
MountPoint ラベル
# /dev/sda1 xfs /boot
/boot
# /dev/sda2 xfs /
/
# /dev/sdb1 swap
swap
「Begin Install」を選択します。
# Begin Install
「ROOT PASSWORD」を選択し、パスワードを設定します。
# ROOT PASSWORD: ********
「Reboot」ボタンが表示されるのを待ち、「Reboot」を選択します。
# Reboot
再起動処理中に Eject されたインストーラをDVDドライブから取り外します。
# Eject DVD
再起動完了後、コンソールにてログインします。
Oracle Linux Server 7.3
Kernel 4.1.12-61.1.18.el7uek.x86_64 on an x86_64
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 8 2680
2690 a,s
2700 a,s
2710
2720
2730 ○
2740
2750 a,s
2760
2770
2780
2790
2800
2810
2820
2830
2840
2850
2860
2870
2880
2890
2900 ○
2910
2920
2930
2940 ○
2950
2960
a
2970
2980
s
2990
3000
3010
3020 ○
3030
3040
3050
3060 ○
3070
#localhost
root
login: root
#Password:
****************
[root@localhost ~] #
MAC アドレスを確認します。
ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
link/ether 00:0c:29:0b:ab:ce brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
link/ether 00:0c:29:0b:ab:b0 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
link/ether 00:0c:29:0b:ab:ba brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
link/ether 00:0c:29:0b:ab:c4 brd ff:ff:ff:ff:ff:ff
qlen 1000
qlen 1000
qlen 1000
qlen 1000
MAC アドレスをもとに、 LAN ケーブルの結線(組み合わせ)を設計通りに修正します。
※　この後の手順で、NIC デバイス名の方を入れ替えても構いません。
IP アドレスを一時的に設定します。
ip addr add 10.110.88.57/26 dev eth0
ip addr add 10.110.88.58/26 dev eth0
※　デバイス名が意図するものとずれている場合、デバイス名は適宜変更する必要があります。
必要に応じて、デフォルトゲートウェイを一時的に設定します。
ip route add default via 10.110.88.1
root にて、ssh でログインします。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 9 3080
a
3090
3100
3110
a
3120
3130
a
3140
3150
3160
s
3170
3180
3190
s
3200
3210
s
3220
3230
3240 ○
3250
3260 a,s
3270
3280
3290
3300
3310
3320
3330
3340
3350
3360
3370 a,s
3380
3390
3400
3410
3420 a,s
3430
3440
3450
3460
3470
ssh [email protected]
The authenticity of host '10.110.88.57 (10.110.88.57)' can't be established.
ECDSA key fingerprint is 95:bc:49:71:b2:a3:dd:ab:63:ad:35:e4:fe:4d:fc:82.
#Are
yesyou sure you want to continue connecting (yes/no)? yes
#Warning:
yes
Permanently added '10.110.88.57' (ECDSA) to the list of known hosts.
#[email protected]'s
********
password: ********
Last login: Sat Oct 29 18:33:24 2016
ssh [email protected]
The authenticity of host '10.110.88.58 (10.110.88.58)' can't be established.
ECDSA key fingerprint is 8f:f6:81:0f:44:e1:83:d5:0a:9d:3f:90:7c:3e:93:73.
#Are
yesyou sure you want to continue connecting (yes/no)? yes
#Warning:
yes
Permanently added '10.110.88.58' (ECDSA) to the list of known hosts.
#[email protected]'s
********
password: ********
Last login: Sat Oct 29 18:33:24 2016
ストレージの情報を確認します。
fdisk -l | grep /dev/ | sort
/dev/sda1
*
2048
1026047
512000 83 Linux
/dev/sda2
1026048
33554431
16264192 83 Linux
/dev/sdb1
2048
2097151
1047552 82 Linux swap / Solaris
Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors
Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors
Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sdd: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sde: 107.4 GB, 107374182400 bytes, 209715200 sectors
Disk /dev/sdf: 107.4 GB, 107374182400 bytes, 209715200 sectors
blkid
/dev/sda1: LABEL="/boot" UUID="0d524bba-e554-4e4c-8594-4293131808af" TYPE="xfs"
/dev/sda2: LABEL="/" UUID="7fe16a0a-7b75-4e49-bef7-3b01194313fc" TYPE="xfs"
/dev/sdb1: LABEL="swap" UUID="9d2b758f-05a3-4aa2-b19a-23684a062a65" TYPE="swap"
cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sat Oct 29 18:33:24 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 10 3480
3490
3500
3510
3520
3530
3540
3550
3560
3570
3580
3590
3600
3610
3620
3630
3640
3650
3660
3670
3680
3690
3700
3710
3720
3730
3740
3750
3760
3770
3780
3790
3800
3810
3820
3830
3840
3850
3860
3870
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=7fe16a0a-7b75-4e49-bef7-3b01194313fc /
xfs
defaults
UUID=0d524bba-e554-4e4c-8594-4293131808af /boot
xfs
defaults
UUID=9d2b758f-05a3-4aa2-b19a-23684a062a65 swap
swap
defaults
○
キーボード、ロケールの情報を確認します。
a,s
cat /etc/vconsole.conf
KEYMAP="jp-OADG109A"
FONT="latarcyrheb-sun16"
a,s
cat /etc/locale.conf
LANG="en_US.UTF-8"
a,s
localectl status
System Locale:
VC Keymap:
X11 Layout:
X11 Variant:
○
0 0
0 0
0 0
LANG=en_US.UTF-8
jp-OADG109A
jp
OADG109A
タイムゾーンの情報を確認します。
a,s
cat /etc/adjtime
0.0 0 0.0
0
UTC
a,s
hwclock --debug
hwclock from util-linux 2.23.2
Using /dev interface to clock.
Last drift adjustment done at 0 seconds after 1969
Last calibration done at 0 seconds after 1969
Hardware clock is on UTC time
Assuming hardware clock is kept in UTC time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2016/10/29 09:33:24
Hw clock time : 2016/10/29 09:33:24 = 1478911709 seconds since 1969
Sat 29 Oct 2016 18:33:24 AM JST -0.520512 seconds
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 11 3880
3890
3900
3910
3920
3930
3940
3950
3960
3970
3980
3990
4000
4010
4020
4030
4040
4050
4060
4070
4080
4090
4100
4110
4120
4130
4140
4150
4160
4170
4180
4190
4200
4210
4220
4230
4240
4250
4260
4270
a,s
ls -l /etc/localtime
lrwxrwxrwx 1 root root 32 Oct 29 18:33 /etc/localtime -> ../usr/share/zoneinfo/Asia/Tokyo
a,s
timedatectl status
Local time: Sat 2016-10-29 18:33:24 JST
Universal time: Sat 2016-10-29 09:33:24 UTC
RTC time: Sat 2016-10-29 09:33:24
Time zone: Asia/Tokyo (JST, +0900)
NTP enabled: n/a
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
○
a,s
SELinux を無効化します。
sed -i -e 's/^SELINUX=.*$/SELINUX=disabled/' /etc/sysconfig/selinux
※　カーネルパラメータで無効化していますが、運用上紛らわしいので設定ファイルも変更します。
○
管理者用一般ユーザを作成します。
a,s
sed -i -e 's/^CREATE_MAIL_SPOOL=.*$/CREATE_MAIL_SPOOL=no/' /etc/default/useradd
a,s
a,s
a,s
groupadd -g 1000 admin
useradd -g admin -G wheel -u 1000 admin
echo 'password' | passwd --stdin admin
Changing password for user admin.
passwd: all authentication tokens updated successfully.
a,s
id admin
uid=1000(admin) gid=1000(admin) groups=1000(admin),10(wheel)
○
a,s
○
a
wheel グループのユーザがパスワードなしで sudo コマンドを使えるように設定します。
echo '%wheel ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/wheel
管理者用一般ユーザにて、ssh でログインします。
ssh [email protected]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 12 4280
4290
4300
4310
4320
4330
4340
4350
4360
4370
4380
4390
4400
4410
4420
4430
4440
4450
4460
4470
4480
4490
4500
4510
4520
4530
4540
4550
4560
4570
4580
4590
4600
4610
4620
4630
4640
4650
4660
4670
a
#[email protected]'s
********
password: ********
s
s
ssh [email protected]
#[email protected]'s
********
password: ********
○
a,s
a,s
○
a,s
a,s
○
wheel グループのユーザのみが su コマンドを使えるように設定します。
sudo sed -i -e '/^#auth.*required.*pam_wheel.so use_uid$/ s/#//' /etc/pam.d/su
echo "SU_WHEEL_ONLY yes" | sudo tee -a /etc/login.defs
root アカウントでのパスワード認証による ssh 接続を禁止します。
sudo sed -i -e 's/^#PermitRootLogin .*$/PermitRootLogin without-password/' /etc/ssh/sshd_config
sudo systemctl restart sshd
参照・監視用一般ユーザを作成します。
a,s
a,s
a,s
sudo groupadd -g 1001 monitor
sudo useradd -g monitor -u 1001 monitor
echo 'password' | sudo passwd --stdin monitor
Changing password for user monitor.
passwd: all authentication tokens updated successfully.
a,s
id monitor
uid=1001(monitor) gid=1001(monitor) groups=1001(monitor)
○
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
NIC のデバイス名をバス情報に基づいて固定します。
sudo cp /dev/null /etc/udev/rules.d/70-persistent-net.rules
NUM=0
while :
do
ip addr show eth$NUM > /dev/null 2>&1 || break
BUS=$(ethtool -i eth$NUM | grep bus-info | awk '{print $2}')
cat << EOF | sudo tee -a /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="$BUS", ATTR{type}=="1", NAME="eth$NUM"
EOF
NUM=$((NUM+1))
done
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:04:00.0", ATTR{type}=="1", NAME="eth0"
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 13 4680
4690
4700
4710
4720
4730
4740
4750
4760
4770
4780
4790
4800
4810
4820
4830
4840
4850
4860
4870
4880
4890
4900
4910
4920
4930
4940
4950
4960
4970
4980
4990
5000
5010
5020
5030
5040
5050
5060
5070
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:0b:00.0", ATTR{type}=="1", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:13:00.0", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:1b:00.0", ATTR{type}=="1", NAME="eth3"
※　このファイルを編集して、NIC デバイス名を入れ替えても構いません。
※　KERNELS=="<バス情報>" を ATTR{address}=="<MAC アドレス>" に入れ替えても構いません。
○
a,s
a,s
a,s
a,s
OS 起動時のカーネルパラメータを変更します。
sudo sed -i -e '/^GRUB_CMDLINE_LINUX=/ s/ *biosdevname=[^ "]*//' /etc/default/grub
sudo sed -i -e '/^GRUB_CMDLINE_LINUX=/ s/ *net\.ifnames=[^ "]*//' /etc/default/grub
sudo sed -i -e '/^GRUB_CMDLINE_LINUX=/ s/rhgb quiet/ipv6.disable=1/' /etc/default/grub
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.1.12-61.1.18.el7uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-61.1.18.el7uek.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-514.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-514.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-4305b3f1881f49358c6f848704b93e32
Found initrd image: /boot/initramfs-0-rescue-4305b3f1881f49358c6f848704b93e32.img
done
※　「net.ifnames=0」「biosdevname=0」があると、前項の udev 設定が機能しません。
※　IPv6 を無効化しています。
○
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
NIC を設定します。
BOND0_UUID=$(uuidgen)
BOND0_BONDING_OPTS="resend_igmp=1 updelay=0 use_carrier=1 miimon=100 downdelay=0 xmit_hash_policy=0"
BOND0_BONDING_OPTS="$BOND0_BONDING_OPTS primary_reselect=0 fail_over_mac=0 arp_validate=0"
BOND0_BONDING_OPTS="$BOND0_BONDING_OPTS mode=active-backup primary=eth0"
※　LAGの場合は「mode=802.3ad」
BOND0_BONDING_OPTS="$BOND0_BONDING_OPTS lacp_rate=0 arp_interval=0 ad_select=0"
BOND1_UUID=$(uuidgen)
BOND1_BONDING_OPTS="resend_igmp=1 updelay=0 use_carrier=1 miimon=100 downdelay=0 xmit_hash_policy=0"
BOND1_BONDING_OPTS="$BOND1_BONDING_OPTS primary_reselect=0 fail_over_mac=0 arp_validate=0"
BOND1_BONDING_OPTS="$BOND1_BONDING_OPTS mode=active-backup primary=eth1"
※　LAGの場合は「mode=802.3ad」
BOND1_BONDING_OPTS="$BOND1_BONDING_OPTS lacp_rate=0 arp_interval=0 ad_select=0"
cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-bond0
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 14 5080
5090
5100
5110
5120
5130
5140
5150
5160
5170
5180
5190
5200
5210
5220
5230
5240
5250
5260
5270
5280
5290
5300
5310
5320
5330
5340
5350
5360
5370
5380
5390
5400
5410
5420
5430
5440
5450
5460
5470
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
DEVICE=bond0
NAME=bond0
TYPE=Bond
UUID=$BOND0_UUID
BONDING_OPTS="$BOND0_BONDING_OPTS"
BONDING_MASTER=yes
ONBOOT=yes
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
EOF
cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1
NAME=bond1
TYPE=Bond
UUID=$BOND1_UUID
BONDING_OPTS="$BOND1_BONDING_OPTS"
BONDING_MASTER=yes
ONBOOT=yes
BOOTPROTO=none
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
EOF
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 15 5480
5490
5500
5510
5520
5530
5540
5550
5560
5570
5580
5590
5600
5610
5620
5630
5640
5650
5660
5670
5680
5690
5700
5710
5720
5730
5740
5750
5760
5770
5780
5790
5800
5810
5820
5830
5840
5850
5860
5870
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
NAME=eth0
TYPE=Ethernet
UUID=$(uuidgen)
MASTER=$BOND0_UUID
SLAVE=yes
ONBOOT=yes
MTU=9000
EOF
cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth2
DEVICE=eth2
NAME=eth2
TYPE=Ethernet
UUID=$(uuidgen)
MASTER=$BOND0_UUID
SLAVE=yes
ONBOOT=yes
MTU=9000
EOF
cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
NAME=eth1
TYPE=Ethernet
UUID=$(uuidgen)
MASTER=$BOND1_UUID
SLAVE=yes
ONBOOT=yes
MTU=9000
EOF
cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth3
DEVICE=eth3
NAME=eth3
TYPE=Ethernet
UUID=$(uuidgen)
MASTER=$BOND1_UUID
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 16 5880
5890
5900
5910
5920
5930
5940
5950
5960
5970
5980
5990
6000
6010
6020
6030
6040
6050
6060
6070
6080
6090
6100
6110
6120
6130
6140
6150
6160
6170
6180
6190
6200
6210
6220
6230
6240
6250
6260
6270
a,s
a,s
a,s
a,s
a,s
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
s
○
SLAVE=yes
ONBOOT=yes
MTU=9000
EOF
# for Active
cat << EOF | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-bond0
IPADDR=10.110.88.57
PREFIX=26
GATEWAY=10.110.88.1
DNS1=10.0.80.11
DNS2=10.0.80.12
DOMAIN=example.com
MTU=9000
EOF
cat << EOF | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-bond1
IPADDR=192.168.1.2
PREFIX=24
MTU=9000
EOF
# for Stand-by
cat << EOF | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-bond0
IPADDR=10.110.88.58
PREFIX=26
GATEWAY=10.110.88.1
DNS1=10.0.80.11
DNS2=10.0.80.12
DOMAIN=example.com
MTU=9000
EOF
cat << EOF | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-bond1
IPADDR=192.168.1.3
PREFIX=24
MTU=9000
EOF
NIC オフロード機能を無効化します。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 17 6280
6290
6300
6310
6320
6330
6340
6350
6360
6370
6380
6390
6400
6410
6420
6430
6440
6450
6460
6470
6480
6490
6500
6510
6520
6530
6540
6550
6560
6570
6580
6590
6600
6610
6620
6630
6640
6650
6660
6670
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
cat << 'EOF' | sudo tee /etc/NetworkManager/dispatcher.d/00-ethertool
#!/bin/sh
if [ "$2" == "up" ]; then
if [ "${1:0:3}" == "eth" ]; then
ethtool -K $1 \
rx off \
tx off \
sg off \
tso off \
ufo off \
gso off \
gro off \
lro off \
rxvlan off \
txvlan off \
ntuple off \
rxhash off \
highdma off \
rx-vlan-filter off \
tx-gso-robust off \
tx-fcoe-segmentation off \
fcoe-mtu off \
tx-nocache-copy off \
loopback off \
rx-fcs off \
rx-all off
#ethtool -K $1 vlan-challenged off tx-lockless off netns-local off
ethtool -G $1 rx 4096 tx 4096 rx-jumbo 2048
fi
fi
EOF
sudo chmod 755 /etc/NetworkManager/dispatcher.d/00-ethertool
※　「VMXNET 3」でのみ動作確認しました。
※　NIC のリンク速度、duplex モードを設定したい場合はこのスクリプトに組み込みます。
○
a,s
hosts を設定します。
cat << 'EOF' | sudo tee /etc/hosts
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 18 6680
6690
6700
6710
6720
6730
6740
6750
6760
6770
6780
6790
6800
6810
6820
6830
6840
6850
6860
6870
6880
6890
6900
6910
6920
6930
6940
6950
6960
6970
6980
6990
7000
7010
7020
7030
7040
7050
7060
7070
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
○
127.0.0.1
::1
10.110.88.57
10.110.88.58
10.110.88.59
192.168.1.2
192.168.1.3
EOF
localhost localhost.localdomain localhost4 localhost4.localdomain4
localhost localhost.localdomain localhost6 localhost6.localdomain6
iscsitgt01a.example.com iscsitgt01a
iscsitgt01s.example.com iscsitgt01s
iscsitgt01.example.com iscsitgt01
iscsitgt01a-ic.example.com iscsitgt01a-ic
iscsitgt01s-ic.example.com iscsitgt01s-ic
hostname を設定します。
a
sudo hostnamectl set-hostname iscsitgt01a.example.com
s
sudo hostnamectl set-hostname iscsitgt01s.example.com
○
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
yum リポジトリを設定します。
cat << 'EOF' | sudo tee /etc/yum.repos.d/media.repo
[media]
name=media
baseurl=file:///mnt
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0
[media-mysql]
name=media-mysql
baseurl=file:///mnt/addons/Mysql
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0
[media-ha]
name=media-ha
baseurl=file:///mnt/addons/HighAvailability
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0
[media-rs]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 19 7080
7090
7100
7110
7120
7130
7140
7150
7160
7170
7180
7190
7200
7210
7220
7230
7240
7250
7260
7270
7280
7290
7300
7310
7320
7330
7340
7350
7360
7370
7380
7390
7400
7410
7420
7430
7440
7450
7460
7470
a,s
a,s
a,s
a,s
a,s
a,s
name=media-rs
baseurl=file:///mnt/addons/ResilientStorage
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0
EOF
※　インストール・メディアを利用可能にします。
a,s
sudo sed -i -e 's/^/#/' /etc/yum.repos.d/public-yum-ol7.repo
※　インターネット接続していないとエラーとなるリポジトリを無効化します。
○
a,s
○
a,s
○
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
以下のインストーラを DVD ドライブにセットします。
# V834394-01.iso (Oracle Linux 7.3)
インストーラをマウントします。
sudo mount /dev/cdrom /mnt
mount: /dev/sr0 is write-protected, mounting read-only
どのような環境でも共通して導入しておいた方がよいと思われる標準パッケージをインストールします。
sudo yum -y --disablerepo=\* --enablerepo=media,media-mysql install \
@development \
@base \
OpenIPMI \
aide \
dos2unix \
dropwatch \
dstat \
expect \
filebench \
freeipmi-bmc-watchdog \
freeipmi-ipmidetectd \
ftp \
haproxy \
hdparm \
iotop \
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 20 7480
7490
7500
7510
7520
7530
7540
7550
7560
7570
7580
7590
7600
7610
7620
7630
7640
7650
7660
7670
7680
7690
7700
7710
7720
7730
7740
7750
7760
7770
7780
7790
7800
7810
7820
7830
7840
7850
7860
7870
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
○
a,s
○
a,s
○
a,s
a,s
ipmitool \
iptables-services \
iptraf-ng \
iptstate \
ipvsadm \
iscsi-initiator-utils \
keepalived \
kernel-uek-devel \
latrace \
lftp \
logwatch \
lrzsz \
ltrace \
nmap \
openssl-devel \
oprofile \
pax \
perf \
prelink \
screen \
sg3_utils \
snapper \
telnet \
tmpwatch \
trace-cmd \
tree \
x86info
インストーラをアンマウントします。
sudo umount /mnt
インストーラをDVDドライブから取り外します。
# Eject DVD
NTP を設定します。
cat << 'EOF' | sudo tee /etc/chrony.conf
server 10.0.77.54 iburst
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 21 7880
7890
7900
7910
7920
7930
7940
7950
7960
7970
7980
7990
8000
8010
8020
8030
8040
8050
8060
8070
8080
8090
8100
8110
8120
8130
8140
8150
8160
8170
8180
8190
8200
8210
8220
8230
8240
8250
8260
8270
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
# server ***.***.***.***
# server ***.***.***.***
# server ***.***.***.***
EOF
# sudo sed -i -e '/^#/d'
iburst
iburst
iburst
/etc/chrony.conf
cat << 'EOF' | sudo tee -a /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# Ignore stratum in source selection.
stratumweight 0
# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift
# Enable kernel RTC synchronization.
rtcsync
# In first three updates step the system clock instead of slew
# if the adjustment is larger than 10 seconds.
makestep 10 3
# Allow NTP client access from local network.
#allow 192.168/16
# Listen for commands only on localhost.
bindcmdaddress 127.0.0.1
#bindcmdaddress ::1
# Serve time even if not synchronized to any NTP server.
#local stratum 10
keyfile /etc/chrony.keys
# Specify the key used as password for chronyc.
commandkey 1
# Generate command key if missing.
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 22 8280
8290
8300
8310
8320
8330
8340
8350
8360
8370
8380
8390
8400
8410
8420
8430
8440
8450
8460
8470
8480
8490
8500
8510
8520
8530
8540
8550
8560
8570
8580
8590
8600
8610
8620
8630
8640
8650
8660
8670
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
○
a,s
a,s
a,s
a,s
generatecommandkey
# Disable logging of client accesses.
noclientlog
# Send a message to syslog if a clock adjustment is larger than 0.5 seconds.
logchange 0.5
logdir /var/log/chrony
#log measurements statistics tracking
EOF
cat << 'EOF' | sudo tee /etc/sysconfig/chronyd
OPTIONS="-4"
EOF
不要なサービスを無効化します。
sudo
sudo
sudo
sudo
systemctl
systemctl
systemctl
systemctl
disable
disable
disable
disable
dmraid-activation.service
firewalld.service
mdmonitor.service
postfix.service
※　仮想環境の場合は、「smartd.service」も無効化します。RAID コントローラが対応していない場合も無効化します。
○
a,s
○
必要なサービスを有効化します。
sudo systemctl enable psacct.service
iSCSI イニシエータ関連サービスの自動起動を無効化します。
a,s
sudo systemctl disable iscsi.service
Removed symlink /etc/systemd/system/sysinit.target.wants/iscsi.service.
a,s
sudo systemctl disable iscsid.socket
Removed symlink /etc/systemd/system/sockets.target.wants/iscsid.socket.
a,s
sudo systemctl disable iscsiuio.socket
Removed symlink /etc/systemd/system/sockets.target.wants/iscsiuio.socket.
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 23 8680
8690
8700
8710
8720
8730
8740
8750
8760
8770
8780
8790
8800
8810
8820
8830
8840
8850
8860
8870
8880
8890
8900
8910
8920
8930
8940
8950
8960
8970
8980
8990
9000
9010
9020
9030
9040
9050
9060
9070
○
IPv6 無効化に伴う不具合を解消するための設定変更を行います。
a,s
a,s
sudo sed -i -e 's/^#AddressFamily .*$/AddressFamily inet/' /etc/ssh/sshd_config
sudo sed -i -e 's/^inet_interfaces .*$/inet_interfaces = 127.0.0.1/' /etc/postfix/main.cf
a,s
sudo sed -i -e 's/^udp6/#udp6/' -e 's/^tcp6/#tcp6/' /etc/netconfig
○
a,s
○
再起動します。
sudo reboot
管理者用一般ユーザにて、ssh でログインします。
a
a
ssh [email protected]
#[email protected]'s
********
password: ********
s
s
ssh [email protected]
#[email protected]'s
********
password: ********
○
a,s
カーネル起動パラメータを確認します。
cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.1.12-61.1.18.el7uek.x86_64 root=UUID=657f59aa-f627-4096-9970-9238b234ef00 ro crashkernel=auto selinux=0 ipv6.disable=1
※　「crashkernel」の値は、搭載メモリサイズに応じて自動的に固定値へ変更される場合があります。
○
kdump の設定を確認します。
a,s
systemctl is-enabled kdump.service
enabled
a,s
sudo kdumpctl status
Kdump is operational
○
a,s
SELinux の設定を確認します。
grep -v ^# /etc/sysconfig/selinux
SELINUX=disabled
SELINUXTYPE=targeted
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 24 9080 a,s
9090
9100
9110 ○
9120
9130 a,s
9140
9150
9160
9170
9180
9190
9200
9210
9220
9230
9240
9250
9260
9270
9280
9290
9300
9310
9320
9330
9340 a,s
9350
9360
9370
9380
9390
9400
9410
9420
9430
9440
9450
9460
9470
getenforce
Disabled
ネットワーク設定を確認します。
ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master
link/ether 00:0c:29:0b:ab:c4 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master
link/ether 00:0c:29:0b:ab:ce brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master
link/ether 00:0c:29:0b:ab:c4 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master
link/ether 00:0c:29:0b:ab:ce brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue
link/ether 00:0c:29:0b:ab:c4 brd ff:ff:ff:ff:ff:ff
inet 10.110.88.57/26 brd 10.110.88.63 scope global bond0
7: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue
link/ether 00:0c:29:0b:ab:ce brd ff:ff:ff:ff:ff:ff
inet 192.168.1.2/24 brd 192.168.0.255 scope global bond1
bond0 state UP qlen 1000
bond1 state UP qlen 1000
bond0 state UP qlen 1000
bond1 state UP qlen 1000
state UNKNOWN
state UNKNOWN
※　IPv6 のリンクローカルアドレスが存在しないことも確認します。
cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 25 9480
9490
9500
9510
9520
9530
9540
9550
9560
9570
9580
9590
9600
9610 a,s
9620
9630
9640
9650
9660
9670
9680
9690
9700
9710
9720
9730
9740
9750
9760
9770
9780
9790
9800
9810
9820
9830
9840
9850
9860
9870
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:0b:ab:b0
Slave queue ID: 0
Slave Interface: eth2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:0b:ab:c4
Slave queue ID: 0
cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth1 (primary_reselect always)
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:0b:ab:ba
Slave queue ID: 0
Slave Interface: eth3
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:0b:ab:ce
Slave queue ID: 0
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 26 9880
9890
9900
9910
9920
9930
9940
9950
9960
9970
9980
9990
10000
10010
10020
10030
10040
10050
10060
10070
10080
10090
10100
10110
10120
10130
10140
10150
10160
10170
10180
10190
10200
10210
10220
10230
10240
10250
10260
10270
※　ボンディング設定時は、このコマンドで個々の NIC の MAC アドレスを確認できます。
※　LAG の場合、以下のように表示されます。
cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 13
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00
Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:8b:ff:e8
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 0
port key: 13
port priority: 255
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 27 10280
10290
10300
10310
10320
10330
10340
10350
10360
10370
10380
10390
10400
10410
10420
10430
10440
10450
10460
10470
10480
10490
10500
10510
10520
10530
10540
10550
10560
10570
10580
10590
10600
10610
10620
10630
10640
10650
10660
10670
port number: 1
port state: 205
details partner lacp pdu:
system priority: 65535
oper key: 1
port priority: 255
port number: 1
port state: 3
Slave Interface: eth3
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:8b:ff:fc
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
system priority: 0
port key: 13
port priority: 255
port number: 2
port state: 197
details partner lacp pdu:
system priority: 65535
oper key: 1
port priority: 255
port number: 1
port state: 3
a
ip route show
default via 10.110.88.1 dev bond0 proto static metric 300
10.110.88.0/26 dev bond0 proto kernel scope link src 10.110.88.57 metric 300
192.168.1.0/24 dev bond1 proto kernel scope link src 192.168.0.2 metric 300
s
ip route show
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 28 10680
10690
10700
10710
10720 a,s
10730
10740
10750
10760
10770
10780 ○
10790
10800
a
10810
10820
10830
10840
10850
10860
10870
10880
10890
10900
10910
10920
s
10930
10940
10950
10960
10970
10980
10990
11000
11010
11020
11030
11040 ○
11050
11060 a,s
11070
default via 10.110.88.1 dev bond0 proto static metric 300
10.110.88.0/26 dev bond0 proto kernel scope link src 10.110.88.58 metric 300
192.168.1.0/24 dev bond1 proto kernel scope link src 192.168.0.3 metric 300
cat /etc/resolv.conf
# Generated by NetworkManager
search example.com
nameserver 10.0.80.11
nameserver 10.0.80.12
hostname 設定を確認します。
hostnamectl status
Static hostname:
Icon name:
Chassis:
Machine ID:
Boot ID:
Virtualization:
Operating System:
CPE OS Name:
Kernel:
Architecture:
iscsitgt01a.example.com
computer-vm
vm
d7806eba789047baa165a57149c83843
b5b36a3403dd403aad4656d2f7f9e7aa
vmware
Oracle Linux Server 7.3
cpe:/o:oracle:linux:7:2:server
Linux 4.1.12-61.1.18.el7uek.x86_64
x86-64
hostnamectl status
Static hostname:
Icon name:
Chassis:
Machine ID:
Boot ID:
Virtualization:
Operating System:
CPE OS Name:
Kernel:
Architecture:
iscsitgt01s.example.com
computer-vm
vm
b325c1c5d682439a91a65f7cfc317b20
a419d4d1ef00452f93da10a227365aca
vmware
Oracle Linux Server 7.3
cpe:/o:oracle:linux:7:2:server
Linux 4.1.12-61.1.18.el7uek.x86_64
x86-64
NIC のオフロード設定を確認します。
ethtool -k eth0
Features for eth0:
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 29 11080
11090
11100
11110
11120
11130
11140
11150
11160
11170
11180
11190
11200
11210
11220
11230
11240
11250
11260
11270
11280
11290
11300
11310
11320
11330
11340
11350
11360
11370
11380
11390
11400
11410
11420
11430
11440
11450
11460
11470
rx-checksumming: off
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off [fixed]
receive-hashing: off
highdma: off
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 30 11480
11490
11500
11510
11520
11530
11540
11550
11560
11570
11580
11590
11600
11610
11620
11630
11640
11650
11660
11670
11680
11690
11700
11710
11720
11730
11740
11750
11760
11770
11780
11790
11800
11810
11820
11830
11840
11850
11860
11870
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
hw-switch-offload: off [fixed]
a,s
a,s
a,s
ethtool -k eth1
ethtool -k eth2
ethtool -k eth3
a,s
ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:
4096
RX Mini:
0
RX Jumbo:
2048
TX:
4096
Current hardware settings:
RX:
4032
RX Mini:
0
RX Jumbo:
2048
TX:
4096
※　搭載メモリサイズに応じて結果が異なります。
a,s
a,s
a,s
○
a,s
ethtool -g eth1
ethtool -g eth2
ethtool -g eth3
NTP の状態を確認します。
systemctl status chronyd.service -l
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2016-10-29 18:33:24 JST; 42min left
Process: 601 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 576 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 583 (chronyd)
CGroup: /system.slice/chronyd.service
┗583 /usr/sbin/chronyd -4
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 31 11880
11890
11900
11910
11920
11930
11940 a,s
11950
11960
11970
11980
11990
12000 a,s
12010
12020
12030
12040
12050
12060
12070
12080
12090
12100 ○
12110
12120 a,s
12130
12140
12150
12160
12170
12180
12190
12200
12210
12220
12230
12240
12250
12260
12270
Oct
Oct
Oct
Oct
Oct
29
29
29
29
29
18:33:24
18:33:24
18:33:24
18:33:24
18:33:24
iscsitgt01a.example.com
iscsitgt01a.example.com
iscsitgt01a.example.com
iscsitgt01a.example.com
iscsitgt01a.example.com
systemd[1]: Starting NTP client/server...
chronyd[584]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
chronyd[584]: Generated key 1
systemd[1]: Started NTP client/server.
chronyd[584]: Selected source 10.0.77.54
chronyc sources
210 Number of sources = 1
MS Name/IP address
Stratum Poll Reach LastRx Last sample
===============================================================================
^* 10.0.77.54
1 10 377 217 -177us[ -161us] +/- 4360us
timedatectl status
Local time: Sat 2016-10-29 18:33:24 JST
Universal time: Sat 2016-10-29 09:33:24 UTC
RTC time: Sat 2016-10-29 09:33:24
Time zone: Asia/Tokyo (JST, +0900)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a
自動起動するサービスを確認します。
systemctl list-unit-files | grep enabled |
NetworkManager-dispatcher.service
NetworkManager.service
abrt-ccpp.service
abrt-oops.service
abrt-vmcore.service
abrt-xorg.service
abrtd.service
atd.service
auditd.service
[email protected]
chronyd.service
crond.service
dbus-org.freedesktop.NetworkManager.service
dbus-org.freedesktop.nm-dispatcher.service
default.target
LANG=C sort
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 32 12280
12290
12300
12310
12320
12330
12340
12350
12360
12370
12380
12390
12400
12410
12420
12430
12440
12450
12460
12470
12480
12490
12500
12510
12520
12530
12540
12550 ○
12560
12570 a,s
12580
12590
dm-event.socket
[email protected]
irqbalance.service
kdump.service
libstoragemgmt.service
lvm2-lvmetad.socket
lvm2-lvmpolld.socket
lvm2-monitor.service
microcode.service
psacct.service
multi-user.target
remote-fs.target
rngd.service
rpcbind.socket
rsyslog.service
runlevel2.target
runlevel3.target
runlevel4.target
smartd.service
sshd.service
sysstat.service
systemd-readahead-collect.service
systemd-readahead-drop.service
systemd-readahead-replay.service
tuned.service
vmtoolsd.service
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
enabled
※　LVM を利用しない場合、不要
※　LVM を利用しない場合、不要
※　LVM を利用しない場合、不要
※　nfs, iSCSI イニシエータを利用しない場合、不要
※　nfs を利用しない場合、不要
※　仮想環境、RAID コントローラ未対応の場合、不要
※　ESXi で動かす場合のみ必要
設定ファイルをバックアップします。
sudo cp -a /etc{,~}
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 33 12600
12610
12620
12630
12640
12650
12660
12670
12680
12690
12700
12710
12720
12730
12740
12750
12760
12770
12780
12790
12800
12810
12820
12830
12840
12850
12860
12870
12880
12890
12900
12910
12920
12930
12940
12950
12960
12970
12980
12990
○
【iSCSI Target クラスタのインストールと初期設定】
○
以下のインストーラを DVD ドライブにセットします。
a,s
○
a,s
○
a,s
a,s
a,s
a,s
a,s
a,s
○
o
○
# V834394-01.iso (Oracle Linux 7.3)
インストーラをマウントします。
sudo mount /dev/cdrom /mnt
mount: /dev/sr0 is write-protected, mounting read-only
当該 OS で必要となる標準パッケージをインストールします。
sudo yum -y --disablerepo=\* --enablerepo=media,media-mysql,media-ha install \
fence-agents-ipmilan \
omping \
pcs \
rubygem-abrt \
targetcli
インターネットと接続可能な端末で以下のコマンドを実行する等して、必要なパッケージを収集します。
curl -O http://elrepo.org/linux/elrepo/el7/x86_64/RPMS/drbd84-utils-8.9.6-1.el7.elrepo.x86_64.rpm
収集したパッケージをホームディレクトリにコピーし、確認します。
a,s
scp xxxx@yyy:drbd84-utils-8.9.6-1.el7.elrepo.x86_64.rpm .
a,s
ls -l *.rpm
-rw-rw-r-- 1 admin admin
a,s
○
a,s
○
410308 Oct 29 18:33 drbd84-utils-8.9.6-1.el7.elrepo.x86_64.rpm
file *.rpm
drbd84-utils-8.9.6-1.el7.elrepo.x86_64.rpm:
RPM v3.0 bin i386/x86_64 drbd84-utils-8.9.6-1.el7.elrepo
drbd 管理ツールをインストールします。Oracle 社サポート外のパッケージです。
sudo yum -y --disablerepo=\* --enablerepo=media localinstall drbd84-utils-*.rpm
インストーラをアンマウントします。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 34 13000
13010
13020
13030
13040
13050
13060
13070
13080
13090
13100
13110
13120
13130
13140
13150
13160
13170
13180
13190
13200
13210
13220
13230
13240
13250
13260
13270
13280
13290
13300
13310
13320
13330
13340
13350
13360
13370
13380
13390
a,s
○
a,s
○
sudo umount /mnt
インストーラをDVDドライブから外します。
# Eject DVD
追加インストールしたパッケージの設定をバックアップします。
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
○
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
sudo
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
cp
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
-a
/etc{,~}/bash_completion.d/drbdadm
/etc{,~}/corosync
/etc{,~}/dbus-1/system.d/corosync-signals.conf
/etc{,~}/drbd.conf
/etc{,~}/drbd.d
/etc{,~}/ha.d
/etc{,~}/libreport/events.d/ruby_event.conf
/etc{,~}/logrotate.d/corosync
/etc{,~}/logrotate.d/pacemaker
/etc{,~}/logrotate.d/pcsd
/etc{,~}/pam.d/pcsd
/etc{,~}/sysconfig/corosync
/etc{,~}/sysconfig/corosync-notifyd
/etc{,~}/sysconfig/crm_mon
/etc{,~}/sysconfig/ipmievd
/etc{,~}/sysconfig/pacemaker
/etc{,~}/sysconfig/pcsd
/etc{,~}/target
/etc{,~}/xen
/etc/passwd
/etc~/passwd_$(date +%Y%m%d_%H%M%S)
/etc/passwd- /etc~/passwd-_$(date +%Y%m%d_%H%M%S)
/etc/shadow
/etc~/shadow_$(date +%Y%m%d_%H%M%S)
/etc/shadow- /etc~/shadow-_$(date +%Y%m%d_%H%M%S)
/etc/group
/etc~/group_$(date +%Y%m%d_%H%M%S)
/etc/group/etc~/group-_$(date +%Y%m%d_%H%M%S)
/etc/gshadow /etc~/gshadow_$(date +%Y%m%d_%H%M%S)
/etc/gshadow- /etc~/gshadow-_$(date +%Y%m%d_%H%M%S)
a,s
a,s
cat << 'EOF' | sudo tee /etc/sysctl.d/tgt.conf
net.core.netdev_max_backlog = 250000
カーネルパラメータを設定します。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 35 13400
13410
13420
13430
13440
13450
13460
13470
13480
13490
13500
13510
13520
13530
13540
13550
13560
13570
13580
13590
13600
13610
13620
13630
13640
13650
13660
13670
13680
13690
13700
13710
13720
13730
13740
13750
13760
13770
13780
13790
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
○
a,s
○
net.core.optmem_max = 16777216
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_mem = 39363 209944 314904
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_wmem = 8192 65536 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
EOF
再起動します。
sudo reboot
管理者用一般ユーザにて、ssh でログインします。
a
a
ssh [email protected]
#[email protected]'s
********
password: ********
s
s
ssh [email protected]
#[email protected]'s
********
password: ********
○
a,s
カーネルパラメータを確認します。
sysctl -a 2> /dev/null | egrep 'net\.core\..*mem|net\.core\.netdev_m|net\.ipv4\.tcp_.*mem|net\.ipv4\.tcp_no_|net\.ipv4\.tcp_sa|net\.ipv4\.tcp_ti'
net.core.netdev_max_backlog = 250000
net.core.optmem_max = 16777216
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_mem = 39363
209944 314904
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_rmem = 8192
87380 16777216
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_wmem = 8192
65536 16777216
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 36 13800
13810
13820
13830
13840
13850
13860
13870
13880
13890
13900
13910
13920
13930
13940
13950
13960
13970
13980
13990
14000
14010
14020
14030
14040
14050
14060
14070
14080
14090
14100
14110
14120
14130
14140
14150
14160
14170
14180
14190
○
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
○
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
LVM の設定を変更します。
sudo sed -i -e 's/obtain_device_list_from_udev =.*$/obtain_device_list_from_udev = 0/' /etc/lvm/lvm.conf
sudo sed -i -e 's/use_blkid_wiping =.*$/use_blkid_wiping = 0/' /etc/lvm/lvm.conf
sudo sed -i -e 's/use_lvmetad =.*$/use_lvmetad = 0/' /etc/lvm/lvm.conf
sudo sed -i -e 's/use_lvmpolld =.*$/use_lvmpolld = 0/' /etc/lvm/lvm.conf
sudo sed -i -e 's/write_cache_state =.*$/write_cache_state = 0/' /etc/lvm/lvm.conf
sudo patch --ignore-whitespace /etc/lvm/lvm.conf << 'EOF'
diff -upr /etc/lvm/lvm.conf /etc/lvm/lvm.conf.new
--- /etc/lvm/lvm.conf
2015-11-21 12:01:29.000000000 +0900
+++ /etc/lvm/lvm.conf.new
2016-09-21 07:52:14.164259555 +0900
@@ -139,6 +139,7 @@ devices {
#
# This configuration option has an automatic default value.
# filter = [ "a|.*/|" ]
+filter = ["r|vg.*|", "a|sd.*|", "a|drbd.*|", "r|.*|"]
# Configuration option devices/global_filter.
# Limit the block devices that are used by LVM system components.
EOF
sudo
sudo
sudo
sudo
sudo
sudo
systemctl stop lvm2-lvmetad.socket
systemctl stop lvm2-lvmpolld.socket
systemctl disable lvm2-lvmetad.socket
systemctl disable lvm2-lvmpolld.socket
rm -f /etc/lvm/cache/.cache
cp -a /etc/lvm/lvm.conf /etc~/lvm/lvm.conf_$(date +%Y%m%d_%H%M%S)
LVM の設定変更を初期化 RAM ディスクに反映します。
for i in /boot/initramfs-*
do
KVER=$(echo $i | sed -n 's%/boot/initramfs-\(.*\).img%\1%p')
if echo $KVER | grep -q -v rescue; then
if echo $KVER | grep -q -v kdump; then
sudo dracut --force /boot/initramfs-$KVER.img $KVER;
fi
fi
done
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 37 14200
14210
14220
14230
14240
14250
14260
14270
14280
14290
14300
14310
14320
14330
14340
14350
14360
14370
14380
14390
14400
14410
14420
14430
14440
14450
14460
14470
14480
14490
14500
14510
14520
14530
14540
14550
14560
14570
14580
14590
○
a,s
a,s
a,s
a,s
a,s
LVM 物理ボリューム用パーティションを作成します。
echo Yes | sudo parted /dev/sdc mklabel gpt
echo Yes | sudo parted /dev/sdd mklabel gpt
echo Yes | sudo parted /dev/sde mklabel gpt
echo Yes | sudo parted /dev/sdf mklabel gpt
sudo parted -l
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 17.2GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number
1
2
Start
1049kB
525MB
End
525MB
17.2GB
mkpart
mkpart
mkpart
mkpart
primary
primary
primary
primary
1MiB
1MiB
1MiB
1MiB
100%
100%
100%
100%
set
set
set
set
1
1
1
1
lvm
lvm
lvm
lvm
on
on
on
on
Size
Type
File system Flags
524MB primary xfs
boot
16.7GB primary xfs
Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 1074MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number
1
Start
1049kB
End
1074MB
Size
Type
File system
Flags
1073MB primary linux-swap(v1)
Model: VMware Virtual disk (scsi)
Disk /dev/sdc: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number
1
Start
1049kB
End
107GB
Size
107GB
File system Name
Flags
primary lvm
Model: VMware Virtual disk (scsi)
Disk /dev/sdd: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 38 14600
14610
14620
14630
14640
14650
14660
14670
14680
14690
14700
14710
14720
14730
14740
14750
14760
14770
14780
14790
14800
14810
14820
14830
14840
14850
14860
14870
14880
14890
14900
14910
14920
14930
14940
14950
14960
14970
14980
14990
Number
1
Start
1049kB
End
107GB
Size
107GB
File system Name
Flags
primary lvm
Model: VMware Virtual disk (scsi)
Disk /dev/sde: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number
1
Start
1049kB
End
107GB
Size
107GB
File system Name
Flags
primary lvm
Model: VMware Virtual disk (scsi)
Disk /dev/sdf: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number
1
○
Start
1049kB
End
107GB
Size
107GB
File system Name
Flags
primary lvm
LVM 物理ボリュームを作成します。
a,s
sudo pvcreate /dev/sdc1
Physical volume "/dev/sdc1" successfully created
a,s
sudo pvcreate /dev/sdd1
Physical volume "/dev/sdd1" successfully created
a,s
sudo pvcreate /dev/sde1
Physical volume "/dev/sde1" successfully created
a,s
sudo pvcreate /dev/sdf1
Physical volume "/dev/sdf1" successfully created
○
a,s
LVM ボリュームグループを作成します。
sudo vgcreate -s 4M vg0 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
Volume group "vg0" successfully created
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 39 15000
15010
15020
15030
15040
15050
15060
15070
15080
15090
15100
15110
15120
15130
15140
15150
15160
15170
15180
15190
15200
15210
15220
15230
15240
15250
15260
15270
15280
15290
15300
15310
15320
15330
15340
15350
15360
15370
15380
15390
○
a,s
○
a,s
LVM 論理ボリュームを作成します。
sudo lvcreate --name lv-drbd0 --extents 90%FREE vg0
Logical volume "lv-drbd0" created.
LVM の状態を確認します。
sudo pvs
PV
/dev/sdc1
/dev/sdd1
/dev/sde1
/dev/sdf1
VG
vg0
vg0
vg0
vg0
Fmt
lvm2
lvm2
lvm2
lvm2
Attr
a-a-a-a--
PSize PFree
100.00g
0
100.00g
0
100.00g
0
100.00g 40.00g
a,s
sudo vgs
VG #PV #LV #SN Attr
VSize VFree
vg0
4
1
0 wz--n- 399.98g 40.00g
a,s
sudo lvs
LV
VG
lv-drbd0 vg0
○
a,s
Attr
LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
-wi-a----- 359.98g
DRBD の設定ファイルを作成します。
cat /etc/drbd.conf
# You can find an example in
/usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
cat << 'EOF' | sudo tee /etc/drbd.d/global_common.conf
global {
usage-count no;
}
common {
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo 1 > /proc/sys/kernel/sysrq; echo b > /proc/sysrq-trigger; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo 1 > /proc/sys/kernel/sysrq; echo o > /proc/sysrq-trigger; halt -f";
fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 40 15400
15410
15420
15430
15440
15450
15460
15470
15480
15490
15500
15510
15520
15530
15540
15550
15560
15570
15580
15590
15600
15610
15620
15630
15640
15650
15660
15670
15680
15690
15700
15710
15720
15730
15740
15750
15760
15770
15780
15790
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 4";
after-resync-target "/usr/lib/drbd/unsnapshot-resync-target-lvm.sh; /usr/lib/drbd/crm-unfence-peer.sh";
}
startup {
#wfc# wfc-timeout 10;
#wfc# degr-wfc-timeout 10;
#wfc# outdated-wfc-timeout 10;
}
disk {
on-io-error detach;
fencing resource-only;
al-extents 6433;
c-plan-ahead 20;
c-delay-target 100;
c-fill-target 0;
c-max-rate 100M;
c-min-rate 1M;
}
net {
protocol C;
max-buffers 128k;
sndbuf-size 0;
rcvbuf-size 0;
cram-hmac-alg sha1;
shared-secret "password";
congestion-fill 100M;
congestion-extents 2000;
csums-alg md5;
verify-alg md5;
use-rle yes;
}
}
EOF
sudo cp -a /etc/drbd.d/global_common.conf /etc~/drbd.d/global_common.conf_$(date +%Y%m%d_%H%M%S)
cat << 'EOF' | sudo tee /etc/drbd.d/r0.res
resource r0 {
volume 0 {
device /dev/drbd0;
disk /dev/vg0/lv-drbd0;
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 41 15800
15810
15820
15830
15840
15850
15860
15870
15880
15890
15900
15910
15920
15930
15940
15950
15960
15970
15980
15990
16000
16010
16020
16030
16040
16050
16060
16070
16080
16090
16100
16110
16120
16130
16140
16150
16160
16170
16180
16190
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
○
a,s
○
a,s
○
a,s
○
a,s
meta-disk internal;
}
on iscsitgt01a.example.com {
address 192.168.1.2:7788;
}
on iscsitgt01s.example.com {
address 192.168.1.3:7788;
}
}
EOF
DRBD リソースを初期化します。
sudo drbdadm create-md r0
initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.
targetcli から exit する際に自動的に設定を save する挙動を無効化します。
sudo targetcli set global auto_save_on_exit=false
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
Parameter auto_save_on_exit is now 'false'.
targetcli から target を追加する際に自動的に portal が作成される挙動を無効化します。
sudo targetcli set global auto_add_default_portal=false
Parameter auto_add_default_portal is now 'false'.
targetcli コマンドのデフォルト設定を確認します。
sudo targetcli get global
GLOBAL CONFIG GROUP
===================
auto_add_default_portal=false
---------------------------If true, adds a portal listening on all IPs to new targets.
auto_add_mapped_luns=true
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 42 16200
16210
16220
16230
16240
16250
16260
16270
16280
16290
16300
16310
16320
16330
16340
16350
16360
16370
16380
16390
16400
16410
16420
16430
16440
16450
16460
16470
16480
16490
16500
16510
16520
16530
16540
16550
16560
16570
16580
16590
------------------------If true, automatically create node ACLs mapped LUNs after creating a new target LUN or a new node ACL
auto_cd_after_create=false
-------------------------If true, changes current path to newly created objects.
auto_enable_tpgt=true
--------------------If true, automatically enables TPGTs upon creation.
auto_save_on_exit=false
---------------------If true, saves configuration on exit.
color_command=cyan
-----------------Color to use for command completions.
color_default=none
-----------------Default text display color.
color_keyword=cyan
-----------------Color to use for keyword completions.
color_mode=true
--------------Console color display mode.
color_parameter=magenta
----------------------Color to use for parameter completions.
color_path=magenta
-----------------Color to use for path completions
export_backstore_name_as_model=true
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 43 16600
16610
16620
16630
16640
16650
16660
16670
16680
16690
16700
16710
16720
16730
16740
16750
16760
16770
16780
16790
16800
16810
16820
16830
16840
16850
16860
16870
16880
16890
16900
16910
16920
16930
16940
16950 ○
16960
16970 a,s
16980 a,s
16990 a,s
----------------------------------If true, the backstore name is used for the scsi inquiry model name.
logfile=/root/.targetcli/log.txt
-------------------------------Logfile to use.
loglevel_console=info
--------------------Log level for messages going to the console.
loglevel_file=debug
------------------Log level for messages going to the log file.
prompt_length=30
---------------Max length of the shell prompt path, 0 for infinite.
tree_max_depth=0
---------------Maximum depth of displayed node tree.
tree_round_nodes=true
--------------------Tree node display style.
tree_show_root=true
------------------Whether or not to display tree root.
tree_status_mode=true
--------------------Whether or not to display status in tree.
LIO のリソース・エージェントを作成します。
cat << 'EOF_LIO' | sudo tee /usr/lib/ocf/resource.d/heartbeat/LIO
#!/bin/bash
#
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 44 17000
17010
17020
17030
17040
17050
17060
17070
17080
17090
17100
17110
17120
17130
17140
17150
17160
17170
17180
17190
17200
17210
17220
17230
17240
17250
17260
17270
17280
17290
17300
17310
17320
17330
17340
17350
17360
17370
17380
17390
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
LIO OCF RA. manages iSCSI target LIO.
(c) 2009-2010 Florian Haas, Dejan Muhamedagic,
and Linux-HA contributors
modified by Katsuaki Hamada ([email protected]), 23 Oct 2016
This program is free software; you can redistribute it and/or modify
it under the terms of version 2 of the GNU General Public License as
published by the Free Software Foundation.
This program is distributed in the hope that it would be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Further, this software is distributed without any warranty that it is
free of the rightful claim of any third person regarding infringement
or the like. Any license provided herein, whether implied or
otherwise, applies only to this software file. Patent licenses, if
any, provided herein do not apply to combinations of this program with
other software, or any other product whatsoever.
You should have received a copy of the GNU General Public License
along with this program; if not, write the Free Software Foundation,
Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.
#######################################################################
# Initialization:
: ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat}
. ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs
# Lockfile, used for selecting a target ID
LOCKFILE=${HA_RSCTMP}/target.lock
#######################################################################
meta_data() {
cat <<END
<?xml version="1.0"?>
<!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 45 17400
17410
17420
17430
17440
17450
17460
17470
17480
17490
17500
17510
17520
17530
17540
17550
17560
17570
17580
17590
17600
17610
17620
17630
17640
17650
17660
17670
17680
17690
17700
17710
17720
17730
17740
17750
17760
17770
17780
17790
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
<resource-agent name="LIO" version="1.0">
<version>0.9</version>
<longdesc lang="en">
Manages iSCSI target LIO. An iSCSI target is a collection of SCSI Logical
Units (LUs) exported via a daemon that speaks the iSCSI protocol.
</longdesc>
<shortdesc lang="en">iSCSI target export agent</shortdesc>
<parameters>
<parameter name="iqn" required="1" unique="1">
<longdesc lang="en">
The target iSCSI Qualified Name (IQN). Should follow the conventional
iqn.yyyy-mm.&lt;reversed domain name&gt;[:identifier] syntax.
</longdesc>
<shortdesc lang="en">iSCSI target IQN</shortdesc>
<content type="string" />
</parameter>
</parameters>
<actions>
<action name="start" timeout="10" />
<action name="stop" timeout="10" />
<action name="status" timeout="10" interval="10" depth="0" />
<action name="monitor" timeout="10" interval="10" depth="0" />
<action name="meta-data" timeout="5" />
<action name="validate-all" timeout="10" />
</actions>
</resource-agent>
END
}
#######################################################################
LIO_usage() {
cat <<END
usage: $0 {start|stop|status|monitor|validate-all|meta-data}
Expects to have a fully populated OCF RA-compliant environment set.
END
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 46 17800
17810
17820
17830
17840
17850
17860
17870
17880
17890
17900
17910
17920
17930
17940
17950
17960
17970
17980
17990
18000
18010
18020
18030
18040
18050
18060
18070
18080
18090
18100
18110
18120
18130
18140
18150
18160
18170
18180
18190
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
}
LIO_start() {
LIO_monitor
[ $? = $OCF_SUCCESS ] && return $OCF_SUCCESS
/usr/bin/targetctl restore
LIO_monitor
}
LIO_stop() {
LIO_monitor
[ $? -eq $OCF_NOT_RUNNING ] || /usr/bin/targetctl clear
return $OCF_SUCCESS
}
LIO_monitor() {
# if we have no configfs entry for the target, it's definitely stopped
[ -d /sys/kernel/config/target/iscsi/${OCF_RESKEY_iqn} ] || return $OCF_NOT_RUNNING
# if the target is there, but its TPG is not enabled, then we also consider it stopped
[ $(cat /sys/kernel/config/target/iscsi/${OCF_RESKEY_iqn}/tpgt_1/enable) -eq 1 ] || return $OCF_NOT_RUNNING
return $OCF_SUCCESS
}
LIO_validate() {
if ! ocf_is_probe; then
# Do we have all required binaries?
check_binary targetctl
fi
return $OCF_SUCCESS
}
case $1 in
meta-data) meta_data; exit $OCF_SUCCESS;;
usage|help) LIO_usage; exit $OCF_SUCCESS;;
esac
# Everything except usage and meta-data must pass the validate test
LIO_validate
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 47 18200
18210
18220
18230
18240
18250
18260
18270
18280
18290
18300
18310
18320
18330
18340
18350
18360
18370
18380
18390
18400
18410
18420
18430
18440
18450
18460
18470
18480
18490
18500
18510
18520
18530
18540
18550
18560
18570
18580
18590
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
○
a,s
a,s
case $__OCF_ACTION in
start)
LIO_start;;
stop)
LIO_stop;;
monitor|status) LIO_monitor;;
reload)
ocf_log err "Reloading..."; LIO_start;;
validate-all)
;;
*)
LIO_usage; exit $OCF_ERR_UNIMPLEMENTED;;
esac
rc=$?
ocf_log debug "${OCF_RESOURCE_INSTANCE} $__OCF_ACTION : $rc"
exit $rc
EOF_LIO
sudo chmod 755 /usr/lib/ocf/resource.d/heartbeat/LIO
VIP に関するリソース・エージェント (IPaddr2) の名前を変更します。
sed -e 's/IPaddr2/VIP/g' /usr/lib/ocf/resource.d/heartbeat/IPaddr2 | sudo tee /usr/lib/ocf/resource.d/heartbeat/VIP > /dev/null
sudo chmod 755 /usr/lib/ocf/resource.d/heartbeat/VIP
※　LVM, LIO, VIP リソース・エージェント名文字数を統一し、「sudo pcs status」等の実行結果を見やすくします。
※　問い合わせを行う時は、IPaddr2 を上記のコマンドで変更している点を伝えないと話が通じないものと思われます。
○
pcs の利用環境を整えます。
a,s
echo 'password' | sudo passwd --stdin hacluster
Changing password for user hacluster.
passwd: all authentication tokens updated successfully.
a,s
a,s
sudo cp -a /etc/shadow
sudo cp -a /etc/shadow-
a,s
sudo usermod -a -G haclient admin
a,s
id admin
uid=1000(admin) gid=1000(admin) groups=1000(admin),10(wheel),189(haclient)
a,s
sudo usermod -a -G haclient monitor
a,s
id monitor
/etc~/shadow_$(date +%Y%m%d_%H%M%S)
/etc~/shadow-_$(date +%Y%m%d_%H%M%S)
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 48 18600
18610
18620
18630
18640
18650
18660
18670
18680
18690
18700
18710
18720
18730
18740
18750
18760
18770
18780
18790
18800
18810
18820
18830
18840
18850
18860
18870
18880
18890
18900
18910
18920
18930
18940
18950
18960
18970
18980
18990
uid=1001(monitor) gid=1001(monitor) groups=1001(monitor),189(haclient)
a,s
a,s
sudo cp -a /etc/group
sudo cp -a /etc/group-
a,s
a,s
sudo systemctl start pcsd
sudo systemctl enable pcsd
Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.
○
a,s
a,s
a,s
a,s
/etc~/group_$(date +%Y%m%d_%H%M%S)
/etc~/group-_$(date +%Y%m%d_%H%M%S)
Corosync のサービス設定を変更します。
sed -e 's/^#Restart=on-failure.*$/Restart=on-failure/' \
-e 's/^#RestartSec=.*$/RestartSec=70/' \
-e 's%^#ExecStartPre=/sbin/modprobe softdog soft_margin=.*$%ExecStartPre=/sbin/modprobe softdog soft_margin=6%' \
/usr/lib/systemd/system/corosync.service | sudo tee /etc/systemd/system/corosync.service
[Unit]
Description=Corosync Cluster Engine
ConditionKernelCommandLine=!nocluster
Requires=network-online.target
After=network-online.target
[Service]
ExecStart=/usr/share/corosync/corosync start
ExecStop=/usr/share/corosync/corosync stop
Type=forking
# The following config is for corosync with enabled watchdog service.
#
# When corosync watchdog service is being enabled and using with
# pacemaker.service, and if you want to exert the watchdog when a
# corosync process is terminated abnormally,
# uncomment the line of the following Restart= and RestartSec=.
Restart=on-failure
# Specify a period longer than soft_margin as RestartSec.
RestartSec=70
# rewrite according to environment.
ExecStartPre=/sbin/modprobe softdog soft_margin=6
[Install]
WantedBy=multi-user.target
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 49 19000
19010
19020
19030
19040 a,s
19050
19060
19070
19080
19090
19100
19110
19120
19130
19140
19150
19160 ○
19170
19180 a,s
19190 a,s
19200
19210
19220
19230
19240
19250
19260
19270
19280
19290
19300
19310
19320
19330
19340
19350
19360
19370
19380
19390
※　カーネル内のソフトウェア watchdog 機能を有効化します。
※　Corosync プロセス障害検知時間を6秒以内とします。
cat /etc/sysconfig/corosync
# Corosync init script configuration file
# COROSYNC_INIT_TIMEOUT specifies number of seconds to wait for corosync
# initialization (default is one minute).
COROSYNC_INIT_TIMEOUT=60
# COROSYNC_OPTIONS specifies options passed to corosync command
# (default is no options).
# See "man corosync" for detailed descriptions of the options.
COROSYNC_OPTIONS=""
Pacemaker のサービス設定を変更します。
sed -e "s%^# ExecStopPost=/bin/sh -c 'pidof crmd || killall -TERM corosync'$%ExecStopPost=/bin/sh -c 'pidof crmd || killall -TERM corosync'%" \
/usr/lib/systemd/system/pacemaker.service | sudo tee /etc/systemd/system/pacemaker.service
[Unit]
Description=Pacemaker High Availability Cluster Manager
After=dbus.service
After=basic.target
After=syslog.service
After=network.target
After=corosync.service
Requires=dbus.service
Requires=basic.target
Requires=corosync.service
# if you use crm_mon, uncomment the line below.
# Wants=crm_mon.service
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 50 19400
19410
19420
19430
19440
19450
19460
19470
19480
19490
19500
19510
19520
19530
19540
19550
19560
19570
19580
19590
19600
19610
19620
19630
19640
19650
19660
19670
19680
19690
19700
19710
19720
19730
19740
19750
19760
19770
19780
19790
KillMode=process
NotifyAccess=main
EnvironmentFile=-/etc/sysconfig/pacemaker
EnvironmentFile=-/etc/sysconfig/sbd
SuccessExitStatus=100
ExecStart=/usr/sbin/pacemakerd -f
# If pacemakerd doesn't stop, its probably waiting on a cluster
# resource. Sending -KILL will just get the node fenced
SendSIGKILL=no
#
#
#
#
#
#
#
#
If we ever hit the StartLimitInterval/StartLimitBurst limit and the
admin wants to stop the cluster while pacemakerd is not running, it
might be a good idea to enable the ExecStopPost directive below.
Although the node will likely end up being fenced as a result so its
not on by default
ExecStopPost=/usr/bin/killall -TERM crmd attrd fenced cib pengine lrmd
# If you want Corosync to stop whenever Pacemaker is stopped,
# uncomment the next line too:
#
ExecStopPost=/bin/sh -c 'pidof crmd || killall -TERM corosync'
# Uncomment this for older versions of systemd that didn't support
# TimeoutStopSec
# TimeoutSec=30min
# Pacemaker can only exit after all managed services have shut down
# A HA database could conceivably take even longer than this
TimeoutStopSec=30min
TimeoutStartSec=60s
# Restart options include: no, on-success, on-failure, on-abort or always
Restart=on-failure
# crm_perror() writes directly to stderr, so ignore it here
# to avoid double-logging with the wrong format
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 51 19800
19810
19820
19830
19840
19850
19860
19870 a,s
19880 a,s
19890 a,s
19900
19910
19920
19930
19940
19950
19960
19970
19980
19990
20000
20010
20020
20030
20040
20050
20060
20070
20080
20090
20100
20110
20120
20130
20140
20150
20160
20170
20180
20190
StandardError=null
# if you use crm_mon, uncomment the line below.
# ExecStopPost=/bin/sh -c 'systemctl status crm_mon >/dev/null && systemctl stop crm_mon'
※　Pacemaker サービス停止時に Corosync サービスを停止します。
sudo sed -i -e 's/^# PCMK_fail_fast=.*$/PCMK_fail_fast=yes/' /etc/sysconfig/pacemaker
sudo cp -a /etc/sysconfig/pacemaker /etc~/sysconfig/pacemaker_$(date +%Y%m%d_%H%M%S)
cat /etc/sysconfig/pacemaker
# For non-systemd based systems, prefix export to each enabled line
# Turn on special handling for CMAN clusters in the init script
# Without this, fenced (and by inference, cman) cannot reliably be made to shut down
# PCMK_STACK=cman
#==#==# Variables that control logging
#
#
#
#
Enable debug logging globally or per-subsystem
Multiple subsystems may me listed separated by commas
eg. PCMK_debug=crmd,pengine
PCMK_debug=yes|no|crmd|pengine|cib|stonith-ng|attrd|pacemakerd
#
#
#
#
Send INFO (and higher) messages to the named log file
Additional messages may also appear here depending on any configured debug and trace settings
By default Pacemaker will inherit the logfile specified in corosync.conf
PCMK_logfile=/var/log/pacemaker.log
#
#
#
#
Specify an alternate syslog target for NOTICE (and higher) messages
Use 'none' to disable - not recommended
The default value is 'daemon'
PCMK_logfacility=none|daemon|user|local0|local1|local2|local3|local4|local5|local6|local7
#
#
#
#
Send all messages up-to-and-including the configured priority to syslog
A value of 'info' will be far too verbose for most installations and 'debug' is almost certain to send you blind
The default value is 'notice'
PCMK_logpriority=emerg|alert|crit|error|warning|notice|info|debug
# Log all messages from a comma-separated list of functions
# PCMK_trace_functions=function1,function2,function3
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 52 20200
20210
20220
20230
20240
20250
20260
20270
20280
20290
20300
20310
20320
20330
20340
20350
20360
20370
20380
20390
20400
20410
20420
20430
20440
20450
20460
20470
20480
20490
20500
20510
20520
20530
20540
20550
20560
20570
20580
20590
# Log all messages from a comma-separated list of files (no path)
# Supports wildcards eg. PCMK_trace_files=prefix*.c
# PCMK_trace_files=file.c,other.h
# Log all messages matching comma-separated list of formats
# PCMK_trace_formats="Sent delete %d"
# Log all messages from a comma-separated list of tags
# PCMK_trace_tags=tag1,tag2
# Dump the blackbox whenever the message at function and line is printed
# eg. PCMK_trace_blackbox=te_graph_trigger:223,unpack_clone:81
# PCMK_trace_blackbox=fn:line,fn2:line2,...
#
#
#
#
#
#
#
#
#
#
Enable blackbox logging globally or per-subsystem
The blackbox contains a rolling buffer of all logs (including info+debug+trace)
and is written after a crash, assertion failure and/or when SIGTRAP is received
The blackbox recorder can also be enabled for Pacemaker daemons at runtime by
sending SIGUSR1 (or SIGTRAP), and disabled by sending SIGUSR2
Multiple subsystems may me listed separated by commas
eg. PCMK_blackbox=crmd,pengine
PCMK_blackbox=yes|no|crmd|pengine|cib|stonith-ng|attrd|pacemakerd
#==#==# Advanced use only
# Enable this for compatibility with older corosync (prior to 2.0)
# based clusters which used the nodes uname as its uuid also
# PCMK_uname_is_uuid=no
# Specify an alternate location for RNG schemas and XSL transforms
# Mostly only useful for developer testing
# PCMK_schema_directory=/some/path
# Enable this for rebooting this machine at the time of process (subsystem) failure
PCMK_fail_fast=yes
#==#==# Pacemaker Remote
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 53 20600
20610
20620
20630
20640
20650
20660
20670
20680
20690
20700
20710
20720
20730
20740
20750
20760
20770
20780
20790
20800
20810
20820
20830
20840
20850
20860
20870
20880
20890
20900
20910
20920
20930
20940
20950
20960
20970
20980
20990
#
#
#
#
#
Use a custom directory for finding the authkey.
PCMK_authkey_location=/etc/pacemaker/authkey
Specify a custom port for Pacemaker Remote connections
PCMK_remote_port=3121
#==#==# IPC
# Force use of a particular class of IPC connection
# PCMK_ipc_type=shared-mem|socket|posix|sysv
# Specify an IPC buffer size in bytes
# Useful when connecting to really big clusters that exceed the default 20k buffer
# PCMK_ipc_buffer=20480
#==#==# Profiling and memory leak testing
#
#
#
#
#
#
#
#
#
Variables for running child daemons under valgrind and/or checking for memory problems
G_SLICE=always-malloc
MALLOC_PERTURB_=221 # or 0
MALLOC_CHECK_=3
# or 0,1,2
PCMK_valgrind_enabled=yes
PCMK_valgrind_enabled=cib,crmd
PCMK_callgrind_enabled=yes
PCMK_callgrind_enabled=cib,crmd
VALGRIND_OPTS="--leak-check=full --trace-children=no --num-callers=25 --log-file=/var/lib/pacemaker/valgrind-%p
--suppressions=/usr/share/pacemaker/tests/valgrind-pcmk.suppressions --gen-suppressions=all"
※　Pacemaker の内部プロセス障害をノード障害として扱うようにします。
○
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
Pacemaker のリソース設定スクリプトを作成します。
cat << 'EOF'
#!/bin/bash
pcs property
pcs property
pcs property
pcs property
pcs property
pcs property
| sudo tee /etc/ha.d/crm.sh
set
set
set
set
set
set
batch-limit=30
cluster-delay=60
cluster-recheck-interval=15min
crmd-finalization-timeout=30min
crmd-integration-timeout=3min
crmd-transition-delay=0s
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 54 21000
21010
21020
21030
21040
21050
21060
21070
21080
21090
21100
21110
21120
21130
21140
21150
21160
21170
21180
21190
21200
21210
21220
21230
21240
21250
21260
21270
21280
21290
21300
21310
21320
21330
21340
21350
21360
21370
21380
21390
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
pcs
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
property
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
set
dc-deadtime=20s
default-action-timeout=20
election-timeout=2min
enable-acl=true --force
enable-startup-probes=true
is-managed-default=true
load-threshold=80%
maintenance-mode=false
migration-limit=-1
no-quorum-policy=ignore
node-action-limit=0
node-health-green=0
node-health-red=-INFINITY
node-health-strategy=none
node-health-yellow=0
notification-agent=/dev/null
pe-error-series-max=100
pe-input-series-max=100
pe-warn-series-max=100
placement-strategy=default
remove-after-stop=false
shutdown-escalation=20min
start-failure-is-fatal=true
startup-fencing=true
stonith-action=reboot
stonith-enabled=false
stonith-timeout=60
stop-all-resources=false
stop-orphan-actions=true
stop-orphan-resources=true
symmetric-cluster=true
pcs resource defaults resource-stickiness=200 migration-threshold=2
pcs acl role create write-access description="Full access" write xpath /cib
pcs acl role create read-only description="Read access to cluster" read xpath /cib
pcs acl user create admin write-access
pcs acl user create monitor read-only
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 55 21400
21410
21420
21430
21440
21450
21460
21470
21480
21490
21500
21510
21520
21530
21540
21550
21560
21570
21580
21590
21600
21610
21620
21630
21640
21650
21660
21670
21680
21690
21700
21710
21720
21730
21740
21750
21760
21770
21780
21790
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
a,s
pcs resource create p_drbd_r0 ocf:linbit:drbd \
params drbd_resource=r0 \
op start
timeout=240 \
op monitor interval=10 role=Master timeout=20 \
op monitor interval=20 role=Slave timeout=20 \
op notify
timeout=90 \
op stop
timeout=100 \
op promote
timeout=90 \
op demote
timeout=90
pcs resource master ms_drbd_r0 p_drbd_r0 \
meta master-max=1 master-node-max=1 clone-max=2 \
clone-node-max=1 notify=true target-role=Started \
is-managed=true
pcs resource create p_lvm ocf:heartbeat:LVM \
params volgrpname=vg1 \
op start
timeout=30 \
op monitor interval=5 timeout=10 \
op stop
timeout=30
pcs resource create p_lio ocf:heartbeat:LIO \
params iqn=iqn.2016-09.com.example:iscsitgt01 \
op start
timeout=10 \
op monitor interval=5 timeout=5 \
op stop
timeout=10
pcs resource create p_vip ocf:heartbeat:VIP \
params ip=10.110.88.59 cidr_netmask=26 nic=bond0 iflabel=1 arp_interval=200 arp_count=5 \
op start
timeout=20 \
op monitor interval=5 timeout=10 \
op stop
timeout=20
pcs resource group add g_tgt p_lvm p_lio p_vip
pcs constraint location add lc_tgt g_tgt iscsitgt01a.example.com 100
pcs constraint colocation add g_tgt \
ms_drbd_r0 INFINITY with-rsc-role=Master
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 56 21800
21810
21820
21830
21840
21850
21860
21870
21880
21890
21900
21910
21920
21930
21940
21950
21960
21970
21980
21990
22000
22010
22020
22030
22040
22050
22060
22070
22080
22090
22100
22110
22120
22130
22140
22150
22160
22170
22180
22190
a,s
a,s
a,s
a,s
pcs constraint order promote ms_drbd_r0 then start p_lvm
EOF
sudo chmod 755 /etc/ha.d/crm.sh
sudo cp -a /etc{,~}/ha.d/crm.sh
※　ここからの作業は、Active 機と Stand-by 機が連動して動作していく前提の操作となります。
○
a,s
Active 機と Stand-by 機の間の疎通を確認します。
ping -c 1 -M do -s 8972 10.110.88.57 || echo Error
PING 10.110.88.57 (10.110.88.57) 8972(9000) bytes of data.
8980 bytes from 10.110.88.57: icmp_seq=1 ttl=64 time=0.136 ms
--- 10.110.88.57 ping statistics --1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.136/0.136/0.136/0.000 ms
a,s
traceroute -F 10.110.88.57 8972
traceroute to 10.110.88.57 (10.110.88.57), 30 hops max, 8972 byte packets
1 iscsitgt01a.example.com (10.110.88.57) 0.303 ms 0.265 ms 0.256 ms
a,s
a,s
ping -c 1 -M do -s 8972 10.110.88.58 || echo Error
traceroute -F 10.110.88.58 8972
a,s
a,s
ping -c 1 -M do -s 8972 192.168.1.2 || echo Error
traceroute -F 192.168.1.2 8972
a,s
a,s
ping -c 1 -M do -s 8972 192.168.1.3 || echo Error
traceroute -F 192.168.1.3 8972
○
a
a
a
a
a
Active 機で ssh 鍵を作成し、Stand-by 機にコピーします。
ssh-keygen -q -f ~/.ssh/id_rsa -N ""
mv -f ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
scp -pr .ssh/ iscsitgt01s:
The authenticity of host 'iscsitgt01s (10.110.88.58)' can't be established.
ECDSA key fingerprint is cf:3a:39:91:fc:c9:ac:5c:4e:16:38:72:97:88:28:b2.
#Are
yesyou sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'iscsitgt01s,10.110.88.58' (ECDSA) to the list of known hosts.
#admin@iscsitgt01s's
********
password: ********
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 57 22200
22210
22220
22230
22240 ○
22250
22260 a,s
22270 a,s
22280
22290 ○
22300
22310 a,s
22320
22330 ○
22340
22350
s
22360
22370
22380
22390
22400
22410
22420
22430 ○
22440
22450
a
22460
22470 ○
22480
22490
22500
22510
22520
22530
22540
22550
22560
22570
22580
22590
id_rsa
authorized_keys
known_hosts
100% 1679
100% 411
100% 186
1.6KB/s
0.4KB/s
0.2KB/s
00:00
00:00
00:00
Active 機と Stand-by 機で、ssh 鍵を root アカウント用にコピーします。
sudo cp -a .ssh/ /root/
sudo chown -R root:root /root/.ssh
Active 機と Stand-by 機でほぼ同時に DRBD サービスを起動します。
sudo systemctl start drbd.service
Stand-by 機で DRBD の状態をワッチします。
watch cat /proc/drbd
Every 2.0s: cat /proc/drbd
Sat Oct 29 18:33:24 2016
version: 8.4.5 (api:1/proto:86-101)
srcversion: 1AEFF755B8BD61B81A0AF27
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:3759976
Active 機で DRBD の初期同期を開始します。
sudo drbdadm primary --force all
Stand-by 機で DRBD の状態を確認します。
Every 2.0s: cat /proc/drbd
Sat Oct 29 18:33:24 2016
version: 8.4.5 (api:1/proto:86-101)
srcversion: 1AEFF755B8BD61B81A0AF27
0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----ns:0 nr:0 dw:0 dr:1244 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:3758732
[>....................] sync'ed: 0.2% (3758732/3759976)K
finish: 0:39:09 speed: 1,244 (1,244) want: 10,400 K/sec
※　この状態でも、Active 機側で作業を続行できます。今回は、初期同期の完了を待つことにします。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 58 22600
22610
22620
22630
22640
22650
22660
22670
22680
22690
22700
22710
22720
22730
22740
22750
22760
22770
22780
22790
22800
22810
22820
22830
22840
22850
22860
22870
22880
22890
22900
22910
22920
22930
22940
22950
22960
22970
22980
22990
Every 2.0s: cat /proc/drbd
Sat Oct 29 18:33:24 2016
version: 8.4.5 (api:1/proto:86-101)
srcversion: 1AEFF755B8BD61B81A0AF27
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----ns:0 nr:0 dw:0 dr:3759976 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
※　「自機/対向機」がともに「UpToDate/UpToDate」となっているのが正常な状態です。
○
Active 機で DRBD デバイス上に LVM を構成します。
a
sudo pvcreate /dev/drbd0
Physical volume "/dev/drbd0" successfully created
a
sudo vgcreate -s 4M vg1 /dev/drbd0
Volume group "vg1" successfully created
a
sudo lvcreate --name lv-lun0 --extents 20%VG vg1
Logical volume "lv-lun0" created.
a
sudo lvcreate --name lv-lun1 --extents 20%VG vg1
Logical volume "lv-lun1" created.
a
sudo lvcreate --name lv-lun2 --extents 20%VG vg1
Logical volume "lv-lun2" created.
a
sudo lvcreate --name lv-lun3 --extents 20%VG vg1
Logical volume "lv-lun3" created.
a
sudo pvs
PV
/dev/drbd0
/dev/sdc1
/dev/sdd1
/dev/sde1
/dev/sdf1
a
VG
Fmt Attr
vg1 lvm2 a-vg0 lvm2 a-vg0 lvm2 a-vg0 lvm2 a-vg0 lvm2 a--
PSize PFree
359.97g 72.00g
100.00g
0
100.00g
0
100.00g
0
100.00g 40.00g
sudo vgs
VG
#PV #LV #SN Attr
VSize VFree
vg0
4
1
0 wz--n- 399.98g 40.00g
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 59 23000
23010
23020
23030
23040
23050
23060
23070
23080
23090
23100
23110
23120
23130
23140
23150
23160
23170
23180
23190
23200
23210
23220
23230
23240
23250
23260
23270
23280
23290
23300
23310
23320
23330
23340
23350
23360
23370
23380
23390
vg1
a
○
a
○
1
4
sudo lvs
LV
lv-drbd0
lv-lun0
lv-lun1
lv-lun2
lv-lun3
VG
vg0
vg1
vg1
vg1
vg1
0 wz--n- 359.97g 72.00g
Attr
LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
-wi-ao---- 359.98g
-wi-a----- 71.99g
-wi-a----- 71.99g
-wi-a----- 71.99g
-wi-a----- 71.99g
Active 機で、targetcli から状態を確認します。
sudo targetcli ls /
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 0]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 0]
o- loopback ..................................................... [Targets: 0]
Active 機で、IQN を定義します。
a
sudo targetcli /iscsi create iqn.2016-09.com.example:iscsitgt01
Created target iqn.2016-09.com.example:iscsitgt01.
Created TPG 1.
a
sudo targetcli ls /
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 0]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 1]
| o- iqn.2016-09.com.example:iscsitgt01 ............................ [TPGs: 1]
|
o- tpg1 ........................................... [no-gen-acls, no-auth]
|
o- acls ...................................................... [ACLs: 0]
|
o- luns ...................................................... [LUNs: 0]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 60 23400
23410
23420
23430
23440
23450
23460
23470
23480
23490
23500
23510
23520
23530
23540
23550
23560
23570
23580
23590
23600
23610
23620
23630
23640
23650
23660
23670
23680
23690
23700
23710
23720
23730
23740
23750
23760
23770
23780
23790
|
o- portals ................................................ [Portals: 0]
o- loopback ..................................................... [Targets: 0]
○
Active 機で、ACL（アクセス許可リスト）にイニシエータ名を登録します。必要に応じて CHAP 認証情報も紐付けします。
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls create iqn.2016-09.com.example:initiator01
Created Node ACL for iqn.2016-09.com.example:initiator01
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator01 set auth userid=iscsiuser01
Parameter userid is now 'iscsiuser01'.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator01 set auth password='password-user01'
Parameter password is now 'password-user01'.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls create iqn.2016-09.com.example:initiator02
Created Node ACL for iqn.2016-09.com.example:initiator02
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator02 set auth userid=iscsiuser02
Parameter userid is now 'iscsiuser02'.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator02 set auth password='password-user02'
Parameter password is now 'password-user02'.
a
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls create iqn.2016-09.com.example:initiator03
Created Node ACL for iqn.2016-09.com.example:initiator03
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator03 set auth userid=iscsiuser03
Parameter userid is now 'iscsiuser03'.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator03 set auth password='password-user03'
Parameter password is now 'password-user03'.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls create iqn.2016-09.com.example:initiator04
Created Node ACL for iqn.2016-09.com.example:initiator04
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator04 set auth userid=iscsiuser04
Parameter userid is now 'iscsiuser04'.
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 61 23800
23810
23820
23830
23840
23850
23860
23870
23880
23890
23900
23910
23920
23930
23940
23950
23960
23970
23980
23990
24000
24010
24020
24030
24040
24050
24060
24070
24080
24090
24100
24110
24120
24130
24140
24150
24160
24170
24180
24190
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator04 set auth password='password-user04'
Parameter password is now 'password-user04'.
a
○
sudo targetcli ls /
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 0]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 1]
| o- iqn.2016-09.com.example:iscsitgt01 ............................ [TPGs: 1]
| o- tpg1 ........................................... [no-gen-acls, no-auth]
|
o- acls ...................................................... [ACLs: 4]
|
| o- iqn.2016-09.com.example:initiator01 .............. [Mapped LUNs: 0]
|
| o- iqn.2016-09.com.example:initiator02 .............. [Mapped LUNs: 0]
|
| o- iqn.2016-09.com.example:initiator03 .............. [Mapped LUNs: 0]
|
| o- iqn.2016-09.com.example:initiator04 .............. [Mapped LUNs: 0]
|
o- luns ...................................................... [LUNs: 0]
|
o- portals ................................................ [Portals: 0]
o- loopback ..................................................... [Targets: 0]
Active 機で、バックエンド・デバイスを指定し、IQN に紐付けます。
a
sudo targetcli /backstores/block create name=lun0 dev=/dev/vg1/lv-lun0
Created block storage object lun0 using /dev/vg1/lv-lun0.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/luns create /backstores/block/lun0
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2016-09.com.example:initiator04
Created LUN 0->0 mapping in node ACL iqn.2016-09.com.example:initiator03
Created LUN 0->0 mapping in node ACL iqn.2016-09.com.example:initiator02
Created LUN 0->0 mapping in node ACL iqn.2016-09.com.example:initiator01
a
sudo targetcli /backstores/block create name=lun1 dev=/dev/vg1/lv-lun1
Created block storage object lun1 using /dev/vg1/lv-lun1.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/luns create /backstores/block/lun1
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 62 24200
24210
24220
24230
24240
24250
24260
24270
24280
24290
24300
24310
24320
24330
24340
24350
24360
24370
24380
24390
24400
24410
24420
24430
24440
24450
24460
24470
24480
24490
24500
24510
24520
24530
24540
24550
24560
24570
24580
24590
Created
Created
Created
Created
Created
LUN
LUN
LUN
LUN
LUN
1.
1->1
1->1
1->1
1->1
mapping
mapping
mapping
mapping
in
in
in
in
node
node
node
node
ACL
ACL
ACL
ACL
iqn.2016-09.com.example:initiator04
iqn.2016-09.com.example:initiator03
iqn.2016-09.com.example:initiator02
iqn.2016-09.com.example:initiator01
a
sudo targetcli /backstores/block create name=lun2 dev=/dev/vg1/lv-lun2
Created block storage object lun2 using /dev/vg1/lv-lun2.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/luns create /backstores/block/lun2
Created LUN 2.
Created LUN 2->2 mapping in node ACL iqn.2016-09.com.example:initiator04
Created LUN 2->2 mapping in node ACL iqn.2016-09.com.example:initiator03
Created LUN 2->2 mapping in node ACL iqn.2016-09.com.example:initiator02
Created LUN 2->2 mapping in node ACL iqn.2016-09.com.example:initiator01
a
sudo targetcli /backstores/block create name=lun3 dev=/dev/vg1/lv-lun3
Created block storage object lun3 using /dev/vg1/lv-lun3.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/luns create /backstores/block/lun3
Created LUN 3.
Created LUN 3->3 mapping in node ACL iqn.2016-09.com.example:initiator04
Created LUN 3->3 mapping in node ACL iqn.2016-09.com.example:initiator03
Created LUN 3->3 mapping in node ACL iqn.2016-09.com.example:initiator02
Created LUN 3->3 mapping in node ACL iqn.2016-09.com.example:initiator01
a
sudo targetcli ls /
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 4]
| | o- lun0 ............... [/dev/vg1/lv-lun0 (732.0MiB) write-thru activated]
| | o- lun1 ............... [/dev/vg1/lv-lun1 (732.0MiB) write-thru activated]
| | o- lun2 ............... [/dev/vg1/lv-lun2 (732.0MiB) write-thru activated]
| | o- lun3 ............... [/dev/vg1/lv-lun3 (732.0MiB) write-thru activated]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 1]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 63 24600
24610
24620
24630
24640
24650
24660
24670
24680
24690
24700
24710
24720
24730
24740
24750
24760
24770
24780
24790
24800
24810
24820
24830
24840
24850
24860
24870
24880
24890
24900
24910
24920
24930
24940
24950
24960
24970
24980
24990
| o- iqn.2016-09.com.example:iscsitgt01 ............................ [TPGs: 1]
| o- tpg1 ........................................... [no-gen-acls, no-auth]
|
o- acls ...................................................... [ACLs: 4]
|
| o- iqn.2016-09.com.example:initiator01 .............. [Mapped LUNs: 4]
|
| | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)]
|
| | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)]
|
| | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)]
|
| | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)]
|
| o- iqn.2016-09.com.example:initiator02 .............. [Mapped LUNs: 4]
|
| | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)]
|
| | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)]
|
| | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)]
|
| | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)]
|
| o- iqn.2016-09.com.example:initiator03 .............. [Mapped LUNs: 4]
|
| | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)]
|
| | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)]
|
| | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)]
|
| | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)]
|
| o- iqn.2016-09.com.example:initiator04 .............. [Mapped LUNs: 4]
|
|
o- mapped_lun0 .............................. [lun0 block/lun0 (rw)]
|
|
o- mapped_lun1 .............................. [lun1 block/lun1 (rw)]
|
|
o- mapped_lun2 .............................. [lun2 block/lun2 (rw)]
|
|
o- mapped_lun3 .............................. [lun3 block/lun3 (rw)]
|
o- luns ...................................................... [LUNs: 4]
|
| o- lun0 .............................. [block/lun0 (/dev/vg1/lv-lun0)]
|
| o- lun1 .............................. [block/lun1 (/dev/vg1/lv-lun1)]
|
| o- lun2 .............................. [block/lun2 (/dev/vg1/lv-lun2)]
|
| o- lun3 .............................. [block/lun3 (/dev/vg1/lv-lun3)]
|
o- portals ................................................ [Portals: 0]
o- loopback ..................................................... [Targets: 0]
○
Active 機で、IQN に portal を作成します。
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/portals create 10.110.88.59 3260
Using default IP port 3260
Created network portal 10.110.88.59:3260.
a
sudo targetcli ls /
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 64 25000
25010
25020
25030
25040
25050
25060
25070
25080
25090
25100
25110
25120
25130
25140
25150
25160
25170
25180
25190
25200
25210
25220
25230
25240
25250
25260
25270
25280
25290
25300
25310
25320
25330
25340
25350
25360
25370
25380
25390
| o- block .............................................. [Storage Objects: 4]
| | o- lun0 ............... [/dev/vg1/lv-lun0 (732.0MiB) write-thru activated]
| | o- lun1 ............... [/dev/vg1/lv-lun1 (732.0MiB) write-thru activated]
| | o- lun2 ............... [/dev/vg1/lv-lun2 (732.0MiB) write-thru activated]
| | o- lun3 ............... [/dev/vg1/lv-lun3 (732.0MiB) write-thru activated]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 1]
| o- iqn.2016-09.com.example:iscsitgt01 ............................ [TPGs: 1]
| o- tpg1 ........................................... [no-gen-acls, no-auth]
|
o- acls ...................................................... [ACLs: 4]
|
| o- iqn.2016-09.com.example:initiator01 .............. [Mapped LUNs: 4]
|
| | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)]
|
| | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)]
|
| | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)]
|
| | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)]
|
| o- iqn.2016-09.com.example:initiator02 .............. [Mapped LUNs: 4]
|
| | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)]
|
| | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)]
|
| | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)]
|
| | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)]
|
| o- iqn.2016-09.com.example:initiator03 .............. [Mapped LUNs: 4]
|
| | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)]
|
| | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)]
|
| | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)]
|
| | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)]
|
| o- iqn.2016-09.com.example:initiator04 .............. [Mapped LUNs: 4]
|
|
o- mapped_lun0 .............................. [lun0 block/lun0 (rw)]
|
|
o- mapped_lun1 .............................. [lun1 block/lun1 (rw)]
|
|
o- mapped_lun2 .............................. [lun2 block/lun2 (rw)]
|
|
o- mapped_lun3 .............................. [lun3 block/lun3 (rw)]
|
o- luns ...................................................... [LUNs: 4]
|
| o- lun0 .............................. [block/lun0 (/dev/vg1/lv-lun0)]
|
| o- lun1 .............................. [block/lun1 (/dev/vg1/lv-lun1)]
|
| o- lun2 .............................. [block/lun2 (/dev/vg1/lv-lun2)]
|
| o- lun3 .............................. [block/lun3 (/dev/vg1/lv-lun3)]
|
o- portals ................................................ [Portals: 1]
|
o- 10.110.88.59:3260 ............................................ [OK]
o- loopback ..................................................... [Targets: 0]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 65 25400
25410
25420
25430
25440
25450
25460
25470
25480
25490
25500
25510
25520
25530
25540
25550
25560
25570
25580
25590
25600
25610
25620
25630
25640
25650
25660
25670
25680
25690
25700
25710
25720
25730
25740
25750
25760
25770
25780
25790
a
○
ss -ant | grep LISTEN.\*3260
LISTEN
0
256
10.110.88.59:3260
*:*
Active 機で、設定ファイルに設定を保存し、Stand-by 機にコピーします。
a
sudo targetcli saveconfig
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
a
sudo scp -p /etc/target/saveconfig.json iscsitgt01s:/etc/target/saveconfig.json
saveconfig.json
100% 11KB 11.0KB/s 00:00
○
Active 機で、設定ファイルを確認します。
a
sudo cat /etc/target/saveconfig.json
{
"fabric_modules": [],
"storage_objects": [
{
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 0,
"emulate_fua_read": 0,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 0,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 65535,
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 66 25800
25810
25820
25830
25840
25850
25860
25870
25880
25890
25900
25910
25920
25930
25940
25950
25960
25970
25980
25990
26000
26010
26020
26030
26040
26050
26060
26070
26080
26090
26100
26110
26120
26130
26140
26150
26160
26170
26180
26190
"optimal_sectors": 2048,
"pi_prot_format": 0,
"pi_prot_type": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0
},
"dev": "/dev/vg1/lv-lun3",
"name": "lun3",
"plugin": "block",
"readonly": false,
"write_back": false,
"wwn": "33ef34a3-b07f-4876-baf8-e71fdc525e04"
},
{
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 0,
"emulate_fua_read": 0,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 0,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 65535,
"optimal_sectors": 2048,
"pi_prot_format": 0,
"pi_prot_type": 0,
"queue_depth": 128,
"unmap_granularity": 0,
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 67 26200
26210
26220
26230
26240
26250
26260
26270
26280
26290
26300
26310
26320
26330
26340
26350
26360
26370
26380
26390
26400
26410
26420
26430
26440
26450
26460
26470
26480
26490
26500
26510
26520
26530
26540
26550
26560
26570
26580
26590
"unmap_granularity_alignment": 0
},
"dev": "/dev/vg1/lv-lun2",
"name": "lun2",
"plugin": "block",
"readonly": false,
"write_back": false,
"wwn": "7d7f8d13-78e6-4b0a-aacb-fee219526219"
},
{
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 0,
"emulate_fua_read": 0,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 0,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 65535,
"optimal_sectors": 2048,
"pi_prot_format": 0,
"pi_prot_type": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0
},
"dev": "/dev/vg1/lv-lun1",
"name": "lun1",
"plugin": "block",
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 68 26600
26610
26620
26630
26640
26650
26660
26670
26680
26690
26700
26710
26720
26730
26740
26750
26760
26770
26780
26790
26800
26810
26820
26830
26840
26850
26860
26870
26880
26890
26900
26910
26920
26930
26940
26950
26960
26970
26980
26990
"readonly": false,
"write_back": false,
"wwn": "43ab697b-6d1e-4d18-b7df-7dd1dd272965"
},
{
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 0,
"emulate_fua_read": 0,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 0,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"max_write_same_len": 65535,
"optimal_sectors": 2048,
"pi_prot_format": 0,
"pi_prot_type": 0,
"queue_depth": 128,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0
},
"dev": "/dev/vg1/lv-lun0",
"name": "lun0",
"plugin": "block",
"readonly": false,
"write_back": false,
"wwn": "37a29689-9f83-4f15-9624-471980ea8f45"
}
],
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 69 27000
27010
27020
27030
27040
27050
27060
27070
27080
27090
27100
27110
27120
27130
27140
27150
27160
27170
27180
27190
27200
27210
27220
27230
27240
27250
27260
27270
27280
27290
27300
27310
27320
27330
27340
27350
27360
27370
27380
27390
"targets": [
{
"fabric": "iscsi",
"tpgs": [
{
"attributes": {
"authentication": 0,
"cache_dynamic_acls": 0,
"default_cmdsn_depth": 64,
"default_erl": 0,
"demo_mode_discovery": 1,
"demo_mode_write_protect": 1,
"fabric_prot_type": 0,
"generate_node_acls": 0,
"login_timeout": 15,
"netif_timeout": 2,
"prod_mode_write_protect": 0,
"t10_pi": 0
},
"enable": true,
"luns": [
{
"index": 3,
"storage_object": "/backstores/block/lun3"
},
{
"index": 2,
"storage_object": "/backstores/block/lun2"
},
{
"index": 1,
"storage_object": "/backstores/block/lun1"
},
{
"index": 0,
"storage_object": "/backstores/block/lun0"
}
],
"node_acls": [
{
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 70 27400
27410
27420
27430
27440
27450
27460
27470
27480
27490
27500
27510
27520
27530
27540
27550
27560
27570
27580
27590
27600
27610
27620
27630
27640
27650
27660
27670
27680
27690
27700
27710
27720
27730
27740
27750
27760
27770
27780
27790
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 30,
"nopin_timeout": 15,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_password": "password-user04",
"chap_userid": "iscsiuser04",
"mapped_luns": [
{
"index": 3,
"tpg_lun": 3,
"write_protect": false
},
{
"index": 2,
"tpg_lun": 2,
"write_protect": false
},
{
"index": 1,
"tpg_lun": 1,
"write_protect": false
},
{
"index": 0,
"tpg_lun": 0,
"write_protect": false
}
],
"node_wwn": "iqn.2016-09.com.example:initiator04"
},
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 71 27800
27810
27820
27830
27840
27850
27860
27870
27880
27890
27900
27910
27920
27930
27940
27950
27960
27970
27980
27990
28000
28010
28020
28030
28040
28050
28060
28070
28080
28090
28100
28110
28120
28130
28140
28150
28160
28170
28180
28190
"default_erl": 0,
"nopin_response_timeout": 30,
"nopin_timeout": 15,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_password": "password-user03",
"chap_userid": "iscsiuser03",
"mapped_luns": [
{
"index": 3,
"tpg_lun": 3,
"write_protect": false
},
{
"index": 2,
"tpg_lun": 2,
"write_protect": false
},
{
"index": 1,
"tpg_lun": 1,
"write_protect": false
},
{
"index": 0,
"tpg_lun": 0,
"write_protect": false
}
],
"node_wwn": "iqn.2016-09.com.example:initiator03"
},
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 30,
"nopin_timeout": 15,
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 72 28200
28210
28220
28230
28240
28250
28260
28270
28280
28290
28300
28310
28320
28330
28340
28350
28360
28370
28380
28390
28400
28410
28420
28430
28440
28450
28460
28470
28480
28490
28500
28510
28520
28530
28540
28550
28560
28570
28580
28590
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_password": "password-user02",
"chap_userid": "iscsiuser02",
"mapped_luns": [
{
"index": 3,
"tpg_lun": 3,
"write_protect": false
},
{
"index": 2,
"tpg_lun": 2,
"write_protect": false
},
{
"index": 1,
"tpg_lun": 1,
"write_protect": false
},
{
"index": 0,
"tpg_lun": 0,
"write_protect": false
}
],
"node_wwn": "iqn.2016-09.com.example:initiator02"
},
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 30,
"nopin_timeout": 15,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 73 28600
28610
28620
28630
28640
28650
28660
28670
28680
28690
28700
28710
28720
28730
28740
28750
28760
28770
28780
28790
28800
28810
28820
28830
28840
28850
28860
28870
28880
28890
28900
28910
28920
28930
28940
28950
28960
28970
28980
28990
},
"chap_password": "password-user01",
"chap_userid": "iscsiuser01",
"mapped_luns": [
{
"index": 3,
"tpg_lun": 3,
"write_protect": false
},
{
"index": 2,
"tpg_lun": 2,
"write_protect": false
},
{
"index": 1,
"tpg_lun": 1,
"write_protect": false
},
{
"index": 0,
"tpg_lun": 0,
"write_protect": false
}
],
"node_wwn": "iqn.2016-09.com.example:initiator01"
}
],
"parameters": {
"AuthMethod": "CHAP,None",
"DataDigest": "CRC32C,None",
"DataPDUInOrder": "Yes",
"DataSequenceInOrder": "Yes",
"DefaultTime2Retain": "20",
"DefaultTime2Wait": "2",
"ErrorRecoveryLevel": "0",
"FirstBurstLength": "65536",
"HeaderDigest": "CRC32C,None",
"IFMarkInt": "2048~65535",
"IFMarker": "No",
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 74 29000
29010
29020
29030
29040
29050
29060
29070
29080
29090
29100
29110
29120
29130
29140
29150
29160
29170
29180
29190
29200
29210
29220
29230
29240
29250
29260
29270
29280
29290
29300
29310
29320
29330
29340
29350
29360
29370
29380
29390
"ImmediateData": "Yes",
"InitialR2T": "Yes",
"MaxBurstLength": "262144",
"MaxConnections": "1",
"MaxOutstandingR2T": "1",
"MaxRecvDataSegmentLength": "8192",
"MaxXmitDataSegmentLength": "262144",
"OFMarkInt": "2048~65535",
"OFMarker": "No",
"TargetAlias": "LIO Target"
},
"portals": [
{
"ip_address": "10.110.88.59",
"iser": false,
"port": 3260
}
],
"tag": 1
}
],
"wwn": "iqn.2016-09.com.example:iscsitgt01"
}
]
}
○
Active 機で、LIO の設定をクリアします。
a
sudo targetctl clear
a
sudo targetcli ls /
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 0]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 0]
o- loopback ..................................................... [Targets: 0]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 75 29400 ○
29410
29420
a
29430
29440
29450
a
29460
29470
29480
29490
29500
29510
29520
29530 ○
29540
29550
a
29560
29570 ○
29580
29590
29600
29610
29620
29630
s
29640
29650
29660 ○
29670
29680 a,s
29690
29700 a,s
29710
29720
29730 ○
29740
29750
a
29760
a
29770
29780
29790
Active 機で、DRBD 上の LVM ボリュームグループを非活性化します。
sudo vgchange -a n vg1
0 logical volume(s) in volume group "vg1" now active
sudo lvs
LV
lv-drbd0
lv-lun0
lv-lun1
lv-lun2
lv-lun3
VG
vg0
vg1
vg1
vg1
vg1
Attr
-wi-ao----wi-------wi-------wi-------wi-------
LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
359.98g
71.99g
71.99g
71.99g
71.99g
Active 機で、DRBD リソースを secondary 化（デモート）します。
sudo drbdadm secondary all
Stand-by 機で、DRBD の状態を確認し、「Ctrl + C」を押下してワッチを停止します。
#Every
[Ctrl
2.0s:
+ C]cat /proc/drbd
Sat Oct 29 18:33:24 2016
#version:
[Ctrl +8.4.5
C] (api:1/proto:86-101)
srcversion: 1AEFF755B8BD61B81A0AF27
# 0:
[Ctrl
cs:Connected
+ C]
ro:Secondary/Secondary ds:UpToDate/UpToDate C r----ns:0 nr:100 dw:100 dr:3759976 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Active 機と Stand-by 機で、drbd.service を停止します。
sudo systemctl stop drbd.service
cat /proc/drbd
cat: /proc/drbd: No such file or directory
Active 機で、Corosync の認証を設定し、起動します。
sudo pcs cluster auth iscsitgt01a.example.com iscsitgt01s.example.com 10.110.88.57 10.110.88.58 \
192.168.1.2 192.168.1.3 -u hacluster -p 'password' --force
iscsitgt01s.example.com: Authorized
iscsitgt01a.example.com: Authorized
10.110.88.58: Authorized
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 76 29800
29810
29820
29830
29840
29850
29860
29870
29880
29890
29900
29910
29920
29930
29940
29950
29960
29970
29980
29990
30000
30010
30020
30030
30040
30050
30060
30070
30080
30090
30100
30110
30120
30130
30140
30150
30160
30170
30180
30190
192.168.1.2: Authorized
192.168.1.3: Authorized
10.110.88.57: Authorized
a
sudo cat /var/lib/pcsd/tokens
{
"format_version": 2,
"data_version": 4,
"tokens": {
"10.110.88.57": "77189e9e-3be0-40ce-b81e-3e5e6525e885",
"10.110.88.58": "9e3f4ae9-b15e-49c9-b6ee-eb8c1b91783a",
"192.168.1.2": "53da862f-ad22-445b-8887-add50d385736",
"192.168.1.3": "4f78d9c6-34a4-4486-8ba1-e69f0d4e1257",
"iscsitgt01a.example.com": "002cd1c0-2ab2-4a4c-a1a7-4bf14b61b822",
"iscsitgt01s.example.com": "1a9981a9-04e6-461b-b904-c5df8b4c9815"
}
}
a
a
sudo pcs cluster setup --name iscsitgt01 10.110.88.57,192.168.1.2 10.110.88.58,192.168.1.3 \
--transport=udp --rrpmode=passive -u hacluster -p 'password' --force
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
10.110.88.57: Succeeded
10.110.88.58: Succeeded
Synchronizing pcsd certificates on nodes 10.110.88.57, 10.110.88.58...
10.110.88.57: Success
10.110.88.58: Success
Restaring pcsd on the nodes in order to reload the certificates...
10.110.88.57: Success
10.110.88.58: Success
a
cat /etc/corosync/corosync.conf
totem {
version: 2
secauth: off
cluster_name: iscsitgt01
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 77 30200
30210
30220
30230
30240
30250
30260
30270
30280
30290
30300
30310
30320
30330
30340
30350
30360
30370
30380
30390
30400
30410
30420
30430
30440
30450
30460
30470
30480
30490
30500
30510
30520
30530
30540
30550
30560
30570
30580
30590
transport: udp
rrp_mode: passive
}
nodelist {
node {
ring0_addr: 10.110.88.57
ring1_addr: 192.168.1.2
nodeid: 1
}
node {
ring0_addr: 10.110.88.58
ring1_addr: 192.168.1.3
nodeid: 2
}
}
quorum {
provider: corosync_votequorum
two_node: 1
}
logging {
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: yes
}
a
sudo pcs cluster start --all
10.110.88.57: Starting Cluster...
10.110.88.58: Starting Cluster...
a
sudo pcs status corosync
Membership information
---------------------Nodeid
Votes Name
1
1 10.110.88.57 (local)
2
1 10.110.88.58
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 78 30600
30610
30620
30630
30640
30650
30660
30670
30680
30690
30700
30710
30720
30730
30740
30750
30760
30770
30780
30790
30800
30810
30820
30830
30840
30850
30860
30870
30880
30890
30900
30910
30920
30930
30940
30950
30960
30970
30980
30990
a
sudo pcs status
Cluster name: iscsitgt01
WARNING: no stonith devices and stonith-enabled is not false
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Tue Oct 29 18:33:24 2016
Last change: Tue Oct 29 18:33:24 2016 by hacluster via crmd on iscsitgt01a.example.com
Stack: corosync
Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 0 resources configured
Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ]
No resources
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
※　「Current DC」が表示されるまで、何回か実行します。20秒以上かかるものと思われます。
※　「Current DC」については、どちらが選ばれていてもあまり意味のある情報ではないので気にしないでください。
※　「WARNING」について、前者は後で対応します。後者は pcs のバグ（RRP mode 未対応）なので無視してください。
○
Active 機と Stand-by 機で、Corosync の状態とプロセスを確認します。
a
sudo corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
id
= 10.110.88.57
status = ring 0 active with no faults
RING ID 1
id
= 192.168.1.2
status = ring 1 active with no faults
s
sudo corosync-cfgtool -s
Printing ring status.
Local node ID 2
RING ID 0
id
= 10.110.88.58
status = ring 0 active with no faults
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 79 31000
31010
31020
31030
31040 a,s
31050
31060
31070
31080
31090
31100
31110
31120
31130
31140 ○
31150
31160
a
31170
31180
31190 ○
31200
31210
a
31220
31230
31240
31250
31260
31270
31280
31290
31300
31310
31320
31330
31340
31350
31360
31370
31380
31390
RING ID 1
id
status
ps -ef |
root
root
haclust+
root
root
haclust+
haclust+
haclust+
= 192.168.1.3
= ring 1 active with no faults
egrep '[c]orosync|[p]acemaker'
38502
1 0 19:49 ?
38517
1 0 19:49 ?
38518 38517 0 19:49 ?
38519 38517 0 19:49 ?
38520 38517 0 19:49 ?
38521 38517 0 19:49 ?
38522 38517 0 19:49 ?
38523 38517 0 19:49 ?
00:00:00
00:00:00
00:00:00
00:00:00
00:00:00
00:00:00
00:00:00
00:00:00
corosync
/usr/sbin/pacemakerd -f
/usr/libexec/pacemaker/cib
/usr/libexec/pacemaker/stonithd
/usr/libexec/pacemaker/lrmd
/usr/libexec/pacemaker/attrd
/usr/libexec/pacemaker/pengine
/usr/libexec/pacemaker/crmd
Active 機で、クラスタにリソースを登録します。
sudo /etc/ha.d/crm.sh
Adding ms_drbd_r0 p_lvm (kind: Mandatory) (Options: first-action=promote then-action=start)
Active 機で、状態を確認します。
sudo pcs status
Cluster name: iscsitgt01
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by root via cibadmin on iscsitgt01s.example.com
Stack: corosync
Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 5 resources configured
Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ]
Full list of resources:
Master/Slave Set: ms_drbd_r0 [p_drbd_r0]
p_drbd_r0 (ocf::linbit:drbd):
FAILED iscsitgt01a.example.com (unmanaged)
p_drbd_r0 (ocf::linbit:drbd):
FAILED iscsitgt01s.example.com (unmanaged)
Resource Group: g_tgt
p_lvm
(ocf::heartbeat:LVM): Stopped
p_lio
(ocf::heartbeat:LIO): Stopped
p_vip
(ocf::heartbeat:VIP): Stopped
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 80 31400
31410
31420
31430
31440
31450
31460
31470
31480
31490
31500
31510
31520
31530
31540
31550
31560
31570
31580
31590
31600
31610
31620
31630
31640
31650
31660
31670
31680
31690
31700
31710
31720
31730
31740
31750
31760
31770
31780
31790
Failed Actions:
* p_drbd_r0_stop_0 on iscsitgt01a.example.com 'not configured' (6): call=6, status=complete, exitreason='none',
last-rc-change='Sat Oct 29 18:33:24 2016', queued=0ms, exec=24ms
* p_lvm_start_0 on iscsitgt01a.example.com 'unknown error' (1): call=11, status=complete, exitreason='Volume group
[vg1] does not exist or contains error!
Volume group "vg1" not found',
last-rc-change='Sat Oct 29 18:33:24 2016', queued=0ms, exec=101ms
* p_drbd_r0_stop_0 on iscsitgt01s.example.com 'not configured' (6): call=6, status=complete, exitreason='none',
last-rc-change='Sat Oct 29 18:33:24 2016', queued=0ms, exec=23ms
* p_lvm_start_0 on iscsitgt01s.example.com 'unknown error' (1): call=15, status=complete, exitreason='Volume group
[vg1] does not exist or contains error!
Volume group "vg1" not found',
last-rc-change='Tue Oct 29 18:33:24 2016', queued=0ms, exec=190ms
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
○
a
○
a
Active 機で、リソースのエラー情報をクリアします。
sudo pcs resource cleanup
Waiting for 1 replies from the CRMd. OK
Active 機で、状態を確認します。
sudo pcs status
Cluster name: iscsitgt01
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by hacluster via crmd on iscsitgt01a.example.com
Stack: corosync
Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 5 resources configured
Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ]
Full list of resources:
Master/Slave Set: ms_drbd_r0 [p_drbd_r0]
Masters: [ iscsitgt01a.example.com ]
Slaves: [ iscsitgt01s.example.com ]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 81 31800
31810
31820
31830
31840
31850
31860
31870
31880
31890
31900 ○
31910
31920 a,s
31930
31940
31950
31960
31970
31980
31990
32000
32010
32020
32030
32040
32050
32060
32070
32080
32090
32100
32110
32120
32130
32140
32150
32160
32170
32180
32190
Resource Group: g_tgt
p_lvm
(ocf::heartbeat:LVM):
p_lio
(ocf::heartbeat:LIO):
p_vip
(ocf::heartbeat:VIP):
Started iscsitgt01a.example.com
Started iscsitgt01a.example.com
Started iscsitgt01a.example.com
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
Active 機と Stand-by 機で、設定情報を保存します。
sudo pcs config | sudo tee /etc/ha.d/crm.conf
Cluster Name: iscsitgt01
Corosync Nodes:
10.110.88.57 10.110.88.58
Pacemaker Nodes:
iscsitgt01a.example.com iscsitgt01s.example.com
Resources:
Master: ms_drbd_r0
Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true target-role=Started is-managed=true
Resource: p_drbd_r0 (class=ocf provider=linbit type=drbd)
Attributes: drbd_resource=r0
Operations: start interval=0s timeout=240 (p_drbd_r0-start-interval-0s)
monitor interval=10 role=Master timeout=20 (p_drbd_r0-monitor-interval-10)
monitor interval=20 role=Slave timeout=20 (p_drbd_r0-monitor-interval-20)
notify interval=0s timeout=90 (p_drbd_r0-notify-interval-0s)
stop interval=0s timeout=100 (p_drbd_r0-stop-interval-0s)
promote interval=0s timeout=90 (p_drbd_r0-promote-interval-0s)
demote interval=0s timeout=90 (p_drbd_r0-demote-interval-0s)
Group: g_tgt
Resource: p_lvm (class=ocf provider=heartbeat type=LVM)
Attributes: volgrpname=vg1
Operations: start interval=0s timeout=30 (p_lvm-start-interval-0s)
monitor interval=10 timeout=30 (p_lvm-monitor-interval-10)
stop interval=0s timeout=30 (p_lvm-stop-interval-0s)
Resource: p_lio (class=ocf provider=heartbeat type=LIO)
Attributes: iqn=iqn.2016-09.com.example:iscsitgt01
Operations: start interval=0s timeout=10 (p_lio-start-interval-0s)
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 82 32200
32210
32220
32230
32240
32250
32260
32270
32280
32290
32300
32310
32320
32330
32340
32350
32360
32370
32380
32390
32400
32410
32420
32430
32440
32450
32460
32470
32480
32490
32500
32510
32520
32530
32540
32550
32560
32570
32580
32590
monitor interval=10 timeout=10 (p_lio-monitor-interval-10)
stop interval=0s timeout=10 (p_lio-stop-interval-0s)
Resource: p_vip (class=ocf provider=heartbeat type=VIP)
Attributes: ip=10.110.88.59 cidr_netmask=26 nic=bond0 iflabel=1 arp_interval=200 arp_count=5
Operations: start interval=0s timeout=20 (p_vip-start-interval-0s)
monitor interval=10 timeout=20 (p_vip-monitor-interval-10)
stop interval=0s timeout=20 (p_vip-stop-interval-0s)
Stonith Devices:
Fencing Levels:
Location Constraints:
Resource: g_tgt
Enabled on: iscsitgt01a.example.com (score:100) (id:lc_tgt)
Ordering Constraints:
promote ms_drbd_r0 then start p_lvm (kind:Mandatory) (id:order-ms_drbd_r0-p_lvm-mandatory)
Colocation Constraints:
g_tgt with ms_drbd_r0 (score:INFINITY) (with-rsc-role:Master) (id:colocation-g_tgt-ms_drbd_r0-INFINITY)
Resources Defaults:
resource-stickiness: 200
migration-threshold: 2
Operations Defaults:
No defaults set
Cluster Properties:
batch-limit: 30
cluster-delay: 60
cluster-infrastructure: corosync
cluster-name: iscsitgt01
cluster-recheck-interval: 15min
crmd-finalization-timeout: 30min
crmd-integration-timeout: 3min
crmd-transition-delay: 0s
dc-deadtime: 20s
dc-version: 1.1.13-10.el7-44eb2dd
default-action-timeout: 20
election-timeout: 2min
enable-acl: true
enable-startup-probes: true
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 83 32600
32610
32620
32630
32640
32650
32660
32670
32680
32690
32700
32710
32720
32730
32740
32750
32760
32770
32780
32790
32800
32810
32820
32830
32840
32850
32860
32870
32880 a,s
32890
32900 ○
32910
32920
a
32930
32940
32950
32960
32970
32980 ○
32990
have-watchdog: false
is-managed-default: true
load-threshold: 80%
maintenance-mode: false
migration-limit: -1
no-quorum-policy: ignore
node-action-limit: 0
node-health-green: 0
node-health-red: -INFINITY
node-health-strategy: none
node-health-yellow: 0
notification-agent: /dev/null
pe-error-series-max: 100
pe-input-series-max: 100
pe-warn-series-max: 100
placement-strategy: default
remove-after-stop: false
shutdown-escalation: 20min
start-failure-is-fatal: true
startup-fencing: true
stonith-action: reboot
stonith-enabled: false
stonith-timeout: 60
stop-all-resources: false
stop-orphan-actions: true
stop-orphan-resources: true
symmetric-cluster: true
sudo cp -a /etc{,~}/ha.d/crm.conf
Active 機で、スイッチオーバ（手動フェイルオーバ）させます。
sudo pcs resource move g_tgt
Warning: Creating location constraint cli-ban-g_tgt-on-iscsitgt01a.example.com with a score of -INFINITY for resource g_tgt on
node iscsitgt01a.example.com.
This will prevent g_tgt from running on iscsitgt01a.example.com until the constraint is removed. This will be the case even if
iscsitgt01a.example.com is the last node in the cluster.
Active 機で、状態を確認します。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 84 33000
33010
33020
33030
33040
33050
33060
33070
33080
33090
33100
33110
33120
33130
33140
33150
33160
33170
33180
33190
33200
33210
33220
33230
33240
33250
33260
33270
33280
33290
33300
33310
33320
33330
33340
33350
33360
33370
33380
33390
a
sudo pcs status
Cluster name: iscsitgt01
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by root via crm_resource on iscsitgt01a.example.com
Stack: corosync
Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 5 resources configured
Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ]
Full list of resources:
Master/Slave Set: ms_drbd_r0 [p_drbd_r0]
Masters: [ iscsitgt01s.example.com ]
Slaves: [ iscsitgt01a.example.com ]
Resource Group: g_tgt
p_lvm
(ocf::heartbeat:LVM): Started iscsitgt01s.example.com
p_lio
(ocf::heartbeat:LIO): Started iscsitgt01s.example.com
p_vip
(ocf::heartbeat:VIP):
Started iscsitgt01s.example.com
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
※　「p_vip」のノードが変わるまで、何回か実行します。
○
a
○
a
a
○
Active 機で、設定変更を確認します。
diff <(grep -v last-lrm-refresh /etc/ha.d/crm.conf) <(sudo pcs config | grep -v last-lrm-refresh)
41a42
>
Disabled on: iscsitgt01a.example.com (score:-INFINITY) (role: Started) (id:cli-ban-g_tgt-on-iscsitgt01a.example.com)
Active 機で、設定変更を元に戻します。
sudo pcs resource clear g_tgt
diff <(grep -v last-lrm-refresh /etc/ha.d/crm.conf) <(sudo pcs config | grep -v last-lrm-refresh)
Active 機でリソースが起動した状態でない場合のみ、スイッチバック（フェイルバック）させます。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 85 33400
a
33410
33420
33430
33440
33450
33460 ○
33470
33480
a
33490
33500
33510
33520
33530
33540
33550
33560
33570
33580
33590
33600
33610
33620
33630
33640
33650
33660
33670
33680
33690
33700
33710
33720
33730 ○
33740
33750
a
33760
33770 ○
33780
33790 a,s
sudo pcs resource move g_tgt; sleep 5; sudo pcs resource clear g_tgt
Warning: Creating location constraint cli-ban-g_tgt-on-iscsitgt01s.example.com with a score of -INFINITY for resource g_tgt on
node iscsitgt01s.example.com.
This will prevent g_tgt from running on iscsitgt01s.example.com until the constraint is removed. This will be the case even if
iscsitgt01s.example.com is the last node in the cluster.
Active 機で、状態を確認します。
sudo pcs status
Cluster name: iscsitgt01
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Tue Oct 29 18:33:24 2016
Last change: Tue Oct 29 18:33:24 2016 by root via crm_resource on iscsitgt01a.example.com
Stack: corosync
Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 6 resources configured
Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ]
Full list of resources:
Master/Slave Set: ms_drbd_r0 [p_drbd_r0]
Masters: [ iscsitgt01a.example.com ]
Slaves: [ iscsitgt01s.example.com ]
Resource Group: g_tgt
p_lvm
(ocf::heartbeat:LVM): Started iscsitgt01a.example.com
p_lio
(ocf::heartbeat:LIO): Started iscsitgt01a.example.com
p_vip
(ocf::heartbeat:VIP):
Started iscsitgt01a.example.com
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
Active 機で、設定変更を確認します。
diff <(grep -v last-lrm-refresh /etc/ha.d/crm.conf) <(sudo pcs config | grep -v last-lrm-refresh)
Active 機と Stand-by 機で、状態を記録します。
sudo pcs status | sudo tee /etc/ha.d/crm.status
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 86 33800
33810
33820
33830
33840
33850
33860
33870
33880
33890
33900
33910
33920
33930
33940
33950
33960
33970
33980
33990
34000
34010
34020
34030
34040 a,s
34050
34060 ○
34070
34080
a
34090
34100
34110
34120
34130
34140 ○
34150
34160 a,s
34170
34180 ○
34190
Cluster name: iscsitgt01
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by root via crm_resource on iscsitgt01a.example.com
Stack: corosync
Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 6 resources configured
Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ]
Full list of resources:
Master/Slave Set: ms_drbd_r0 [p_drbd_r0]
Masters: [ iscsitgt01a.example.com ]
Slaves: [ iscsitgt01s.example.com ]
Resource Group: g_tgt
p_lvm
(ocf::heartbeat:LVM): Started iscsitgt01a.example.com
p_lio
(ocf::heartbeat:LIO): Started iscsitgt01a.example.com
p_vip
(ocf::heartbeat:VIP):
Started iscsitgt01a.example.com
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sudo cp -a /etc{,~}/ha.d/crm.status
Active 機で、クラスタを停止します。
sudo pcs cluster stop --all
10.110.88.57: Stopping Cluster
10.110.88.58: Stopping Cluster
10.110.88.58: Stopping Cluster
10.110.88.57: Stopping Cluster
(pacemaker)...
(pacemaker)...
(corosync)...
(corosync)...
Active 機と Stand-by 機で、再起動します。
sudo reboot
Active 機と Stand-by 機へ、管理者用一般ユーザにて、ssh でログインします。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 87 34200
34210
34220
34230
34240
34250
34260
34270
34280
34290
34300
34310
34320
34330
34340
34350
34360
34370
34380
34390
34400
34410
34420
34430
34440
34450
34460
34470
34480
34490
34500
34510
34520
34530
34540
34550
34560
34570
34580
34590
a
a
ssh [email protected]
#[email protected]'s
********
password: ********
s
s
ssh [email protected]
#[email protected]'s
********
password: ********
○
a
○
a
Active 機で、クラスタを起動します。
sudo pcs cluster start --all
10.110.88.57: Starting Cluster...
10.110.88.58: Starting Cluster...
Active 機で、状態を確認します。
sudo pcs status
Cluster name: iscsitgt01
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by root via crm_resource on iscsitgt01a.example.com
Stack: corosync
Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 6 resources configured
Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ]
Full list of resources:
Master/Slave Set: ms_drbd_r0 [p_drbd_r0]
Masters: [ iscsitgt01a.example.com ]
Slaves: [ iscsitgt01s.example.com ]
Resource Group: g_tgt
p_lvm
(ocf::heartbeat:LVM): Started iscsitgt01a.example.com
p_lio
(ocf::heartbeat:LIO): Started iscsitgt01a.example.com
p_vip
(ocf::heartbeat:VIP):
Started iscsitgt01a.example.com
PCSD Status:
iscsitgt01a.example.com (10.110.88.57): Online
iscsitgt01s.example.com (10.110.88.58): Online
Daemon Status:
corosync: active/disabled
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 88 34600
34610
34620
pacemaker: active/disabled
pcsd: active/enabled
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 89 -
当文書で紹介した構成で初期構築をご希望の方は、メール(mailto: [email protected])にてお問い合わせください。
リモートアクセスによる構築サービスを、弊社提示のヒアリング事項（当文書の太緑で記載した部分）以外のカスタマイズ無し前提で、
サーバ1台あたり10万円（税別）という超特価にてご提供中です。
1号機と2号機(Active 機と Stand-by 機)サーバの物理構成は同一構成との前提です。
カスタマイズやドキュメントの提供、個別訪問等は、別途ご相談となります。
クラスタの起動停止、スイッチオーバ(手動フェイルオーバ)以外の動作確認、結合試験、障害試験、性能試験等は、別途ご相談となります。
インストールメディアからインストールできない環境やコンソールにリモートアクセスできない環境でのインストールも別途ご相談となります。
監視設定（障害監視、リソース監視、セキュリティ監視等）については、別途ご相談となります。
UEFI 対応、ハードウェア固有のドライバや管理ソフト等のインストールについては、別途ご相談となります。
当該サーバには、消失したら困るデータは存在していない前提での作業となります。
既に動いている CentOS を置き換えるインストールの場合、ヒアリング事項を弊社で調査して提示することも可能です。
例えば、IBM Bluemix(SoftLayer) のベアメタルサーバの場合、CentOS7をあらかじめインストールした状態で弊社へお引き渡し頂ければ、
お客様の手間を省くことができます。
弊社での正式サポートは、お客様が当該サーバ用の Oracle Linux サブスクリプションを契約中か NRI OpenStandia に相談窓口をお持ちで、
代理で問い合わせを行うという前提を取らさせていただくことになります。費用はご相談ください。
サーバの調達・CE作業費用、OS 等のライセンス費用、サブスクリプション費用等はすべて別料金となります。
OS を Oracle Linux (UEK カーネル) に置き換えずに、RHEL・CentOS で構築することも可能ですが、制限事項についてご相談ください。
MySQL や PostgreSQL、Oracle の冗長構成構築サービスも鋭意開発中です。商品開発に関するリクエストがあればお知らせください。
検討させていただきます。
貴社のアプリケーション、サービス等を冗長化する共同開発も承ります。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.

みなさんこんにちは

RoboCup 3D シミュレーションの環境構築方法(Ubuntu)

ハンズオン -‐‑‒ 準備編

「Rubyインストールマニュアル」（PDF:273KB）

セミナー情報更新日 2013.11.29

.com/.net/.org Authコード/その他キー発行申込書 0 N H 0 8 0 (.com

drbd_proxy_wan

～SIESTAのインストール方法～