Pacemaker : Set Fence Device2024/07/23

Set Fence Device on Cluster. (see about Fencing on the site below)

https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/s1-fencing-haao

It's possible to use many kinds of devices for fencing, APC or IPMI and so on.
On this example, Configure storage based fencing by using SCSI storage like follows.

Configure basic Cluster settings first, refer to here.


                        +--------------------+
                        | [  ISCSI Target  ] |
                        |    dlp.srv.world   |
                        +---------+----------+
                         10.0.0.30|
                                  |
+----------------------+          |          +----------------------+
| [  Cluster Node#1  ] |10.0.0.51 | 10.0.0.52| [  Cluster Node#2  ] |
|   node01.srv.world   +----------+----------+   node02.srv.world   |
|                      |                     |                      |
+----------------------+                     +----------------------+

[1]	Configure ISCSI Target and Create a storage for fence device, refer to here. On this example, it created ISCSI storage as IQN [iqn.2024-04.world.srv:dlp.target01] with [1M] size.
[2]	On all Cluster Nodes, Login to ISCSI Target, refer to here.
[3]	On all Cluster Nodes, Install SCSI Fence Agent.

root@node01:~#

apt -y install resource-agents-base watchdog

root@node01:~#

mkdir /etc/watchdog.d

root@node01:~#

cp /usr/share/cluster/fence_scsi_check /etc/watchdog.d/

root@node01:~#

systemctl stop watchdog.service

root@node01:~#

systemctl start watchdog.service

[4]	Configure Fencing on a Node. [sda] of the example below is the storage from ISCSI target.

# confirm disk ID

root@node01:~#

ll /dev/disk/by-id | grep sda | grep wwn

lrwxrwxrwx 1 root root   9 Jul 23 00:08 wwn-0x6001405fd08aa7cd2fe4f8cad7b28412 -> ../../sda

# set fencing
# [scsi-shooter] : any name
# [pcmk_host_list=***] : specify cluster nodes
# [devices=***] : disk ID

root@node01:~#

pcs stonith create scsi-shooter fence_scsi pcmk_host_list="node01.srv.world node02.srv.world" devices=/dev/disk/by-id/wwn-0x6001405fd08aa7cd2fe4f8cad7b28412 meta provides=unfencing

# show config

root@node01:~#

pcs stonith config scsi-shooter

Resource: scsi-shooter (class=stonith type=fence_scsi)
  Attributes: scsi-shooter-instance_attributes
    devices=/dev/disk/by-id/wwn-0x6001405fd08aa7cd2fe4f8cad7b28412
    pcmk_host_list="node01.srv.world node02.srv.world"
  Meta Attributes: scsi-shooter-meta_attributes
    provides=unfencing
  Operations:
    monitor: scsi-shooter-monitor-interval-60s
      interval=60s

# show status
# OK if the status of fence device is [Started]

root@node01:~#

pcs status

Cluster name: ha_cluster
Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: node01.srv.world (version 2.1.6-6fdc9deea29) - partition with quorum
  * Last updated: Tue Jul 23 00:13:10 2024 on node01.srv.world
  * Last change:  Tue Jul 23 00:12:36 2024 by root via cibadmin on node01.srv.world
  * 2 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node01.srv.world node02.srv.world ]

Full List of Resources:
  * scsi-shooter        (stonith:fence_scsi):    Started node01.srv.world

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

[5]	Try to test fencing.

root@node02:~#

pcs status

Cluster name: ha_cluster
Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: node01.srv.world (version 2.1.6-6fdc9deea29) - partition with quorum
  * Last updated: Tue Jul 23 00:13:58 2024 on node02.srv.world
  * Last change:  Tue Jul 23 00:12:36 2024 by root via cibadmin on node01.srv.world
  * 2 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node01.srv.world node02.srv.world ]

Full List of Resources:
  * scsi-shooter        (stonith:fence_scsi):    Started node01.srv.world

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

# fencing

root@node02:~#

pcs stonith fence node01.srv.world

Node: node01.srv.world fenced

# target node turns to [OFFLINE] and it will be restarted

root@node02:~#

pcs status

Cluster name: ha_cluster
Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: node02.srv.world (version 2.1.6-6fdc9deea29) - partition with quorum
  * Last updated: Tue Jul 23 00:14:35 2024 on node02.srv.world
  * Last change:  Tue Jul 23 00:12:36 2024 by root via cibadmin on node01.srv.world
  * 2 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node02.srv.world ]
  * OFFLINE: [ node01.srv.world ]

Full List of Resources:
  * scsi-shooter        (stonith:fence_scsi):    Started node02.srv.world

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

# after rebooting, if you manually start the node, do like follows

root@node02:~#

pcs cluster start node01.srv.world

Matched Content