配置集群文件系统：

安装软件包： yum -y install   lvm2* gfs2* dlm*

1.安装rpm包

 yum -y install   lvm2* gfs2* dlm* fence-agents*

2.修改LVM配置文件改为集群模式-----暂时不做

/etc/lvm/lvm.conf

 locking_type = 1 改为 locking_type = 3

[root@qhdb2 lvm]# lvmconf -h

Usage: /usr/sbin/lvmconf <command>

Commands:

Enable clvm:  --enable-cluster [--lockinglibdir <dir>] [--lockinglib <lib>]

Disable clvm: --disable-cluster

Enable halvm: --enable-halvm

Disable halvm: --disable-halvm

Set locking library: --lockinglibdir <dir> [--lockinglib <lib>]

Global options:

Config file location: --file <configfile>

Set services: --services [--mirrorservice] [--startstopservices]

Use the separate command 'lvmconfig' to display configuration information

重启主机生效

------------  创建cluster CLVM必须得创建fence机制

禁止集群投票，防止脑裂（由于我们集群是2个节点）

pcs   property   set no-quorum-policy=freeze

在生产环境中，可以用

[root@qhdb1 ~]# pcs    stonith list

fence_aliyun - Fence agent for Aliyun (Aliyun Web Services)

fence_amt_ws - Fence agent for AMT (WS)

fence_apc - Fence agent for APC over telnet/ssh

fence_apc_snmp - Fence agent for APC, Tripplite PDU over SNMP

fence_aws - Fence agent for AWS (Amazon Web Services)

fence_azure_arm - Fence agent for Azure Resource Manager

fence_bladecenter - Fence agent for IBM BladeCenter

fence_brocade - Fence agent for HP Brocade over telnet/ssh

fence_cisco_mds - Fence agent for Cisco MDS

fence_cisco_ucs - Fence agent for Cisco UCS

fence_compute - Fence agent for the automatic resurrection of OpenStack compute instances

fence_drac5 - Fence agent for Dell DRAC CMC/5

fence_eaton_snmp - Fence agent for Eaton over SNMP

fence_emerson - Fence agent for Emerson over SNMP

fence_eps - Fence agent for ePowerSwitch

fence_evacuate - Fence agent for the automatic resurrection of OpenStack compute instances

fence_gce - Fence agent for GCE (Google Cloud Engine)

fence_heuristics_ping - Fence agent for ping-heuristic based fencing

fence_hpblade - Fence agent for HP BladeSystem

fence_ibmblade - Fence agent for IBM BladeCenter over SNMP

fence_idrac - Fence agent for IPMI

fence_ifmib - Fence agent for IF MIB

fence_ilo - Fence agent for HP iLO

fence_ilo2 - Fence agent for HP iLO

fence_ilo3 - Fence agent for IPMI

fence_ilo3_ssh - Fence agent for HP iLO over SSH

fence_ilo4 - Fence agent for IPMI

fence_ilo4_ssh - Fence agent for HP iLO over SSH

fence_ilo5 - Fence agent for IPMI

fence_ilo5_ssh - Fence agent for HP iLO over SSH

fence_ilo_moonshot - Fence agent for HP Moonshot iLO

fence_ilo_mp - Fence agent for HP iLO MP

fence_ilo_ssh - Fence agent for HP iLO over SSH

fence_imm - Fence agent for IPMI

fence_intelmodular - Fence agent for Intel Modular

fence_ipdu - Fence agent for iPDU over SNMP

fence_ipmilan - Fence agent for IPMI

fence_kdump - fencing agent for use with kdump crash recovery service

fence_lpar - Fence agent for IBM LPAR

fence_mpath - Fence agent for multipath persistent reservation

fence_redfish - I/O Fencing agent for Redfish

fence_rhevm - Fence agent for RHEV-M REST API

fence_rsa - Fence agent for IBM RSA

fence_rsb - I/O Fencing agent for Fujitsu-Siemens RSB

fence_sanlock - Fence agent for watchdog and shared storage

fence_sbd - Fence agent for sbd

fence_scsi - Fence agent for SCSI persistent reservation

fence_virsh - Fence agent for virsh

fence_virt - Fence agent for virtual machines

fence_vmware_rest - Fence agent for VMware REST API

fence_vmware_soap - Fence agent for VMWare over SOAP API

fence_wti - Fence agent for WTI

fence_xvm - Fence agent for virtual machines

在我的生产环境中、我则会优先选择使用fence_mpath作为stonith资源限制

配置：我的存储使用的日立的盘

修改/etc/multipath.conf 配置文件

defaults {

        user_friendly_names yes

        find_multipaths yes

        reservation_key        0x2    依次命名节点1 2 3

}

2节点：

重启multipath

systemctl restart multipathd

multipath

配置stonith设备：

pcs stonith create qhdb1_pcmk fence_mpath key=1 pcmk_host_list="qhdb1" pcmk_reboot_action="off"   devices="/dev/mapper/mpathe,/dev/mapper/mpathf" meta provides=unfencing

pcs stonith create qhdb2_pcmk fence_mpath key=2 pcmk_host_list="qhdb2" pcmk_reboot_action="off"   devices="/dev/mapper/mpathe,/dev/mapper/mpathf" meta provides=unfencing

或者

pcs  stonith create  pg_cluster_st  fence_mpath devices="/dev/mapper/mpathe,/dev/mapper/mpathf"   pcmk_host_map="qhdb1:1;qhdb2:2"  pcmk_monitor_action="metadata" pcmk_reboot_action="off" pcmk_host_argument="key" meta provides=unfencing

禁止集群投票，防止脑裂（由于我们集群是2个节点）

pcs   property   set no-quorum-policy=freeze

添加dlm资源

pcs resource create dlm ocf:pacemaker:controld op monitor interval=30s on-fail=fence clone interleave=true ordered=true

启用集群锁模式的clvm

lvmconf  --enable-cluster    ---

添加clvm资源

pcs resource create clvmd ocf:heartbeat:clvm op monitor interval=30s on-fail=fence clone interleave=true ordered=true

创建dlm资源和clvm资源的依赖关系和启动顺序：

pcs constraint order start dlm-clone then clvmd-clone

pcs constraint colocation add clvmd-clone with dlm-clone

创建文件系统gfs2

pvcreate  /dev/mapper/mpathe   /dev/mapper/mpathf  

vgcreate -Ay -cy  vg01 /dev/mapper/mpathe  /dev/mapper/mpathf

lvcreate  -L +200G -n lv_01   vg01

mkfs.gfs2  -p lock_dlm  -j 2 -t pg_cluster:file1  /dev/vg01/lv_01

添加clusterfs资源：

pcs  resource  create pg01_file1   ocf:heartbeat:Filesystem  device="/dev/vg01/lv_01" directory="/pg01" fstype="gfs2"   "options=noatime" op monitor interval=10s on-fail=fence clone interleave=true

或者？？？？

pcs  resource  create pg01_file1   Filesystem  device="/dev/vg01/lv_01" directory="/pg01" fstype="gfs2"   "options=noatime" op monitor interval=10s on-fail=fence clone interleave=true

创建 gfs2和clvmd的启动顺序和依赖关系：

pcs  constraint order  start  clvmd-clone then  pg01_file1-clone

pcs  constraint colocation add pg01_file1-clone with clvmd-clone

查看文件系统：