一、环境
nfsserver01：192.168.127.101 心跳：192.168.42.101 centos7.3
nfsserver02：192.168.127.102 心跳：192.168.42.102 centos7.3
VIP:192.168.127.103
nfsclient01：192.168.127.100 centos7.3
二、原理
1、DRDB，分布式复制块设备（Distributed Replicated Block Device），基于linux的软件组件；primary负责接受写入的数据，并且把写入的数据发送给Secondary端。
支持底层设备：磁盘/分区、LVM逻辑卷、soft raid设备等。
复制模式：
协议A：异步复制协议。一旦本地磁盘写入已经完成，数据包已在发送队列中，则写被认为是完成的，在一个节点发生故障时，可能发生数据丢失，因为被写入到远程节点上的数据可能仍在发送队列尽管，在故障转移节点上的数据是一致的，但没有及时更新，这通常用于地理上分开的点。
协议B：内存同步（半同步）复制协议。一旦本地磁盘写入已完成，且复制数据包达到了对等节点则认为写在主节点上被认为是完成的，数据丢失可能发生在参加的两个节点同时故障的情况下，因为在传输中的数据可能不会被提交到磁盘。
协议C：同步复制协议。只有在本地和远程节点的磁盘已经确认了写操作完成，写才被认为完成。没有任何数据丢失，所以这是一个群集节点的流行模式，但I/0吞吐量依赖于网络带宽。
三、安装
1、DRDB
下载地址：
nfsserver01 [root@nfsserver01 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.127.101 nfsserver01 192.168.127.102 nfsserver02 [root@nfsserver01 ~]# fdisk -l|grep sdb 磁盘 /dev/sdb：1073 MB, 1073741824 字节，2097152 个扇区 [root@nfsserver01 ~]# crontab -e*/5 * * * * ntpdate cn.pool.ntp.org [root@nfsserver01 ~]# crontab -l */5 * * * * ntpdate cn.pool.ntp.org [root@node01 ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org [root@node01 ~]# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm ####配置elrepo源 获取http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm 获取http://elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm 准备中... ################################# [100%] 正在升级/安装... 1:elrepo-release-7.0-3.el7.elrepo ################################# [100%] [root@node01 ~]# yum install -y kmod-drbd84 drbd84-utils kernel* ####安装drbd,完成后重启 [root@nfsserver01 ~]# cat /etc/drbd.conf # You can find an example in /usr/share/doc/drbd.../drbd.conf.example include "drbd.d/global_common.conf"; include "drbd.d/*.res"; [root@nfsserver01 ~]# cp /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.ori ####全局配置文件 您在 /var/spool/mail/root 中有新邮件 [root@nfsserver01 ~]# vim /etc/drbd.d/global_common.conf # DRBD is the result of over a decade of development by LINBIT. # In case you need professional services for DRBD or have # feature requests visit http://www.linbit.com global { usage-count no; # Decide what kind of udev symlinks you want for "implicit" volumes # (those without explicit volume {} block, implied vnr=0): # /dev/drbd/by-resource/ / (explicit volumes) # /dev/drbd/by-resource/ (default for implict) udev-always-use-vnr; # treat implicit the same as explicit volumes # minor-count dialog-refresh disable-ip-verification # cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600; } common { protocol C; handlers { # These are EXAMPLE handlers only. # They may have severe implications, # like hard resetting the node under certain circumstances. # Be careful when choosing your poison. pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f"; # fence-peer "/usr/lib/drbd/crm-fence-peer.sh"; # split-brain "/usr/lib/drbd/notify-split-brain.sh root"; # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root"; # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k"; # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh; # quorum-lost "/usr/lib/drbd/notify-quorum-lost.sh root"; } startup { # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb } options { # cpu-mask on-no-data-accessible # RECOMMENDED for three or more storage nodes with DRBD 9: # quorum majority; # on-no-quorum suspend-io | io-error; } disk { on-io-error detach; # size on-io-error fencing disk-barrier disk-flushes # disk-drain md-flushes resync-rate resync-after al-extents # c-plan-ahead c-delay-target c-fill-target c-max-rate # c-min-rate disk-timeout } net { # protocol timeout max-epoch-size max-buffers # connect-int ping-int sndbuf-size rcvbuf-size ko-count # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri # after-sb-1pri after-sb-2pri always-asbp rr-conflict # ping-timeout data-integrity-alg tcp-cork on-congestion # congestion-fill congestion-extents csums-alg verify-alg # use-rle } syncer { rate 1024M; } } [root@nfsserver01 ~]# vim /etc/drbd.d/nfsserver.res resource nfsserver { protocol C; meta-disk internal; device /dev/drbd1; syncer { verify-alg sha1; } net { allow-two-primaries; } on nfsserver01 { disk /dev/sdb; address 192.168.127.101:7789; } on nfsserver02 { disk /dev/sdb; address 192.168.127.102:7789; } } [root@nfsserver01 ~]# scp -rp /etc/drbd.d/* nfsserver02:/etc/drbd.d/ The authenticity of host 'nfsserver02 (192.168.127.102)' can't be established. ECDSA key fingerprint is 47:2b:8b:e2:56:38:a5:45:6f:1c:c7:62:5b:de:9d:20. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'nfsserver02,192.168.127.102' (ECDSA) to the list of known hosts. root@nfsserver02's password: global_common.conf 100% 2621 2.6KB/s 00:00 global_common.conf.ori 100% 2563 2.5KB/s 00:00 nfsserver.res 100% 261 0.3KB/s 00:00 [root@nfsserver01 ~]# drbdadm create-md nfsserver #####初始化 initializing activity log initializing bitmap (32 KB) to all zero Writing meta data... New drbd meta data block successfully created [root@nfsserver01 /]# modprobe drbd #####加载到内核 [root@nfsserver01 /]# drbdadm up nfsserver ###启用nfsserver资源组 [root@nfsserver01 /]# drbdadm primary nfsserver --force ###将nfsserver01设为主节点 [root@nfsserver01 /]# systemctl start drbd [root@nfsserver01 /]# systemctl enable drbd Created symlink from /etc/systemd/system/multi-user.target.wants/drbd.service to /usr/lib/systemd/system/drbd.service. [root@nfsserver01 /]# drbdadm cstate nfsserver Connected [root@nfsserver01 /]# drbdadm role nfsserver Primary [root@nfsserver01 /]# drbdadm dstate nfsserver UpToDate/UpToDate ######################################### 本地和对等节点的硬盘有可能为下列状态之一：
注：
Diskless 无盘：本地没有块设备分配给DRBD使用，这表示没有可用的设备，或者使用drbdadm命令手工分离或是底层的I/O错误导致自动分离
Attaching：读取无数据时候的瞬间状态
Failed 失败：本地块设备报告I/O错误的下一个状态，其下一个状态为Diskless无盘
Negotiating：在已经连接的DRBD设置进行Attach读取无数据前的瞬间状态
Inconsistent：数据是不一致的，在两个节点上（初始的完全同步前）这种状态出现后立即创建一个新的资源。此外，在同步期间（同步目标）在一个节点上出现这种状态
Outdated：数据资源是一致的，但是已经过时
DUnknown：当对等节点网络连接不可用时出现这种状态
Consistent：一个没有连接的节点数据一致，当建立连接时，它决定数据是UpToDate或是Outdated
UpToDate：一致的最新的数据状态，这个状态为正常状态
#############################################
[root@nfsserver01 /]# drbd-overview NOTE: drbd-overview will be deprecated soon. Please consider using drbdtop. 1:nfsserver/0 Connected(2*) Primar/Second UpToDa/UpToDa /share xfs 1021M 33M 989M 4%
#######在nfsserver02上做同样操作，但是不要设置为主节点，默认都为从节点
[root@nfsserver01 ~]# mkfs.xfs /dev/drbd1 meta-data=/dev/drbd1 isize=512 agcount=4, agsize=65532 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=262127, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=855, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 您在 /var/spool/mail/root 中有新邮件 [root@nfsserver01 ~]# mount /dev/drbd1 /share/ [root@nfsserver01 ~]# df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/sda2 18G 1.7G 17G 10% / devtmpfs 99M 0 99M 0% /dev tmpfs 110M 0 110M 0% /dev/shm tmpfs 110M 4.6M 105M 5% /run tmpfs 110M 0 110M 0% /sys/fs/cgroup /dev/sda1 197M 149M 49M 76% /boot tmpfs 22M 0 22M 0% /run/user/0 /dev/drbd1 1021M 33M 989M 4% /share #######验证如下 #####nfsserver01 [root@nfsserver01 /]# cd /share/ [root@nfsserver01 share]# touch {1..9}.txt [root@nfsserver01 share]# ls 1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt [root@nfsserver01 /]# umount /share/ [root@nfsserver01 /]# drbdadm secondary nfsserver #######nfsserver02 [root@nfsserver02 /]# drbdadm primary nfsserver 您在 /var/spool/mail/root 中有新邮件 [root@nfsserver02 /]# mount /dev/drbd1 /share/ [root@nfsserver02 /]# ls /share/ 1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt 2、heartbeat安装
参考：https://www.cnblogs.com/suffergtf/p/9525416.html
########nfsserver01上
[root@nfsserver01 ~]# ssh-keygen [root@nfsserver01 ~]# ssh-copy-id node02 [root@nfsserver01 ~]# yum groupinstall "Development tools" ####安装开发工具 [root@nfsserver01 ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo ####安装epel源 [root@nfsserver01 ~]# yum -y install glib2-devel libtool-ltdl-devel net-snmp-devel bzip2-devel ncurses-devel openssl-devel libtool \ libxml2-devel gettext bison flex zlib-devel mailx which libxslt docbook-dtds docbook-style-xsl PyXML shadow-utils opensp autoconf automake bzip2 [root@nfsserver01 ~]# yum install -y glib2-devel libxml2 libxml2-devel bzip2-devel e2fsprogs-devel libxslt-devel libtool-ltdl-devel \ make wget docbook-dtds docbook-style-xsl bzip2-devel asciidoc libuuid-devel psmisc ####安装heartbeat依赖 [root@nfsserver01 ~]# groupadd haclient [root@nfsserver01 ~]# useradd -g haclient hacluster -M -s /sbin/nologin ###添加heartbeat默认用户和组 [root@nfsserver01 ~]# cd /server/tools/ [root@nfsserver01 tools]# tar -jxf 0a7add1d9996.tar.bz2 [root@nfsserver01 tools]# cd Reusable-Cluster-Components-glue--0a7add1d9996/ [root@nfsserver01 Reusable-Cluster-Components-glue--0a7add1d9996]# ./autogen.sh ... Now run ./configure [root@nfsserver01 Reusable-Cluster-Components-glue--0a7add1d9996]# ./configure --prefix=/application/heartbeat [root@nfsserver01 Reusable-Cluster-Components-glue--0a7add1d9996]# make && make install [root@nfsserver01 Reusable-Cluster-Components-glue--0a7add1d9996]# cd .. [root@nfsserver01 tools]# rz [root@nfsserver01 tools]# tar xf resource-agents-3.9.6.tar.gz [root@nfsserver01 tools]# cd resource-agents-3.9.6 [root@nfsserver01 resource-agents-3.9.6]# ./autogen.sh [root@nfsserver01 resource-agents-3.9.6]# export CFLAGS="$CFLAGS -I/application/heartbeat/include -L/application/heartbeat/lib" [root@nfsserver01 resource-agents-3.9.6]# ./configure --prefix=/application/heartbeat/ [root@nfsserver01 resource-agents-3.9.6]# ln -s /application/heartbeat/lib/* /lib/ [root@nfsserver01 resource-agents-3.9.6]# ln -s /application/heartbeat/lib/* /lib64/ [root@nfsserver01 resource-agents-3.9.6]# make &&make install [root@nfsserver01 tools]# tar -jxf 958e11be8686.tar.bz2 [root@nfsserver01 tools]# cd Heartbeat-3-0-958e11be8686/ [root@nfsserver01 Heartbeat-3-0-958e11be8686]# ./bootstrap [root@nfsserver01 Heartbeat-3-0-958e11be8686]# ./configure --prefix=/application/heartbeat/ [root@nfsserver01 Heartbeat-3-0-958e11be8686]# vim /application/heartbeat/include/heartbeat/glue_config.h #define HA_HBCONF_DIR "/application/heartbeat/etc/ha.d/" #########删除该行（最后） [root@nfsserver01 Heartbeat-3-0-958e11be8686]# make && make install [root@nfsserver01 heartbeat]# cd /application/heartbeat/share/doc/heartbeat/ [root@nfsserver01 heartbeat]# cp -a ha.cf authkeys haresources /application/heartbeat/etc/ha.d/ [root@nfsserver01 heartbeat]# chmod 600 authkeys [root@nfsserver01 heartbeat]# ln -svf /application/heartbeat/lib64/heartbeat/plugins/RAExec/* /application/heartbeat/lib/heartbeat/plugins/RAExec/ "/application/heartbeat/lib/heartbeat/plugins/RAExec/*" -> "/application/heartbeat/lib64/heartbeat/plugins/RAExec/*" [root@nfsserver01 heartbeat]# ln -svf /application/heartbeat/lib64/heartbeat/plugins/* /application/heartbeat/lib/heartbeat/plugins/ "/application/heartbeat/lib/heartbeat/plugins/HBauth" -> "/application/heartbeat/lib64/heartbeat/plugins/HBauth" "/application/heartbeat/lib/heartbeat/plugins/HBcomm" -> "/application/heartbeat/lib64/heartbeat/plugins/HBcomm" "/application/heartbeat/lib/heartbeat/plugins/quorum" -> "/application/heartbeat/lib64/heartbeat/plugins/quorum" "/application/heartbeat/lib/heartbeat/plugins/tiebreaker" -> "/application/heartbeat/lib64/heartbeat/plugins/tiebreaker" [root@nfsserver01 heartbeat]# cd /application/heartbeat/etc/ha.d/ [root@nfsserver01 ha.d]# vim ha.cf debugfile /var/log/ha-debug logfile /var/log/ha-log keepalive 2 deadtime 30 warntime 10 initdead 120 udpport 694 bcast ens37 ucast ens37 192.168.42.102 auto_failback on node nfsserver01 node nfsserver02 ping 192.168.42.101 respawn hacluster /application/heartbeat/libexec/heartbeat/ipfail apiauth ipfail gid=haclient uid=hacluster [root@nfsserver01 heartbeat]# vim authkeys ####主备相同 auth 2 2 sha1 hello! [root@nfsserver01 heartbeat]# vim haresources ######主备相同 nfsserver01 192.168.127.103/24/ens33:0
3、nfs安装
参考：https://www.cnblogs.com/suffergtf/p/9486250.html
########服务端nfsserver01 [root@nfsserver01 ~]# yum install rpcbind nfs-utils -y [root@nfsserver01 ~]# systemctl start rpcbind [root@nfsserver01 ~]# systemctl enable rpcbind [root@nfsserver01 ~]# systemctl start nfs [root@nfsserver01 ~]# systemctl enable nfs Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service. [root@nfsserver01 ~]# vim /etc/exports/share 192.168.127.0/24(rw,async) [root@nfsserver01 ~]# showmount -e localhost Export list for localhost: /share 192.168.127.0/24 [root@nfsserver01 ~]# chown nfsnobody.nfsnobody /share ##########nfsserver02如上 ##########客户端 [root@nfsclient01 ~]# yum install rpcbind nfs-utils -y [root@nfsclient01 ~]# mount -t nfs 192.168.127.103:/share /mnt [root@nfsclient01 ~]# df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/sda2 18G 1.3G 17G 7% / devtmpfs 101M 0 101M 0% /dev tmpfs 111M 0 111M 0% /dev/shm tmpfs 111M 4.6M 106M 5% /run tmpfs 111M 0 111M 0% /sys/fs/cgroup /dev/sda1 197M 94M 104M 48% /boot tmpfs 23M 0 23M 0% /run/user/0 192.168.127.103:/share 18G 2.2G 16G 12% /mnt
4、整合
heartbeat服务 [root@nfsserver01 resource.d]# vim /application/heartbeat/etc/ha.d/haresources nfsserver01 IPaddr::192.168.127.103/24/ens33:0 drbddisk::nfsserver Filesystem::/dev/drbd1::/share::xfs killnfsd
注：该文件内IPaddr,Filesystem等脚本存放路径在安装路径下的/etc/ha.d/resource.d/下，也可在该目录下存放服务启动脚本（例如：mysql,www），将相同脚本名称添加到/etc/ha.d/haresources内容中，从而跟随heartbeat启动而启动该脚本。
IPaddr::192.168.0.190/24/eth0：用IPaddr脚本配置对外服务的浮动虚拟IPdrbddisk::r0：用drbddisk脚本实现DRBD主从节点资源组的挂载和卸载，Filesystem::/dev/drbd1::/share::xfs：用Filesystem脚本实现磁盘挂载和卸载killnfsd这个为控制nfs启动的脚本，自己写
调用脚本请先确定脚本存在于安装路径下的/etc/ha.d/resource.d/
[root@nfsserver01 resource.d]# vim /application/heartbeat/etc/ha.d/resource.d/killnfsd
#!/bin/bash killall -9 nfs;systemctl restart nfs; exit 0nfs服务
客户端挂载
[root@nfsclient01 ~]# mount -t nfs 192.168.127.103:/share /mnt
四、测试
1、关闭nfsserver01，查看客户端是否还能正常访问共享，如果正常，表示DRBD正常切换主从，heartbeat虚拟ip正常漂移。
2、开启nfsserver01后，关闭nfsserver02，查看客户端是否还能正常访问共享