目前打算在192.168.6.243和192.168.6.244這兩臺機器上安裝Keepalived(首先需要先安裝好MySQL,配置好Master-Master主主復制),虛擬IP(VIP)是192.168.6.252,在1臺機器上的MySQL出現故障后,VIP自動地漂移到另外1臺機器上。
查看下192.168.6.243的環境:
OS: Linuxmds-Master 2.6.18⑵38.el5 #1 SMP Sun Dec 19 14:22:44 EST 2010 x86_64 x86_64x86_64 GNU/Linux
CPU:8*Intel(R)Xeon(R) CPU E5506 @ 2.13GHz
Mem:32G
Keepalived的相干信息:
Version:1.2.1(注意:不要使用最新版本1.2.2,編譯有毛病)
下載地址:http://www.keepalived.org/download.html
MySQL的相干信息
Version:5.1.47+XtraDB
MySQL_Install_Dictory: /usr/local/mysql
MySQL_Config_File:/home/mysql/mysq/my1.cnf
[mysql@mds-Master tool]$ cd /home/wangwei/tools
[mysql@mds-Master tool]$ tar xvzf keepalived⑴.2.1.tar.gz
[wangwei@mds-Masterkeepalived⑴.2.1]$./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.18⑵38.el5-x86_64
編譯出錯
configure: error:
!!! OpenSSLis not properly installed on your system. !!!
!!! Can notinclude OpenSSL headers files. !!!
說明系統中沒有安裝OpenSSL
那就安裝下OpenSS:L吧,出錯
[mysql@mds-Master tool]# yum install openssl-devel
Loaded plugins: rhnplugin, security
This system is not registered with RHN
緣由是linux沒有在紅帽網絡上注冊,所以沒法下載上面的軟件包,替換方案可使用centos(使用中國科技大學的源 http://mirror.centos.org),具體如何配置我就不寫了。
配置終了后,使用yum安裝openssl-devel:
[root@mds-Master tools]# yum install openssl-devel
Dependency Installed:
e2fsprogs-devel.x86_64 0:1.39⑵3.el5_5.1 keyutils-libs-devel.x86_64 0:1.2⑴.el5 krb5-devel.x86_64 0:1.6.1⑸5.el5_6.2
libselinux-devel.x86_64 0:1.33.4⑸.7.el5 libsepol-devel.x86_64 0:1.15.2⑶.el5 zlib-devel.x86_64 0:1.2.3⑶
Complete
再次編譯
[root@mds-Master keepalived⑴.2.1]#./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.18⑵38.el5-x86_64
編譯后的信息以下:
Keepalived configuration
------------------------
Keepalived version : 1.2.1
Compiler : gcc
Compiler flags : -g -O2 -DETHERTYPE_IPV6=0x86dd
Extra Lib : -lpopt -lssl -lcrypto
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
Use VRRP Framework : Yes
Use Debug flags : No
Keepalived的安裝路徑為:/usr/local/keepalived,--with-kernel-dir表示指定Linux的內核位置,由于Keepalived是建立在LVS之上的(Linux2.4內核以后,已完全內置了LVS的各個功能模塊,無需給內核打任何補釘,可以直接使用LVS提供的各種功能,關于LVS是甚么東東?可以百度之,也算是國人對Linux的1大貢獻,LVS的作者是畢業國防科技大學的章文嵩博士,現就職于淘寶網)。
[root@mds-Master keepalived⑴.2.1]# make
Make complete
make[1]: Leaving directory`/home/wangwei/tools/keepalived⑴.2.1/genhash'
Make complete
[root@mds-Masterkeepalived⑴.2.1]# make install
make -C keepalived install
。。。。。。。。。。。。。。。。
install -m 644 ../doc/man/man1/genhash.1 /usr/local/keepalived/share/man/man1
make[1]: Leaving directory`/home/wangwei/tools/keepalived⑴.2.1/genhash'
這樣的話,安裝就告1段落了,在192.168.6.243上也安裝一樣版本的Keepalived。
[root@mds-Master ~]# ls -lh /usr/local/keepalived/
total 16K
drwxr-xr-x 2 root root 4.0K Jul 5 08:54 bin
drwxr-xr-x 5 root root 4.0K Jul 5 08:54 etc
drwxr-xr-x 2 root root 4.0K Jul 5 08:54 sbin
drwxr-xr-x 3 root root 4.0K Jul 5 08:54 share
[root@mds-Master ~]#/usr/local/keepalived/sbin/keepalived --help
Keepalived v1.2.1 (07/04,2011)
Usage:
/usr/local/keepalived/sbin/keepalived
/usr/local/keepalived/sbin/keepalived -n
/usr/local/keepalived/sbin/keepalived -f keepalived.conf
/usr/local/keepalived/sbin/keepalived -d
/usr/local/keepalived/sbin/keepalived -h
/usr/local/keepalived/sbin/keepalived-v
。。。。。。。。
可見版本是1.2.1,顯示也很正常
[root@mds-Master ~]# more/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
wei.wang@99bill.com
}
notification_email_from wei.wang@99bill.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id MySQL-HA
}
vrrp_instance VI_1 {
state Backup #兩臺配置此處均是Backup
interface eth0 #VIP綁定在那塊網卡上
smtp_alert #開啟郵件報警
virtual_router_id 51
priority 100 #優先級,另外一臺改成90
advert_int1 #發送廣播信息的間隔時間,單位秒
preempt #搶占,只在優先級高的機器上設置便可,優先級低的機器不設置
authentication {
auth_typePASS
auth_pass1111
}
virtual_ipaddress {
192.168.6.252 #VIP
}
}
virtual_server 192.168.6.252 4321 {
delay_loop 2 #每一個2秒檢查1次real_server狀態
lb_algo wrr #LVS算法
lb_kind DR #LVS模式
persistence_timeout 60 #會話保持時間
protocol TCP
real_server 192.168.6.243 4321 {
weight 3
notify_down"/etc/keepalived/shutdown_keepalived.sh" #檢測到服務down后履行的腳本
TCP_CHECK {
connect_timeout 10 #連接超時時間
nb_get_retry 3 #重連次數
delay_before_retry #重連間隔時間
connect_port 4321 #健康檢查端口
}
}
}
global_defs :全局定義,
notification_email:指定Keepalived在產生事件(如:自動切換),需要發送Email到的對象,可以為多個地址
notification_email_from:郵件發送者
smtp_server :發送郵件的郵件服務器,如果本地有開啟sendmail的話,可以采取127.0.0.1
vrrp_instance VI_1 :VRRP實例配置,包括了主從狀態、VIP、權重等
virtual_server: 虛擬主機,包括了虛擬主機的IP+Port,實際主機的IP+Port,和檢查實際主機的IP+Port異常后的操作等
lb_algo:LVS中的負載均衡調度算法
lb_kind:LVS中的負載均衡轉發規則,1般包括DR,NAT,TUN3種,DR是效力最好的1種規則,直接路由轉發
[root@mds-Master ~]# /etc/init.d/sendmail status
sendmail (pid 31476) is running...
You have new mail in /var/spool/mail/root
[mysql@mds-Master tool]$ more/etc/keepalived/shutdown_keepalived.sh
#!/bin/sh
pkill keepalived
就是檢查到MySQL宕機后,自動殺死Keepalived進程,那末另外1臺機器上的Keepalived進程在接受廣播信息超時后,自動接收VIP,進入到Master狀態
上圖是Keepalived自動發送的郵件,這個很好,可對Keepalived的狀態變更自動監控
[root@mds-Master ~]#/usr/local/keepalived/sbin/keepalived -D
默許情況下,Keepalived的配置文件是/etc/keepalived/keepalived.conf
查看日志文件,Keepalived默許的日志文件是/var/log/messages
[root@mds-Master ~]# tail ⑴0 /var/log/messages
Jul 11 17:26:10 mds-Master Keepalived_vrrp:Registering Kernel netlink reflector
Jul 11 17:26:10 mds-Master Keepalived_vrrp:Registering Kernel netlink command channel
Jul 11 17:26:10 mds-Master Keepalived_vrrp:Registering gratutious ARP shared channel
Jul 11 17:26:10 mds-Master Keepalived_vrrp: Openingfile '/etc/keepalived/keepalived.conf'.
Jul 11 17:26:10 mds-Master Keepalived_vrrp:Configuration is using : 63222 Bytes
Jul 11 17:26:10 mds-Master Keepalived_vrrp: UsingLinkWatch kernel netlink reflector...
Jul 11 17:26:10 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Entering Backup STATE
Jul 11 17:26:10 mds-Master Keepalived_vrrp: VRRPsockpool: [ifindex(2), proto(112), fd(11,12)]
Jul 11 17:26:11 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) forcing a new Master election
Jul 11 17:26:12 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Transition to Master STATE
Jul 11 17:26:13 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Entering Master STATE
Jul 11 17:26:13 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) setting protocol VIPs.
Jul 11 17:26:13 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.6.252
至此,Keepalived的安裝配置告1段落了。
Keepalived在RHEL 6.2中,不會自動的加載ipvs模塊,可以通過下面命令來確認是不是已加載:
lsmod|grep ip_vs
所以依照上述方式安裝后,日志文件(默許是:/var/log/messages)中出現以下報警信息,“Keepalived_healthcheckers:IPVS: Can't initialize ipvs: Protocol not available”,ipvs稱之為IP 虛擬服務器(IP Virtual Server),主要是運行在LVS下的提供負載平衡功能的1種技術,而實際上,我們使用的keepalived并沒有使用負載均衡的功能,只是使用虛擬IP自動漂移的部份,所以呢,即使是沒有找到,keepalived也是正常運轉的,不影響正常使用。
這樣,我們就需要手工加載ipvs模塊,命令以下:
modprobeip_vs
modprobeip_vs_wrr
再重啟keepalived便可,即下面的兩條命令:
Sh/etc/keepalived/shutdown_keepalived.sh --關閉
/usr/local/keepalived/sbin/keepalived -D --啟動
再次視察日志文件,可以看到,上述的報警信息就再也沒有出現了。
在啟動192.168.6.243和192.168.6.244兩臺機器上的Keepalived后(正常與否,需要檢查日志文件/var/log/messages),,利用MySQL的客戶端連接之,看是不是可用登陸
下面我就在192.168.6.49這臺機器上,遠程登陸192.168.6.252:4321(虛擬IP+虛擬端口),看是不是可以正常登陸之
[root@mrs_test ~]# hostname -i
192.168.6.49
[root@mrs_test ~]# mysql -h192.168.6.252 -P4321-uwangwei -ptianma
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 295
Server version: 5.1.47-log Source distribution
Type 'help;' or 'h' for help. Type 'c' to clearthe buffer.
mysql> status
--------------
mysql Ver14.14 Distrib 5.1.33, for pc-linux-gnu (i686) using readline 5.1
Connection id: 295
Current database:
Current user: wangwei@192.168.6.49
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 5.1.47-log Source distribution
Protocol version: 10
Connection: 192.168.6.252 via TCP/IP
Server characterset: utf8
Db characterset: utf8
Client characterset: latin1
Conn. characterset: latin1
TCP port: 4321
Uptime: 10 min 32 sec
Threads: 2 Questions: 10 Slow queries:0 Opens: 14 Flush tables: 1 Open tables: 8 Queries per second avg: 0.15
--------------
mysql>
可見1切才正常,可以正常登陸之
下面我們摹擬MySQL掛掉后,看VIP是不是會自動地被另外1臺機器接收。
[root@mds-Master ~]# tail ⑴0 /var/log/messages
Jul 12 10:26:24 mds-Master Keepalived_vrrp: RemoteSMTP server [127.0.0.1:25] connected.
Jul 12 10:26:24 mds-Master Keepalived_vrrp: SMTPalert successfully sent.
Jul 12 10:26:25 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) forcing a new Master election
Jul 12 10:26:26 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Transition to Master STATE
Jul 12 10:26:27 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Entering Master STATE
Jul 12 10:26:27 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) setting protocol VIPs.
Jul 12 10:26:27 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.6.252
Jul 12 10:26:27 mds-Master Keepalived_vrrp: RemoteSMTP server [127.0.0.1:25] connected.
Jul 12 10:26:27 mds-Master Keepalived_vrrp: SMTPalert successfully sent.
Jul 12 10:26:32 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.6.252
可見192.168.6.243上的Keepalived是Master狀態
[root@mds-slave ~]# tail ⑴0 /var/log/messages
Jul 12 10:26:40 mds-slave avahi-daemon[4968]:Registering new address record for 192.168.6.252 on eth0.
Jul 12 10:26:40 mds-slave avahi-daemon[4968]:Registering new address record for 192.168.6.244 on eth0.
Jul 12 10:26:40 mds-slave avahi-daemon[4968]:Registering HINFO record with values 'X86_64'/'LINUX'.
Jul 12 10:26:56 mds-slave Keepalived_vrrp:VRRP_Instance(VI_1) Received higher prio advert
Jul 12 10:26:56 mds-slave Keepalived_vrrp:VRRP_Instance(VI_1) Entering Backup STATE
Jul 12 10:26:56 mds-slave Keepalived_vrrp:VRRP_Instance(VI_1) removing protocol VIPs.
Jul 12 10:26:56 mds-slave Keepalived_vrrp: Netlinkreflector reports IP 192.168.6.244 removed
Jul 12 10:26:56 mds-slave Keepalived_healthcheckers:Netlink reflector reports IP 192.168.6.244 removed
Jul 12 10:26:56 mds-slave avahi-daemon[4968]:Server startup complete. Host name is mds-slave⑸1.local. Local service cookieis 479334863.
Jul 12 10:26:56 mds-slave avahi-daemon[4968]:Withdrawing address record for 192.168.6.252 on eth0.
可見192.168.6.243上的Keepalived處于Slave狀態
[mysql@mds-Master mysql]$ mysqladmin--defaults-file=my1.cnf -uroot -p99billshutdown
110712 10:50:16 mysqld_safe mysqld from pid file/home/mysql/mysql/mysql/mysqld.pid ended
[1]+ Done /usr/local/mysql/bin/mysqld_safe --defaults-file=my1.cnf
[root@mds-Master ~]# tail ⑴0 /var/log/messages
Jul 12 10:26:32 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.6.252
Jul 12 10:50:09 mds-MasterKeepalived_healthcheckers: TCP connection to [192.168.6.243:4321] failed !!!
Jul 12 10:50:09 mds-MasterKeepalived_healthcheckers: Removing service [192.168.6.243:4321] from VS[192.168.6.252:4321]
Jul 12 10:50:09 mds-MasterKeepalived_healthcheckers: Executing [/etc/keepalived/shutdown_keepalived.sh]for service [192.168.6.243:4321] in VS [192.168.6.252:4321]
Jul 12 10:50:09 mds-MasterKeepalived_healthcheckers: Lost quorum 1-0=1 > 0 for VS [192.168.6.252:4321]
Jul 12 10:50:09 mds-MasterKeepalived_healthcheckers: Remote SMTP server [127.0.0.1:25] connected.
Jul 12 10:50:09 mds-Master Keepalived: Terminatingon signal
Jul 12 10:50:09 mds-Master Keepalived_vrrp:Terminating VRRP child process on signal
Jul 12 10:50:09 mds-Master Keepalived: StoppingKeepalived v1.2.1 (07/04,2011)
Jul 12 10:50:09 mds-MasterKeepalived_healthcheckers: Terminating Healthchecker child process on signal
可見192.168.6.243上的Keepalived對實際服務器進行了健康檢查,失敗之,觸發腳本/etc/keepalived/shutdown_keepalived.sh的履行,發送報警郵件
3.2.5 查看192.168.6.243上的Keepalived進程
[root@mds-Master ~]# ps aux |grep keep
root 20370 0.0 0.0 61188 768 pts/4 S+ 10:54 0:00 grep keep
Keepalived進程已不存在
[root@mds-slave ~]# tail ⑴0 /var/log/messages
Jul 12 10:52:38 mds-slave avahi-daemon[4968]:Registering new address record for 192.168.6.252 on eth0.
Jul 12 10:52:38 mds-slave avahi-daemon[4968]:Registering new address record for 192.168.6.244 on eth0.
Jul 12 10:52:38 mds-slave avahi-daemon[4968]:Registering HINFO record with values 'X86_64'/'LINUX'.
Jul 12 10:52:58 mds-slave avahi-daemon[4968]:Withdrawing address record for fe80::baac:6fff:fe15:bf64 on eth0.
Jul 12 10:52:58 mds-slave avahi-daemon[4968]:Withdrawing address record for 192.168.6.244 on eth0.
Jul 12 10:52:58 mds-slave avahi-daemon[4968]:Registering new address record for fe80::baac:6fff:fe15:bf64 on eth0.
Jul 12 10:52:58 mds-slave avahi-daemon[4968]:Registering new address record for 192.168.6.252 on eth0.
Jul 12 10:52:58 mds-slave avahi-daemon[4968]:Registering new address record for 192.168.6.244 on eth0.
Jul 12 10:52:58 mds-slave avahi-daemon[4968]:Registering HINFO record with values 'X86_64'/'LINUX'.
可見192.168.6.243自動接收了192.168.6.252的管理
[root@mrs_test ~]# hostname -i
192.168.6.49
[root@mrs_test ~]# mysql -h192.168.6.252 -P4321-uwangwei -ptianma
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 293927
Server version: 5.1.47-log Source distribution
Type 'help;' or 'h' for help. Type 'c' to clearthe buffer.
mysql> status
--------------
mysql Ver14.14 Distrib 5.1.33, for pc-linux-gnu (i686) using readline 5.1
Connection id: 293927
Current database:
Current user: wangwei@192.168.6.49
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 5.1.47-log Source distribution
Protocol version: 10
Connection: 192.168.6.252 via TCP/IP
Server characterset: utf8
Db characterset: utf8
Client characterset: latin1
Conn. characterset: latin1
TCP port: 4321
Uptime: 6 days 19 hours 13 min 2 sec
Threads: 2 Questions: 1645 Slow queries:0 Opens: 175 Flush tables: 1 Open tables: 62 Queries per second avg: 0.2
--------------
mysql>
可見VIP+虛擬Port仍然可以正常訪問,這時候候實際服務器是MySQL⑴92.168.6.243
[mysql@mds-Master mysql]$ /usr/local/mysql/bin/mysqld_safe --defaults-file=my1.cnf &
[1] 20374
[mysql@mds-Master mysql]$ 110712 10:55:06mysqld_safe Logging to '/home/mysql/mysql/mysql/mysqld.log'.
110712 10:55:06 mysqld_safe Starting mysqld daemonwith databases from /home/mysql/mysql/mysql
[root@mds-Master ~]#/usr/local/keepalived/sbin/keepalived -D
[root@mds-Master ~]# tail ⑴0 /var/log/messages
Jul 12 10:55:47 mds-Master Keepalived_vrrp: RemoteSMTP server [127.0.0.1:25] connected.
Jul 12 10:55:48 mds-Master Keepalived_vrrp: SMTPalert successfully sent.
Jul 12 10:55:48 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) forcing a new Master election
Jul 12 10:55:49 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Transition to Master STATE
Jul 12 10:55:50 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Entering Master STATE
Jul 12 10:55:50 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) setting protocol VIPs.
Jul 12 10:55:50 mds-Master Keepalived_vrrp:VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.6.252
Jul 12 10:55:50 mds-Master Keepalived_vrrp: RemoteSMTP server [127.0.0.1:25] connected.
Jul 12 10:55:50 mds-Master Keepalived_vrrp: SMTPalert successfully sent.
Jul 12 10:55:55 mds-Master Keepalived_vrrp: VRRP_Instance(VI_1)Sending gratuitous ARPs on eth0 for 192.168.6.252
可見192.168.6.243重新進入Master狀態,這里需要注意的是:配置文件中我配置的是搶占式,如果是不搶占的話,192.168.6.243重啟后會進入Slave狀態,不會產生Master的搶占。
[root@mds-slave ~]# tail ⑴0 /var/log/messages
Jul 12 10:56:18 mds-slave avahi-daemon[4968]:Registering new address record for 192.168.6.252 on eth0.
Jul 12 10:56:18 mds-slave avahi-daemon[4968]:Registering new address record for 192.168.6.244 on eth0.
Jul 12 10:56:18 mds-slave avahi-daemon[4968]: RegisteringHINFO record with values 'X86_64'/'LINUX'.
Jul 12 10:56:19 mds-slave Keepalived_vrrp:VRRP_Instance(VI_1) Received higher prio advert
Jul 12 10:56:19 mds-slave Keepalived_vrrp:VRRP_Instance(VI_1) Entering Backup STATE
Jul 12 10:56:19 mds-slave Keepalived_vrrp:VRRP_Instance(VI_1) removing protocol VIPs.
Jul 12 10:56:19 mds-slave Keepalived_vrrp: Netlinkreflector reports IP 192.168.6.244 removed
Jul 12 10:56:19 mds-slave avahi-daemon[4968]:Withdrawing address record for 192.168.6.252 on eth0.
Jul 12 10:56:19 mds-slaveKeepalived_healthcheckers: Netlink reflector reports IP 192.168.6.244 removed
Jul 12 10:56:20 mds-slave avahi-daemon[4968]:Server startup complete. Host name is mds-slave⑻1.local. Local service cookieis 479334863.
可見192.168.6.243自動的進入Slave狀態
其實3.2中已包括了3.3(摹擬Keepalived掛掉)這類的場景,這里就不再敘述,詳見3.2