注意:本高可用方案不仅适用于本文的K8S主控节点的高可用,还适用于任何需要高可用的业务场景,haproxy可改用nginx或其他负载均衡器实现
大家都知道在生产环境部署服务一定要坚持一条:不允许出现单点故障。我们在测试环境部署k8s的架构一般是单主控Master节点多个工作Node节点,生产上部署K8S集群要避免主控节点宕机,我们需要对主控节点进行高可用部署。
生产环境对主控节点的高可用的解决方案:对主控节点部署多台(3台以上),然后多部署多台(一般2台以上)负载均衡器(一般选用Nginx或者Haproxy)来对主控节点的api-server服务进行负载以防止单点故障。本文将详细说明怎么对主控节点的api-server服务高可用,主要讲负载均衡器配置值部署这一块,集群的详细搭建在后面的文章中。


主工作节点:192.168.100.107
从工作节点:192.168.100.108
虚拟IP :192.168.100.110

环境说明

系统环境:CentOS7.7
Keepalived版本:2.0.19
Haproxy版本:2.0.8

安装配置Keepalived服务

下载Keepalived源码包

官网地址:https://www.keepalived.org/
下载地址:https://www.keepalived.org/software/keepalived-2.0.19.tar.gz

上传并解压Keepalived源码包

tar -zxvf keepalived-2.0.19.tar.gz

编译Keepalived准备

进入解压目录:cd keepalived-2.0.19
执行编译准备:./configure –prefix=/work/keepalived
注意:一定要有gcc和openssl编译相关的依赖

编译安装Keepalived

make && make install

安装配置Keepalived

keepalived启动时会从/etc/keepalived/中相关的目录下查找keepalived.conf配置文件,因此将keepalived安装目录/usr/local/keepalived/etc/keepalived.conf 拷贝到/etc/keepalived/中。
mkdir /etc/keepalived/
cp /work/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
cp /work/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived

设置Keepalived开机启动项

systemctl enable keepalived
然后就能使用systemctl start/stop/status keepalived管理keepalived了

配置Keepalived服务

107机器的配置信息:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
vrrp_script check_haproxy {
interval 3
script "/work/script/check_haproxy.sh"
}

vrrp_instance kube_master{
state master
interface ens33
virtual_router_id 110
priority 100
advert_int 3
authentication {
auth_type PASS
auth_pass kube_master_password
}
virtual_ipaddress {
192.168.100.110
}
track_script {
check_haproxy
}
}

108机器的配置信息:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
vrrp_script check_haproxy {
interval 3
script "/work/script/check_haproxy.sh"
}

vrrp_instance kube_master{
state backup
interface ens33
virtual_router_id 110
priority 90
advert_int 3
authentication {
auth_type PASS
auth_pass kube_master_password
}
virtual_ipaddress {
192.168.100.110
}
track_script {
check_haproxy
}
}

编写haproxy服务检测脚本

vi /work/script/check_haproxy.sh

1
2
3
4
5
6
7
#!/bin/bash
active_status=`netstat -lntp|grep haproxy|wc -l`
if [ $active_status -gt 0 ]; then
exit 0
else
exit 1
fi

然后给脚本赋予执行权限:chmod +x /work/script/check_haproxy.sh

Haproxy安装部署

下载Haproxy源码包

官网地址:https://www.haproxy.org/
下载地址:https://www.haproxy.org/download/2.0/src/haproxy-2.0.8.tar.gz

上传并解压Haproxy源码包

tar -zxvf haproxy-2.0.8.tar.gz

编译Haproxy

需要的依赖库:openssl openssl-devel systemd-deve pcre zlib
make TARGET=linux-glibc USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1 USE_ZLIB=1 USE_CRYPT_H=1 USE_LIBCRYPT=1
开启https模式:USE_OPENSSL=1
指定systemd模式:USE_SYSTEMD=1
支持pcre库:USE_PCRE=1
支持zlib库:USE_ZLIB=1
支持crypt_h库:USE_CRYPT_H=1
支持libcrypt库:USE_LIBCRYPT=1

安装haproxy

make install PREFIX=/work/haproxy
指定安装目录:PREFIX=/work/haproxy

注册到系统服务

vi /usr/lib/systemd/system/haproxy.service

1
2
3
4
5
6
7
8
9
10
11
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target

[Service]
ExecStartPre=/work/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg   -c -q
ExecStart=/work/haproxy/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg  -p /var/run/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID

[Install]
WantedBy=multi-user.target

编写Haproxy配置文件

vi /etc/haproxy/haproxy.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
global
log 127.0.0.1 local0
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user root
group root
stats socket /var/lib/haproxy/stats
daemon

listen admin_stats
stats enable
bind *:8080
mode http
option httplog
log global
maxconn 10
stats refresh 30s
stats uri /admin
stats realm haproxy
stats auth admin:admin
stats hide-version
stats admin if TRUE

listen kube_cluster_api_server
log global
bind 192.168.100.110:6443
mode tcp
option tcplog
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
balance roundrobin
server kube_cluster_master01 192.168.100.111:6443 check inter 5000 rise 2 fall 3
server kube_cluster_master02 192.168.100.112:6443 check inter 5000 rise 2 fall 3
server kube_cluster_master03 192.168.100.113:6443 check inter 5000 rise 2 fall 3

创建所需目录

创建/var/lib/haproxy/stats文件
mkdir -p /var/lib/haproxy
touch /var/lib/haproxy/stats

修改内核参数

vi /etc/sysctl.conf
增加如下内容:

1
2
net.ipv4.ip_nonlocal_bind = 1  #启动haproxy的时候,允许忽视VIP的存在
net.ipv4.ip_forward = 1  #允许转发

执行sysctl -p 保存结果,使结果生效
如果没有配置以上内核参数,那么haproxy在启动的会报出cannot bind socket的错误

开放监控页面端口

iptables -I INPUT -p tcp –dport 8080 -j ACCEPT

安装验证

两台机器上都完成了如上的安装配置后

分别启动Keepalived服务

systemctl start keepalived

分别启动Haproxy服务

systemctl start haproxy
分别登陆两台机器查看haproxy服务监控页面:


分别查看两台机器的keepalived服务是否正常


分别停止两台机器keepalived服务查看VIP分配的情况:

常见问题

1.configure: error: no acceptable C compiler found in $PATH See `config.log’ for more details.
解决方法:安装gcc库
2.!!! OpenSSL is not properly installed on your system. !!! !!! Can not include OpenSSL headers files.
解决方法:安装openssl openssl-devel
3.*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.
解决方法:安装libnl libnl-devel