K8S(03)模拟生产环境搭建高可用集群之Master节点高可用方案

By | 2021年5月6日
目录
[隐藏]

注意:本高可用方案不仅适用于本文的 K8S 主控节点的高可用,还适用于任何需要高可用的业务场景,haproxy 可改用 nginx 或其他负载均衡器实现
大家都知道在生产环境部署服务一定要坚持一条:不允许出现单点故障。我们在测试环境部署 k8s 的架构一般是单主控 Master 节点多个工作 Node 节点,生产上部署 K8S 集群要避免主控节点宕机,我们需要对主控节点进行高可用部署。
生产环境对主控节点的高可用的解决方案:对主控节点部署多台(3 台以上),然后多部署多台(一般 2 台以上)负载均衡器(一般选用 Nginx 或者 Haproxy)来对主控节点的 api-server 服务进行负载以防止单点故障。本文将详细说明怎么对主控节点的 api-server 服务高可用,主要讲负载均衡器配置值部署这一块,集群的详细搭建在后面的文章中。


主工作节点:192.168.100.107
从工作节点:192.168.100.108
虚拟 IP :192.168.100.110

环境说明

系统环境:CentOS7.7
Keepalived 版本:2.0.19
Haproxy 版本:2.0.8

 

安装配置 Keepalived 服务

下载 Keepalived 源码包
官网地址:https://www.keepalived.org/
下载地址:https://www.keepalived.org/software/keepalived-2.0.19.tar.gz

 

上传并解压 Keepalived 源码包
tar -zxvf keepalived-2.0.19.tar.gz

编译 Keepalived 准备
进入解压目录:cd keepalived-2.0.19
执行编译准备:./configure –prefix=/work/keepalived
注意:一定要有 gcc 和 openssl 编译相关的依赖

编译安装 Keepalived
make && make install

安装配置 Keepalived
keepalived 启动时会从 /etc/keepalived/ 中相关的目录下查找 keepalived.conf 配置文件,因此将 keepalived 安装目录 /usr/local/keepalived/etc/keepalived.conf 拷贝到 /etc/keepalived/ 中。
mkdir /etc/keepalived/
cp /work/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
cp /work/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived

设置 Keepalived 开机启动项
systemctl enable keepalived
然后就能使用 systemctl start/stop/status keepalived 管理 keepalived 了

配置 Keepalived 服务
107 机器的配置信息:

vrrp_script check_haproxy {
interval 3
script “/work/script/check_haproxy.sh”
}

vrrp_instance kube_master{
state master
interface ens33
virtual_router_id 110
priority 100
advert_int 3
authentication {
auth_type PASS
auth_pass kube_master_password
}
virtual_ipaddress {
192.168.100.110
}
track_script {
check_haproxy
}
}
108 机器的配置信息:

vrrp_script check_haproxy {
interval 3
script “/work/script/check_haproxy.sh”
}

vrrp_instance kube_master{
state backup
interface ens33
virtual_router_id 110
priority 90
advert_int 3
authentication {
auth_type PASS
auth_pass kube_master_password
}
virtual_ipaddress {
192.168.100.110
}
track_script {
check_haproxy
}
}
编写 haproxy 服务检测脚本
vi /work/script/check_haproxy.sh

#!/bin/bash
active_status=`netstat -lntp|grep haproxy|wc -l`
if [ $active_status -gt 0 ]; then
exit 0
else
exit 1
fi
然后给脚本赋予执行权限:chmod +x /work/script/check_haproxy.sh

Haproxy 安装部署

下载 Haproxy 源码包
官网地址:https://www.haproxy.org/
下载地址:https://www.haproxy.org/download/2.0/src/haproxy-2.0.8.tar.gz

 

上传并解压 Haproxy 源码包
tar -zxvf haproxy-2.0.8.tar.gz

编译 Haproxy
需要的依赖库:openssl openssl-devel systemd-deve pcre zlib
make TARGET=linux-glibc USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1 USE_ZLIB=1 USE_CRYPT_H=1 USE_LIBCRYPT=1
开启 https 模式:USE_OPENSSL=1
指定 systemd 模式:USE_SYSTEMD=1
支持 pcre 库:USE_PCRE=1
支持 zlib 库:USE_ZLIB=1
支持 crypt_h 库:USE_CRYPT_H=1
支持 libcrypt 库:USE_LIBCRYPT=1

安装 haproxy
make install PREFIX=/work/haproxy
指定安装目录:PREFIX=/work/haproxy

注册到系统服务
vi /usr/lib/systemd/system/haproxy.service

[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target

[Service]
ExecStartPre=/work/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg   -c -q
ExecStart=/work/haproxy/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg  -p /var/run/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID

[Install]
WantedBy=multi-user.target
编写 Haproxy 配置文件
vi /etc/haproxy/haproxy.cfg

global
log 127.0.0.1 local0
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user root
group root
stats socket /var/lib/haproxy/stats
daemon

listen admin_stats
stats enable
bind *:8080
mode http
option httplog
log global
maxconn 10
stats refresh 30s
stats uri /admin
stats realm haproxy
stats auth admin:admin
stats hide-version
stats admin if TRUE

listen kube_cluster_api_server
log global
bind 192.168.100.110:6443
mode tcp
option tcplog
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
balance roundrobin
server kube_cluster_master01 192.168.100.111:6443 check inter 5000 rise 2 fall 3
server kube_cluster_master02 192.168.100.112:6443 check inter 5000 rise 2 fall 3
server kube_cluster_master03 192.168.100.113:6443 check inter 5000 rise 2 fall 3
创建所需目录
创建 /var/lib/haproxy/stats 文件
mkdir -p /var/lib/haproxy
touch /var/lib/haproxy/stats

修改内核参数
vi /etc/sysctl.conf
增加如下内容:

net.ipv4.ip_nonlocal_bind = 1  #启动haproxy的时候,允许忽视VIP的存在
net.ipv4.ip_forward = 1  #允许转发
执行 sysctl -p 保存结果,使结果生效
如果没有配置以上内核参数,那么 haproxy 在启动的会报出 cannot bind socket 的错误

开放监控页面端口
iptables -I INPUT -p tcp –dport 8080 -j ACCEPT

安装验证
两台机器上都完成了如上的安装配置后

分别启动 Keepalived 服务
systemctl start keepalived

分别启动 Haproxy 服务
systemctl start haproxy
分别登陆两台机器查看 haproxy 服务监控页面:

分别查看两台机器的 keepalived 服务是否正常


分别停止两台机器 keepalived 服务查看 VIP 分配的情况:

常见问题

1.configure: error: no acceptable C compiler found in $PATH See `config.log’ for more details.
解决方法:安装 gcc 库
2.!!! OpenSSL is not properly installed on your system. !!! !!! Can not include OpenSSL headers files.
解决方法:安装 openssl openssl-devel
3.*** WARNING – this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.
解决方法:安装 libnl libnl-devel

 

发表评论

您的电子邮箱地址不会被公开。 必填项已用*标注