kubernetes v1.20项目之二进制部署Nginx+Keepalived高可用(四层)负载均衡器
| 阿里云国内75折 回扣 微信号：monov8 |
| 阿里云国际，腾讯云国际，低至75折。AWS 93折 免费开户实名账号 代冲值 优惠多多 微信号：monov8 飞机：@monov6 |
kubernetes v1.20项目之二进制部署Nginx+Keepalived高可用(四层)负载均衡器
本文档主要是利用nginx+keepalive来进行实现的高可用架构
Master节点主要有三个服务kube-apiserver、kube-controller-manager和kube-scheduler，其中kube-controller-manager和kube-scheduler组件自身通过选择机制已经实现了高可用，所以Master高可用主要针对kube-apiserver组件，而该组件是以HTTP API提供服务，因此对他高可用与Web服务器类似，增加负载均衡器对其负载均衡即可，并且可水平扩容
简单的理解就是nginx做的一个apiserver的负载均衡，keepalive主要作用是对nginx做的一个高可用作用。
为了节省机器，这里与K8s Master节点机器复用。也可以独立于k8s集群之外部署，只要nginx与apiserver能通信就行
公有云和私有云上面都不支持keepalive的哈
相关所需资源下载
链接：https://pan.baidu.com/s/1emtDOy7bzxlR_hUw6vY2GQ
提取码：a7j4
--来自百度网盘超级会员V2的分享
**部分文件需要更改ip地址或其他的配置，请改成自己的使用**安装软件包
####master1上面操作
[root@k8s-master01 ~]# yum install epel-release -y
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.njupt.edu.cn
* extras: mirrors.njupt.edu.cn
* updates: mirrors.bupt.edu.cn
base | 3.6 kB 00:00
extras | 2.9 kB 00:00
updates | 2.9 kB 00:00
正在解决依赖关系
--> 正在检查事务
---> 软件包 epel-release.noarch.0.7-11 将被 安装
--> 解决依赖关系完成
依赖关系解决
=======================================================================
Package 架构 版本 源 大小
=======================================================================
正在安装:
epel-release noarch 7-11 extras 15 k
事务概要
=======================================================================
安装 1 软件包
总下载量：15 k
安装大小：24 k
Downloading packages:
epel-release-7-11.noarch.rpm | 15 kB 00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : epel-release-7-11.noarch 1/1
验证中 : epel-release-7-11.noarch 1/1
已安装:
epel-release.noarch 0:7-11
完毕！
[root@k8s-master01 ~]# yum install nginx keepalived -y
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink | 6.0 kB 00:00
* base: mirrors.njupt.edu.cn
* epel: mirrors.nipa.cloud
* extras: mirrors.njupt.edu.cn
* updates: mirrors.bupt.edu.cn
epel | 4.7 kB 00:00
(1/3): epel/x86_64/group_gz | 96 kB 00:00
(2/3): epel/x86_64/primary_db | 7.0 MB 00:01
(3/3): epel/x86_64/updateinfo | 1.0 MB 01:03
正在解决依赖关系
--> 正在检查事务
---> 软件包 keepalived.x86_64.0.1.3.5-19.el7 将被 安装
--> 正在处理依赖关系 libnetsnmpmibs.so.31()(64bit)，它被软件包 keepalived-1.3.5-19.el7.x86_64 需要
--> 正在处理依赖关系 libnetsnmpagent.so.31()(64bit)，它被软件包 keepalived-1.3.5-19.el7.x86_64 需要
--> 正在处理依赖关系 libnetsnmp.so.31()(64bit)，它被软件包 keepalived-1.3.5-19.el7.x86_64 需要
---> 软件包 nginx.x86_64.1.1.20.1-9.el7 将被 安装
--> 正在处理依赖关系 nginx-filesystem = 1:1.20.1-9.el7，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libcrypto.so.1.1(OPENSSL_1_1_0)(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libssl.so.1.1(OPENSSL_1_1_0)(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libssl.so.1.1(OPENSSL_1_1_1)(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 nginx-filesystem，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 redhat-indexhtml，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libcrypto.so.1.1()(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libprofiler.so.0()(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libssl.so.1.1()(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在检查事务
---> 软件包 centos-indexhtml.noarch.0.7-9.el7.centos 将被 安装
---> 软件包 gperftools-libs.x86_64.0.2.6.1-1.el7 将被 安装
---> 软件包 net-snmp-agent-libs.x86_64.1.5.7.2-49.el7_9.1 将被 安装
--> 正在处理依赖关系 libsensors.so.4()(64bit)，它被软件包 1:net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_64 需要
---> 软件包 net-snmp-libs.x86_64.1.5.7.2-49.el7_9.1 将被 安装
---> 软件包 nginx-filesystem.noarch.1.1.20.1-9.el7 将被 安装
---> 软件包 openssl11-libs.x86_64.1.1.1.1g-3.el7 将被 安装
--> 正在检查事务
---> 软件包 lm_sensors-libs.x86_64.0.3.4.0-8.20160601gitf9185e5.el7 将被 安装
--> 解决依赖关系完成
依赖关系解决
=======================================================================
Package 架构 版本 源 大小
=======================================================================
正在安装:
keepalived x86_64 1.3.5-19.el7 base 332 k
nginx x86_64 1:1.20.1-9.el7 epel 587 k
为依赖而安装:
centos-indexhtml noarch 7-9.el7.centos base 92 k
gperftools-libs x86_64 2.6.1-1.el7 base 272 k
lm_sensors-libs x86_64 3.4.0-8.20160601gitf9185e5.el7 base 42 k
net-snmp-agent-libs
x86_64 1:5.7.2-49.el7_9.1 updates 707 k
net-snmp-libs x86_64 1:5.7.2-49.el7_9.1 updates 751 k
nginx-filesystem noarch 1:1.20.1-9.el7 epel 24 k
openssl11-libs x86_64 1:1.1.1g-3.el7 epel 1.5 M
事务概要
=======================================================================
安装 2 软件包 (+7 依赖软件包)
总下载量：4.2 M
安装大小：13 M
Downloading packages:
(1/9): centos-indexhtml-7-9.el7.centos.noarch.rpm | 92 kB 00:00
(2/9): gperftools-libs-2.6.1-1.el7.x86_64.rpm | 272 kB 00:00
(3/9): lm_sensors-libs-3.4.0-8.20160601gitf9185e5 | 42 kB 00:00
(4/9): keepalived-1.3.5-19.el7.x86_64.rpm | 332 kB 00:00
(5/9): net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_6 | 707 kB 00:00
(6/9): net-snmp-libs-5.7.2-49.el7_9.1.x86_64.rpm | 751 kB 00:00
warning: /var/cache/yum/x86_64/7/epel/packages/nginx-1.20.1-9.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
nginx-1.20.1-9.el7.x86_64.rpm 的公钥尚未安装
(7/9): nginx-1.20.1-9.el7.x86_64.rpm | 587 kB 00:01
(8/9): nginx-filesystem-1.20.1-9.el7.noarch.rpm | 24 kB 00:00
(9/9): openssl11-libs-1.1.1g-3.el7.x86_64.rpm | 1.5 MB 00:00
-----------------------------------------------------------------------
总计 1.6 MB/s | 4.2 MB 00:02
从 file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 检索密钥
导入 GPG key 0x352C64E5:
用户ID : "Fedora EPEL (7) <epel@fedoraproject.org>"
指纹 : 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
软件包 : epel-release-7-11.noarch (@extras)
来自 : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : 1:net-snmp-libs-5.7.2-49.el7_9.1.x86_64 1/9
正在安装 : centos-indexhtml-7-9.el7.centos.noarch 2/9
正在安装 : 1:openssl11-libs-1.1.1g-3.el7.x86_64 3/9
正在安装 : gperftools-libs-2.6.1-1.el7.x86_64 4/9
正在安装 : lm_sensors-libs-3.4.0-8.20160601gitf9185e5.el7.x 5/9
正在安装 : 1:net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_64 6/9
正在安装 : 1:nginx-filesystem-1.20.1-9.el7.noarch 7/9
正在安装 : 1:nginx-1.20.1-9.el7.x86_64 8/9
正在安装 : keepalived-1.3.5-19.el7.x86_64 9/9
验证中 : keepalived-1.3.5-19.el7.x86_64 1/9
验证中 : 1:nginx-filesystem-1.20.1-9.el7.noarch 2/9
验证中 : 1:nginx-1.20.1-9.el7.x86_64 3/9
验证中 : lm_sensors-libs-3.4.0-8.20160601gitf9185e5.el7.x 4/9
验证中 : gperftools-libs-2.6.1-1.el7.x86_64 5/9
验证中 : 1:openssl11-libs-1.1.1g-3.el7.x86_64 6/9
验证中 : 1:net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_64 7/9
验证中 : 1:net-snmp-libs-5.7.2-49.el7_9.1.x86_64 8/9
验证中 : centos-indexhtml-7-9.el7.centos.noarch 9/9
已安装:
keepalived.x86_64 0:1.3.5-19.el7 nginx.x86_64 1:1.20.1-9.el7
作为依赖被安装:
centos-indexhtml.noarch 0:7-9.el7.centos
gperftools-libs.x86_64 0:2.6.1-1.el7
lm_sensors-libs.x86_64 0:3.4.0-8.20160601gitf9185e5.el7
net-snmp-agent-libs.x86_64 1:5.7.2-49.el7_9.1
net-snmp-libs.x86_64 1:5.7.2-49.el7_9.1
nginx-filesystem.noarch 1:1.20.1-9.el7
openssl11-libs.x86_64 1:1.1.1g-3.el7
完毕！
####master2上面操作
[root@k8s-master2 ~]# yum install epel-release -y
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.bfsu.edu.cn
* extras: mirrors.bupt.edu.cn
* updates: mirrors.bfsu.edu.cn
base | 3.6 kB 00:00
extras | 2.9 kB 00:00
updates | 2.9 kB 00:00
正在解决依赖关系
--> 正在检查事务
---> 软件包 epel-release.noarch.0.7-11 将被 安装
--> 解决依赖关系完成
依赖关系解决
=======================================================================
Package 架构 版本 源 大小
=======================================================================
正在安装:
epel-release noarch 7-11 extras 15 k
事务概要
=======================================================================
安装 1 软件包
总下载量：15 k
安装大小：24 k
Downloading packages:
epel-release-7-11.noarch.rpm | 15 kB 00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : epel-release-7-11.noarch 1/1
验证中 : epel-release-7-11.noarch 1/1
已安装:
epel-release.noarch 0:7-11
完毕！
[root@k8s-master2 ~]# yum install nginx keepalived -y
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink | 6.0 kB 00:00
* base: mirrors.bfsu.edu.cn
* epel: mirrors.bfsu.edu.cn
* extras: mirrors.bupt.edu.cn
* updates: mirrors.bfsu.edu.cn
epel | 4.7 kB 00:00
(1/3): epel/x86_64/group_gz | 96 kB 00:00
(2/3): epel/x86_64/updateinfo | 1.0 MB 00:00
(3/3): epel/x86_64/primary_db | 7.0 MB 00:00
正在解决依赖关系
--> 正在检查事务
---> 软件包 keepalived.x86_64.0.1.3.5-19.el7 将被 安装
--> 正在处理依赖关系 libnetsnmpmibs.so.31()(64bit)，它被软件包 keepalived-1.3.5-19.el7.x86_64 需要
--> 正在处理依赖关系 libnetsnmpagent.so.31()(64bit)，它被软件包 keepalived-1.3.5-19.el7.x86_64 需要
--> 正在处理依赖关系 libnetsnmp.so.31()(64bit)，它被软件包 keepalived-1.3.5-19.el7.x86_64 需要
---> 软件包 nginx.x86_64.1.1.20.1-9.el7 将被 安装
--> 正在处理依赖关系 nginx-filesystem = 1:1.20.1-9.el7，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libcrypto.so.1.1(OPENSSL_1_1_0)(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libssl.so.1.1(OPENSSL_1_1_0)(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libssl.so.1.1(OPENSSL_1_1_1)(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 nginx-filesystem，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 redhat-indexhtml，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libcrypto.so.1.1()(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libprofiler.so.0()(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在处理依赖关系 libssl.so.1.1()(64bit)，它被软件包 1:nginx-1.20.1-9.el7.x86_64 需要
--> 正在检查事务
---> 软件包 centos-indexhtml.noarch.0.7-9.el7.centos 将被 安装
---> 软件包 gperftools-libs.x86_64.0.2.6.1-1.el7 将被 安装
---> 软件包 net-snmp-agent-libs.x86_64.1.5.7.2-49.el7_9.1 将被 安装
--> 正在处理依赖关系 libsensors.so.4()(64bit)，它被软件包 1:net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_64 需要
---> 软件包 net-snmp-libs.x86_64.1.5.7.2-49.el7_9.1 将被 安装
---> 软件包 nginx-filesystem.noarch.1.1.20.1-9.el7 将被 安装
---> 软件包 openssl11-libs.x86_64.1.1.1.1g-3.el7 将被 安装
--> 正在检查事务
---> 软件包 lm_sensors-libs.x86_64.0.3.4.0-8.20160601gitf9185e5.el7 将被 安装
--> 解决依赖关系完成
依赖关系解决
=======================================================================
Package 架构 版本 源 大小
=======================================================================
正在安装:
keepalived x86_64 1.3.5-19.el7 base 332 k
nginx x86_64 1:1.20.1-9.el7 epel 587 k
为依赖而安装:
centos-indexhtml noarch 7-9.el7.centos base 92 k
gperftools-libs x86_64 2.6.1-1.el7 base 272 k
lm_sensors-libs x86_64 3.4.0-8.20160601gitf9185e5.el7 base 42 k
net-snmp-agent-libs
x86_64 1:5.7.2-49.el7_9.1 updates 707 k
net-snmp-libs x86_64 1:5.7.2-49.el7_9.1 updates 751 k
nginx-filesystem noarch 1:1.20.1-9.el7 epel 24 k
openssl11-libs x86_64 1:1.1.1g-3.el7 epel 1.5 M
事务概要
=======================================================================
安装 2 软件包 (+7 依赖软件包)
总下载量：4.2 M
安装大小：13 M
Downloading packages:
(1/9): centos-indexhtml-7-9.el7.centos.noarch.rpm | 92 kB 00:00
(2/9): lm_sensors-libs-3.4.0-8.20160601gitf9185e5 | 42 kB 00:00
warning: /var/cache/yum/x86_64/7/epel/packages/nginx-1.20.1-9.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
nginx-1.20.1-9.el7.x86_64.rpm 的公钥尚未安装
(3/9): nginx-1.20.1-9.el7.x86_64.rpm | 587 kB 00:00
(4/9): nginx-filesystem-1.20.1-9.el7.noarch.rpm | 24 kB 00:00
(5/9): openssl11-libs-1.1.1g-3.el7.x86_64.rpm | 1.5 MB 00:00
(6/9): gperftools-libs-2.6.1-1.el7.x86_64.rpm | 272 kB 00:01
(7/9): keepalived-1.3.5-19.el7.x86_64.rpm | 332 kB 00:01
(8/9): net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_6 | 707 kB 00:01
(9/9): net-snmp-libs-5.7.2-49.el7_9.1.x86_64.rpm | 751 kB 00:02
-----------------------------------------------------------------------
总计 1.7 MB/s | 4.2 MB 00:02
从 file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 检索密钥
导入 GPG key 0x352C64E5:
用户ID : "Fedora EPEL (7) <epel@fedoraproject.org>"
指纹 : 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
软件包 : epel-release-7-11.noarch (@extras)
来自 : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : 1:net-snmp-libs-5.7.2-49.el7_9.1.x86_64 1/9
正在安装 : centos-indexhtml-7-9.el7.centos.noarch 2/9
正在安装 : 1:openssl11-libs-1.1.1g-3.el7.x86_64 3/9
正在安装 : gperftools-libs-2.6.1-1.el7.x86_64 4/9
正在安装 : lm_sensors-libs-3.4.0-8.20160601gitf9185e5.el7.x 5/9
正在安装 : 1:net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_64 6/9
正在安装 : 1:nginx-filesystem-1.20.1-9.el7.noarch 7/9
正在安装 : 1:nginx-1.20.1-9.el7.x86_64 8/9
正在安装 : keepalived-1.3.5-19.el7.x86_64 9/9
验证中 : keepalived-1.3.5-19.el7.x86_64 1/9
验证中 : 1:nginx-filesystem-1.20.1-9.el7.noarch 2/9
验证中 : 1:nginx-1.20.1-9.el7.x86_64 3/9
验证中 : lm_sensors-libs-3.4.0-8.20160601gitf9185e5.el7.x 4/9
验证中 : gperftools-libs-2.6.1-1.el7.x86_64 5/9
验证中 : 1:openssl11-libs-1.1.1g-3.el7.x86_64 6/9
验证中 : 1:net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_64 7/9
验证中 : 1:net-snmp-libs-5.7.2-49.el7_9.1.x86_64 8/9
验证中 : centos-indexhtml-7-9.el7.centos.noarch 9/9
已安装:
keepalived.x86_64 0:1.3.5-19.el7 nginx.x86_64 1:1.20.1-9.el7
作为依赖被安装:
centos-indexhtml.noarch 0:7-9.el7.centos
gperftools-libs.x86_64 0:2.6.1-1.el7
lm_sensors-libs.x86_64 0:3.4.0-8.20160601gitf9185e5.el7
net-snmp-agent-libs.x86_64 1:5.7.2-49.el7_9.1
net-snmp-libs.x86_64 1:5.7.2-49.el7_9.1
nginx-filesystem.noarch 1:1.20.1-9.el7
openssl11-libs.x86_64 1:1.1.1g-3.el7
完毕！配置nginx的配置文件
###master1上面操作
[root@k8s-master01 ~]# cat > /etc/nginx/nginx.conf << "EOF"
> user nginx;
> worker_processes auto;
> error_log /var/log/nginx/error.log;
> pid /run/nginx.pid;
>
> include /usr/share/nginx/modules/*.conf;
>
> events {
> worker_connections 1024;
> }
>
> # 四层负载均衡，为两台Master apiserver组件提供负载均衡
> stream {
>
> log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
>
> access_log /var/log/nginx/k8s-access.log main;
>
> upstream k8s-apiserver {
> server 192.168.100.13:6443; # Master1 APISERVER IP:PORT
> server 192.168.100.16:6443; # Master2 APISERVER IP:PORT
> }
>
> server {
> listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
> proxy_pass k8s-apiserver;
> }
> }
>
> http {
> log_format main '$remote_addr - $remote_user [$time_local] "$request" '
> '$status $body_bytes_sent "$http_referer" '
> '"$http_user_agent" "$http_x_forwarded_for"';
>
> access_log /var/log/nginx/access.log main;
>
> sendfile on;
> tcp_nopush on;
> tcp_nodelay on;
> keepalive_timeout 65;
> types_hash_max_size 2048;
>
> include /etc/nginx/mime.types;
> default_type application/octet-stream;
>
> server {
> listen 80 default_server;
> server_name _;
>
> location / {
> }
> }
> }
> EOF
[root@k8s-master01 ~]# cat /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 192.168.100.13:6443; # Master1 APISERVER IP:PORT
server 192.168.100.16:6443; # Master2 APISERVER IP:PORT
}
server {
listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
proxy_pass k8s-apiserver;
}
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
server {
listen 80 default_server;
server_name _;
location / {
}
}
}
[root@k8s-master01 ~]# scp /etc/nginx/nginx.conf root@192.168.100.16:/etc/nginx/nginx.conf
root@192.168.100.16's password:
nginx.conf 100% 1372 1.1MB/s 00:00keepalived配置文件（Nginx Master）
###把master1当成nginx master上面操作
[root@k8s-master01 ~]# cat > /etc/keepalived/keepalived.conf << EOF
> global_defs {
> notification_email {
> acassen@firewall.loc
> failover@firewall.loc
> sysadmin@firewall.loc
> }
> notification_email_from Alexandre.Cassen@firewall.loc
> smtp_server 127.0.0.1
> smtp_connect_timeout 30
> router_id NGINX_MASTER
> }
>
> vrrp_script check_nginx {
> script "/etc/keepalived/check_nginx.sh"
> }
>
> vrrp_instance VI_1 {
> state MASTER
> interface ens33 # 修改为实际网卡名
> virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
> priority 100 # 优先级，备服务器设置 90
> advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
> authentication {
> auth_type PASS
> auth_pass 1111
> }
> # 虚拟IP
> virtual_ipaddress {
> 192.168.100.88/24
> }
> track_script {
> check_nginx
> }
> }
> EOF
[root@k8s-master01 ~]# cat /etc/keepalived/keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER
interface ens33 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.100.88/24
}
track_script {
check_nginx
}
}
###准备上述配置文件中检查nginx运行状态的脚本
[root@k8s-master01 ~]# cat > /etc/keepalived/check_nginx.sh << "EOF"
> #!/bin/bash
> count=$(ss -antp |grep 16443 |egrep -cv "grep|$$")
>
> if [ "$count" -eq 0 ];then
> exit 1
> else
> exit 0
> fi
> EOF
[root@k8s-master01 ~]# cat /etc/keepalived/check_nginx.sh
#!/bin/bash
count=$(ss -antp |grep 16443 |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
exit 1
else
exit 0
fi
[root@k8s-master01 ~]# chmod +x /etc/keepalived/check_nginx.shkeepalived配置文件（Nginx Backup）
####master2上面操作
[root@k8s-master2 ~]# cat > /etc/keepalived/keepalived.conf << EOF
> global_defs {
> notification_email {
> acassen@firewall.loc
> failover@firewall.loc
> sysadmin@firewall.loc
> }
> notification_email_from Alexandre.Cassen@firewall.loc
> smtp_server 127.0.0.1
> smtp_connect_timeout 30
> router_id NGINX_BACKUP
> }
>
> vrrp_script check_nginx {
> script "/etc/keepalived/check_nginx.sh"
> }
>
> vrrp_instance VI_1 {
> state BACKUP
> interface ens33
> virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
> priority 90
> advert_int 1
> authentication {
> auth_type PASS
> auth_pass 1111
> }
> virtual_ipaddress {
> 192.168.100.88/24
> }
> track_script {
> check_nginx
> }
> }
> EOF
[root@k8s-master2 ~]# cat /etc/keepalived/keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.100.88/24
}
track_script {
check_nginx
}
}
###准备上述配置文件中检查nginx运行状态的脚本
[root@k8s-master2 ~]# cat > /etc/keepalived/check_nginx.sh << "EOF"
> #!/bin/bash
> count=$(ss -antp |grep 16443 |egrep -cv "grep|$$")
>
> if [ "$count" -eq 0 ];then
> exit 1
> else
> exit 0
> fi
> EOF
[root@k8s-master2 ~]# cat /etc/keepalived/check_nginx.sh
#!/bin/bash
count=$(ss -antp |grep 16443 |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
exit 1
else
exit 0
fi
[root@k8s-master2 ~]# chmod +x /etc/keepalived/check_nginx.sh启动并设置开机启动
##master1上操作
[root@k8s-master01 ~]# systemctl daemon-reload
[root@k8s-master01 ~]# systemctl start nginx keepalived
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details. ###可以看到nginx没有启动成功
[root@k8s-master01 ~]# systemctl enable nginx keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
####查看nginx当前状态
[root@k8s-master01 ~]# systemctl status nginx
● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since 三 2021-11-17 21:14:03 CST; 34s ago
11月 17 21:14:03 k8s-master01 systemd[1]: Starting The nginx HTTP a...
11月 17 21:14:03 k8s-master01 nginx[31364]: nginx: [emerg] unknown ...
11月 17 21:14:03 k8s-master01 nginx[31364]: nginx: configuration fi...
11月 17 21:14:03 k8s-master01 systemd[1]: nginx.service: control pr...
11月 17 21:14:03 k8s-master01 systemd[1]: Failed to start The nginx...
11月 17 21:14:03 k8s-master01 systemd[1]: Unit nginx.service entere...
11月 17 21:14:03 k8s-master01 systemd[1]: nginx.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-master01 ~]# nginx -t
nginx: [emerg] unknown directive "stream" in /etc/nginx/nginx.conf:13 ####因为使用的是四层的负载均衡，主要使用的是nginx的stream模块，看来是通过yum安装的时候没有安装上去
nginx: configuration file /etc/nginx/nginx.conf test failed
###查看当前安装的模块
[root@k8s-master01 ~]# nginx -V
nginx version: nginx/1.20.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.1.1g FIPS 21 Apr 2020
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E'
###确实没有--with-stream
##为nginx添加stream模块
[root@k8s-master01 ~]# rpm -qa nginx
nginx-1.20.1-9.el7.x86_64
[root@k8s-master01 ~]# wget http://nginx.org/download/nginx-1.20.1.tar.gz
--2021-11-17 21:50:37-- http://nginx.org/download/nginx-1.20.1.tar.gz
正在解析主机 nginx.org (nginx.org)... 52.58.199.22, 3.125.197.172, 2a05:d014:edb:5704::6, ...
正在连接 nginx.org (nginx.org)|52.58.199.22|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：1061461 (1.0M) [application/octet-stream]
正在保存至: “nginx-1.20.1.tar.gz”
100%[=======================================================================================================>] 1,061,461 800KB/s 用时 1.3s
2021-11-17 21:50:39 (800 KB/s) - 已保存 “nginx-1.20.1.tar.gz” [1061461/1061461])
[root@k8s-master01 ~]# where nginx
-bash: where: 未找到命令
[root@k8s-master01 ~]# sz nginx-1.20.1.tar.gz
[root@k8s-master01 ~]# ^C
[root@k8s-master01 ~]# tar xf nginx-1.20.1.tar.gz && cd nginx-1.20.1
[root@k8s-master01 nginx-1.20.1]# mv /usr/sbin/nginx /usr/sbin/nginx.bak
[root@k8s-master01 nginx-1.20.1]# cp -r /etc/nginx /etc/nginx.bak
[root@k8s-master01 nginx-1.20.1]# yum -y install libxml2 libxml2-dev libxslt-devel gd-devel perl-devel perl-ExtUtils-Embed GeoIP GeoIP-devel GeoIP-data
[root@k8s-master01 nginx-1.20.1]# yum -y install --skip-broken gcc gcc-c++ autoconf automake gperftools
[root@k8s-master01 nginx-1.20.1]# yum -y install --skip-broken zlib zlib-devel openssl-devel pcre-devel
[root@k8s-master01 nginx-1.20.1]# yum -y install redhat-rpm-config.noarch
[root@k8s-master01 nginx-1.20.1]# ./configure --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' --with-stream
[root@k8s-master01 nginx-1.20.1]# make
[root@k8s-master01 nginx-1.20.1]# make install
make -f objs/Makefile install
make[1]: 进入目录“/root/nginx-1.20.1”
cd objs/src/http/modules/perl && make install
make[2]: 进入目录“/root/nginx-1.20.1/objs/src/http/modules/perl”
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /usr/local/lib64/perl5/auto/nginx/nginx.so
Installing /usr/local/lib64/perl5/auto/nginx/nginx.bs
Installing /usr/local/lib64/perl5/nginx.pm
Installing /usr/local/share/man/man3/nginx.3pm
Appending installation info to /usr/lib64/perl5/perllocal.pod
make[2]: 离开目录“/root/nginx-1.20.1/objs/src/http/modules/perl”
test -d '/usr/share/nginx' || mkdir -p '/usr/share/nginx'
test -d '/usr/sbin' \
|| mkdir -p '/usr/sbin'
test ! -f '/usr/sbin/nginx' \
|| mv '/usr/sbin/nginx' \
'/usr/sbin/nginx.old'
cp objs/nginx '/usr/sbin/nginx'
test -d '/etc/nginx' \
|| mkdir -p '/etc/nginx'
cp conf/koi-win '/etc/nginx'
cp conf/koi-utf '/etc/nginx'
cp conf/win-utf '/etc/nginx'
test -f '/etc/nginx/mime.types' \
|| cp conf/mime.types '/etc/nginx'
cp conf/mime.types '/etc/nginx/mime.types.default'
test -f '/etc/nginx/fastcgi_params' \
|| cp conf/fastcgi_params '/etc/nginx'
cp conf/fastcgi_params \
'/etc/nginx/fastcgi_params.default'
test -f '/etc/nginx/fastcgi.conf' \
|| cp conf/fastcgi.conf '/etc/nginx'
cp conf/fastcgi.conf '/etc/nginx/fastcgi.conf.default'
test -f '/etc/nginx/uwsgi_params' \
|| cp conf/uwsgi_params '/etc/nginx'
cp conf/uwsgi_params \
'/etc/nginx/uwsgi_params.default'
test -f '/etc/nginx/scgi_params' \
|| cp conf/scgi_params '/etc/nginx'
cp conf/scgi_params \
'/etc/nginx/scgi_params.default'
test -f '/etc/nginx/nginx.conf' \
|| cp conf/nginx.conf '/etc/nginx/nginx.conf'
cp conf/nginx.conf '/etc/nginx/nginx.conf.default'
test -d '/run' \
|| mkdir -p '/run'
test -d '/var/log/nginx' \
|| mkdir -p '/var/log/nginx'
test -d '/usr/share/nginx/html' \
|| cp -R html '/usr/share/nginx'
test -d '/var/log/nginx' \
|| mkdir -p '/var/log/nginx'
test -d '/usr/lib64/nginx/modules' \
|| mkdir -p '/usr/lib64/nginx/modules'
test ! -f '/usr/lib64/nginx/modules/ngx_http_xslt_filter_module.so' \
|| mv '/usr/lib64/nginx/modules/ngx_http_xslt_filter_module.so' \
'/usr/lib64/nginx/modules/ngx_http_xslt_filter_module.so.old'
cp objs/ngx_http_xslt_filter_module.so '/usr/lib64/nginx/modules/ngx_http_xslt_filter_module.so'
test ! -f '/usr/lib64/nginx/modules/ngx_http_image_filter_module.so' \
|| mv '/usr/lib64/nginx/modules/ngx_http_image_filter_module.so' \
'/usr/lib64/nginx/modules/ngx_http_image_filter_module.so.old'
cp objs/ngx_http_image_filter_module.so '/usr/lib64/nginx/modules/ngx_http_image_filter_module.so'
test ! -f '/usr/lib64/nginx/modules/ngx_http_perl_module.so' \
|| mv '/usr/lib64/nginx/modules/ngx_http_perl_module.so' \
'/usr/lib64/nginx/modules/ngx_http_perl_module.so.old'
cp objs/ngx_http_perl_module.so '/usr/lib64/nginx/modules/ngx_http_perl_module.so'
test ! -f '/usr/lib64/nginx/modules/ngx_mail_module.so' \
|| mv '/usr/lib64/nginx/modules/ngx_mail_module.so' \
'/usr/lib64/nginx/modules/ngx_mail_module.so.old'
cp objs/ngx_mail_module.so '/usr/lib64/nginx/modules/ngx_mail_module.so'
make[1]: 离开目录“/root/nginx-1.20.1”
[root@k8s-master01 nginx-1.20.1]# which nginx
/usr/sbin/nginx
[root@k8s-master01 nginx-1.20.1]# nginx -V
nginx version: nginx/1.20.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' --with-stream
#### 编辑nginx的配置文件
[root@k8s-master01 nginx-1.20.1]# vim /etc/nginx/nginx.conf
[root@k8s-master01 nginx-1.20.1]# cat /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 192.168.100.13:6443; # Master1 APISERVER IP:PORT
server 192.168.100.16:6443; # Master2 APISERVER IP:PORT
}
server {
listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
proxy_pass k8s-apiserver;
}
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
server {
listen 80 default_server;
server_name _;
location / {
}
}
}
[root@k8s-master01 nginx-1.20.1]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@k8s-master01 nginx-1.20.1]# systemctl start nginx
[root@k8s-master01 nginx-1.20.1]# ps -ef | grep nginx
root 81574 1 0 22:14 ? 00:00:00 nginx: master process /usr/sbin/nginx
nginx 81575 81574 0 22:14 ? 00:00:00 nginx: worker process
nginx 81576 81574 0 22:14 ? 00:00:00 nginx: worker process
root 81754 16236 0 22:14 pts/0 00:00:00 grep --color=auto nginx
###master2上面操作
[root@k8s-master2 ~]# wget http://nginx.org/download/nginx-1.20.1.tar.gz
--2021-11-17 22:31:09-- http://nginx.org/download/nginx-1.20.1.tar.gz
正在解析主机 nginx.org (nginx.org)... 52.58.199.22, 3.125.197.172, 2a05:d014:edb:5702::6, ...
正在连接 nginx.org (nginx.org)|52.58.199.22|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：1061461 (1.0M) [application/octet-stream]
正在保存至: “nginx-1.20.1.tar.gz”
100%[=======================================================================================================>] 1,061,461 684KB/s 用时 1.5s
2021-11-17 22:31:11 (684 KB/s) - 已保存 “nginx-1.20.1.tar.gz” [1061461/1061461])
[root@k8s-master2 ~]# ls
anaconda-ks.cfg nginx-1.20.1.tar.gz
[root@k8s-master2 ~]# tar xf nginx-1.20.1.tar.gz && cd nginx-1.20.1
[root@k8s-master2 nginx-1.20.1]# mv /usr/sbin/nginx /usr/sbin/nginx.bak
[root@k8s-master2 nginx-1.20.1]# cp -r /etc/nginx /etc/nginx.bak
[root@k8s-master2 nginx-1.20.1]# yum -y install libxml2 libxml2-dev libxslt-devel gd-devel perl-devel perl-ExtUtils-Embed GeoIP GeoIP-devel GeoIP-data
[root@k8s-master2 nginx-1.20.1]# yum -y install --skip-broken gcc gcc-c++ autoconf automake gperftools
[root@k8s-master2 nginx-1.20.1]# yum -y install --skip-broken zlib zlib-devel openssl-devel pcre-devel
[root@k8s-master2 nginx-1.20.1]# yum -y install redhat-rpm-config.noarch
[root@k8s-master2 nginx-1.20.1]# ls
auto CHANGES CHANGES.ru conf configure contrib html LICENSE man README src
[root@k8s-master2 nginx-1.20.1]# ./configure --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' --with-stream
[root@k8s-master2 nginx-1.20.1]# make
[root@k8s-master2 nginx-1.20.1]# make install
[root@k8s-master2 nginx-1.20.1]# which nginx
/usr/sbin/nginx
[root@k8s-master2 nginx-1.20.1]# nginx -V
nginx version: nginx/1.20.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' --with-stream
[root@k8s-master2 nginx-1.20.1]# vim /etc/nginx/nginx.conf
[root@k8s-master2 nginx-1.20.1]# cat /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 192.168.100.13:6443; # Master1 APISERVER IP:PORT
server 192.168.100.16:6443; # Master2 APISERVER IP:PORT
}
server {
listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
proxy_pass k8s-apiserver;
}
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
server {
listen 80 default_server;
server_name _;
location / {
}
}
}
[root@k8s-master2 nginx-1.20.1]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
####启动并配置开机启动
[root@k8s-master2 nginx-1.20.1]# systemctl daemon-reload
[root@k8s-master2 nginx-1.20.1]# systemctl start nginx keepalived
[root@k8s-master2 nginx-1.20.1]# systemctl enable nginx keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.查看keepalived工作状态
###master1上面操作
[root@k8s-master01 nginx-1.20.1]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:fb:d7:e0 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.100.88/24 scope global secondary ens33 ##出现了vip
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fefb:d7e0/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:38:2b:cc:5f brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
####master2上面操作
[root@k8s-master2 nginx-1.20.1]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:b7:99:32 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.16/24 brd 192.168.100.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:feb7:9932/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:48:a8:2e:5b brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever对高可用进行测试
####master1上面操作
[root@k8s-master01 nginx-1.20.1]# pkill nginx
[root@k8s-master01 nginx-1.20.1]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:fb:d7:e0 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fefb:d7e0/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:38:2b:cc:5f brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
VIP已经飘逸
###master2上操作
[root@k8s-master2 nginx-1.20.1]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:b7:99:32 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.16/24 brd 192.168.100.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.100.88/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:feb7:9932/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:48:a8:2e:5b brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
VIP已经漂移到master2上面了访问负载均衡器测试
找K8s集群中任意一个节点，使用curl查看K8s版本测试，使用VIP访问
####
[root@k8s-master2 nginx-1.20.1]# curl -k https://192.168.100.88:16443/version
curl: (35) Encountered end of file
###出现报错
[root@k8s-master2 nginx-1.20.1]# curl -k https://192.168.100.88:16443/version -vvv
* About to connect() to 192.168.100.88 port 16443 (#0)
* Trying 192.168.100.88...
* Connected to 192.168.100.88 (192.168.100.88) port 16443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file
检查各个服务和端口，发现etcd挂了，是自己中途虚拟机全部关了一下机，进行的重启。etcd没有设置开机自启动导致，开启后正常
[root@k8s-master2 nginx-1.20.1]# curl -k https://192.168.100.88:16443/version
{
"major": "1",
"minor": "20",
"gitVersion": "v1.20.5",
"gitCommit": "6b1d87acf3c8253c123756b9e61dac642678305f",
"gitTreeState": "clean",
"buildDate": "2021-03-18T01:02:01Z",
"goVersion": "go1.15.8",
"compiler": "gc",
"platform": "linux/amd64"
}[root@k8s-master2 nginx-1.20.1]#
可以正确获取到K8s版本信息，说明负载均衡器搭建正常。该请求数据流程：curl -> vip(nginx) -> apiserver修改所有的Worker Node连接LB VIP
###先检查所有的节点状态，任意master上执行
[root@k8s-master01 nginx-1.20.1]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready <none> 24h v1.20.5
k8s-master2 Ready <none> 9h v1.20.5
k8s-node01 Ready <none> 23h v1.20.5
###要改**所有**Worker Node（kubectl get node命令查看到的节点）组件配置文件，由原来192.168.100.13修改为192.168.100.88（VIP）
###所有node上面执行
[root@k8s-master01 nginx-1.20.1]# sed -i 's#192.168.100.13:6443#192.168.100.88:16443#' /opt/kubernetes/cfg/*
[root@k8s-master01 nginx-1.20.1]# systemctl restart kubelet kube-proxy
[root@k8s-master01 nginx-1.20.1]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready <none> 24h v1.20.5
k8s-master2 Ready <none> 9h v1.20.5
k8s-node01 Ready <none> 23h v1.20.5
[root@k8s-master01 nginx-1.20.1]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready <none> 24h v1.20.5
k8s-master2 Ready <none> 9h v1.20.5
k8s-node01 Ready <none> 23h v1.20.5
[root@k8s-master01 nginx-1.20.1]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready <none> 24h v1.20.5
k8s-master2 Ready <none> 9h v1.20.5
k8s-node01 Ready <none> 23h v1.20.5
Kubernetes 高可用集群就部署完成了结束语
明天是最美好了，虽然会经历风雨，但不能阻挡自己前进的脚步
上一篇内容：kubernetes v1.20项目之二进制扩容多Master
| 阿里云国内75折 回扣 微信号：monov8 |
| 阿里云国际，腾讯云国际，低至75折。AWS 93折 免费开户实名账号 代冲值 优惠多多 微信号：monov8 飞机：@monov6 |