008.Kubernetes二进制部署Nginx实现高可用
程序员文章站
2022-07-02 13:06:16
一 Nginx代理实现kube-apiserver高可用 1.1 Nginx实现高可用 基于 nginx 代理的 kube-apiserver 高可用方案。 控制节点的 kube-controller-manager、kube-scheduler 是多实例部署,所以只要有一个实例正常,就可以保证高可 ......
一 nginx代理实现kube-apiserver高可用
1.1 nginx实现高可用
基于 nginx 代理的 kube-apiserver 高可用方案。
控制节点的 kube-controller-manager、kube-scheduler 是多实例部署,所以只要有一个实例正常,就可以保证高可用;
集群内的 pod 使用 k8s 服务域名 kubernetes 访问 kube-apiserver, kube-dns 会自动解析出多个 kube-apiserver 节点的 ip,所以也是高可用的;
在每个节点起一个 nginx 进程,后端对接多个 apiserver 实例,nginx 对它们做健康检查和负载均衡;
kubelet、kube-proxy、controller-manager、scheduler 通过本地的 nginx(监听 127.0.0.1)访问 kube-apiserver,从而实现 kube-apiserver 的高可用;
从而基于 nginx 4 层透明代理功能实现 k8s 节点( master 节点和 worker 节点)高可用访问 kube-apiserver 。
1.2 下载编译nginx
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# wget http://nginx.org/download/nginx-1.15.3.tar.gz 3 [root@k8smaster01 work]# tar -xzvf nginx-1.15.3.tar.gz 4 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/ 5 [root@k8smaster01 nginx-1.15.3]# mkdir nginx-prefix 6 [root@k8smaster01 nginx-1.15.3]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module 7 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/ 8 [root@k8smaster01 nginx-1.15.3]# make && make install
解释:
- --with-stream:开启 4 层透明转发(tcp proxy)功能;
- --without-xxx:关闭所有其他功能,这样生成的动态链接二进制程序依赖最小。
- [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/
- [root@k8smaster01 nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v
1.3 验证编译后的nginx
1 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3 2 [root@k8smaster01 nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v 3 nginx version: nginx/1.15.3 4 [root@k8smaster01 nginx-1.15.3]# ldd ./nginx-prefix/sbin/nginx #查看 nginx 动态链接的库 5 linux-vdso.so.1 => (0x00007ffdda980000) 6 libdl.so.2 => /lib64/libdl.so.2 (0x00007feb37300000) 7 libpthread.so.0 => /lib64/libpthread.so.0 (0x00007feb370e4000) 8 libc.so.6 => /lib64/libc.so.6 (0x00007feb36d17000) 9 /lib64/ld-linux-x86-64.so.2 (0x00007feb37504000)
提示:由于只开启了 4 层透明转发功能,所以除了依赖 libc 等操作系统核心 lib 库外,没有对其它 lib 的依赖(如 libz、libssl 等),以便达到精简编译的目的。
1.4 安装和部署nginx
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 4 do 5 echo ">>> ${master_ip}" 6 mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin} 7 done #创建nginx目录 8 [root@k8smaster01 ~]# cd /opt/k8s/work 9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 10 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 11 do 12 echo ">>> ${master_ip}" 13 scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx root@${master_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx 14 ssh root@${master_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*" 15 ssh root@${master_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}" 16 done #分发nginx二进制
1.5 配置nginx 四层透明转发
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kube-nginx.conf <<eof 3 worker_processes 1; 4 5 events { 6 worker_connections 1024; 7 } 8 9 stream { 10 upstream backend { 11 hash $remote_addr consistent; 12 server 172.24.8.71:6443 max_fails=3 fail_timeout=30s; 13 server 172.24.8.72:6443 max_fails=3 fail_timeout=30s; 14 server 172.24.8.73:6443 max_fails=3 fail_timeout=30s; 15 } 16 17 server { 18 listen 127.0.0.1:8443; 19 proxy_connect_timeout 1s; 20 proxy_pass backend; 21 } 22 } 23 eof 24 [root@k8smaster01 ~]# cd /opt/k8s/work 25 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 26 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 27 do 28 echo ">>> ${master_ip}" 29 scp kube-nginx.conf root@${master_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf 30 done #分发nginx四层透明代理配置文件
1.6 配置nginx system
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kube-nginx.service <<eof 3 [unit] 4 description=kube-apiserver nginx proxy 5 after=network.target 6 after=network-online.target 7 wants=network-online.target 8 9 [service] 10 type=forking 11 execstartpre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t 12 execstart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx 13 execreload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload 14 privatetmp=true 15 restart=always 16 restartsec=5 17 startlimitinterval=0 18 limitnofile=65536 19 20 [install] 21 wantedby=multi-user.target 22 eof
1.7 分发nginx systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp kube-nginx.service root@${master_ip}:/etc/systemd/system/ 7 done
二 启动并验证
2.1 启动nginx
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 4 do 5 echo ">>> ${master_ip}" 6 ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx" 7 done
2.2 检查nginx服务
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 4 do 5 echo ">>> ${master_ip}" 6 ssh root@${master_ip} "systemctl status kube-nginx |grep 'active:'" 7 done
推荐阅读
-
MySQL之MHA高可用配置及故障切换实现详细部署步骤
-
详解Keepalived+Nginx实现高可用(HA)
-
Nginx配置upstream实现负载均衡及keepalived实现nginx高可用
-
JAVAEE——宜立方商城03:Nginx负载均衡高可用、Keepalived+Nginx实现主备
-
013.Kubernetes二进制部署worker节点Nginx实现高可用
-
Redis Sentinel安装与部署,实现redis的高可用
-
Keepalived+Nginx实现高可用
-
Nginx+Keepalived实现高可用解决方案
-
Keepalived+Nginx实现负载均衡高可用
-
企业级自动化部署方案——ansible实现lvs+keepalived高可用架构