016.Kubernetes二进制部署所有节点kube-proxy
程序员文章站
2022-08-03 15:54:23
一 部署 kube-proxy kube-proxy 运行在所有节点上,它监听 apiserver 中 service 和 endpoint 的变化情况,创建路由规则以提供服务 IP 和负载均衡功能。 1.1 安装kube-proxy 提示:k8smaster01节点已下载相应二进制,可直接分发至n ......
一 部署 kube-proxy
kube-proxy 运行在所有节点上,它监听 apiserver 中 service 和 endpoint 的变化情况,创建路由规则以提供服务 ip 和负载均衡功能。
1.1 安装kube-proxy
提示:k8smaster01节点已下载相应二进制,可直接分发至node节点。
1.2 分发kube-proxy
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_ip in ${all_ips[@]} 4 do 5 echo ">>> ${all_ip}" 6 scp kubernetes/server/bin/kube-proxy root@${all_ip}:/opt/k8s/bin/ 7 ssh root@${all_ip} "chmod +x /opt/k8s/bin/*" 8 done
1.3 创建kube-scheduler证书和私钥
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kube-proxy-csr.json <<eof 3 { 4 "cn": "system:kube-proxy", 5 "key": { 6 "algo": "rsa", 7 "size": 2048 8 }, 9 "names": [ 10 { 11 "c": "cn", 12 "st": "shanghai", 13 "l": "shanghai", 14 "o": "k8s", 15 "ou": "system" 16 } 17 ] 18 } 19 eof 20 #创建kube-scheduler的ca证书请求文件
解释:
- cn:指定该证书的 user 为 system:kube-proxy;
- 预定义的 rolebinding system:node-proxier 将user system:kube-proxy 与 role system:node-proxier 绑定,该 role 授予了调用 kube-apiserver proxy 相关 api 的权限;
- 该证书只会被 kube-proxy 当做 client 证书使用,所以 hosts 字段为空。
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \ 3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \ 4 -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy #生成ca密钥(ca-key.pem)和证书(ca.pem)
1.4 创建和分发kubeconfig
kube-proxy 使用 kubeconfig 文件访问 apiserver,该文件提供了 apiserver 地址、嵌入的 ca 证书和 kube-proxy 证书:
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# kubectl config set-cluster kubernetes \ 4 --certificate-authority=/opt/k8s/work/ca.pem \ 5 --embed-certs=true \ 6 --server=${kube_apiserver} \ 7 --kubeconfig=kube-proxy.kubeconfig 8 9 [root@k8smaster01 work]# kubectl config set-credentials kube-proxy \ 10 --client-certificate=kube-proxy.pem \ 11 --client-key=kube-proxy-key.pem \ 12 --embed-certs=true \ 13 --kubeconfig=kube-proxy.kubeconfig 14 15 [root@k8smaster01 work]# kubectl config set-context default \ 16 --cluster=kubernetes \ 17 --user=kube-proxy \ 18 --kubeconfig=kube-proxy.kubeconfig 19 20 [root@k8smaster01 work]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig 21 22 [root@k8smaster01 ~]# cd /opt/k8s/work 23 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 24 [root@k8smaster01 work]# for node_name in ${node_names[@]} 25 do 26 echo ">>> ${node_name}" 27 scp kube-proxy.kubeconfig root@${node_name}:/etc/kubernetes/ 28 done
1.5 创建kube-proxy 配置文件
从 v1.10 开始,kube-proxy 部分参数可以配置文件中配置。可以使用 --write-config-to 选项生成该配置文件。
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kube-proxy-config.yaml.template <<eof 3 kind: kubeproxyconfiguration 4 apiversion: kubeproxy.config.k8s.io/v1alpha1 5 clientconnection: 6 burst: 200 7 kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig" 8 qps: 100 9 bindaddress: ##all_ip## 10 healthzbindaddress: ##all_ip##:10256 11 metricsbindaddress: ##all_ip##:10249 12 enableprofiling: true 13 clustercidr: ${cluster_cidr} 14 hostnameoverride: ##all_name## 15 mode: "ipvs" 16 portrange: "" 17 kubeproxyiptablesconfiguration: 18 masqueradeall: false 19 kubeproxyipvsconfiguration: 20 scheduler: rr 21 excludecidrs: [] 22 eof
解释:
- bindaddress: 监听地址;
- clientconnection.kubeconfig: 连接 apiserver 的 kubeconfig 文件;
- clustercidr: kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量,指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 service ip 的请求做 snat;
- hostnameoverride: 参数值必须与 kubelet 的值一致,否则 kube-proxy 启动后会找不到该 node,从而不会创建任何 ipvs 规则;
- mode: 使用 ipvs 模式。
1.6 分发配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for (( i=0; i < 6; i++ )) 4 do 5 echo ">>> ${all_names[i]}" 6 sed -e "s/##all_name##/${all_names[i]}/" -e "s/##all_ip##/${all_ips[i]}/" kube-proxy-config.yaml.template > kube-proxy-config-${all_names[i]}.yaml.template 7 scp kube-proxy-config-${all_names[i]}.yaml.template root@${all_names[i]}:/etc/kubernetes/kube-proxy-config.yaml 8 done
1.7 创建kube-proxy的systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# cat > kube-proxy.service <<eof 4 [unit] 5 description=kubernetes kube-proxy server 6 documentation=https://github.com/googlecloudplatform/kubernetes 7 after=network.target 8 9 [service] 10 workingdirectory=${k8s_dir}/kube-proxy 11 execstart=/opt/k8s/bin/kube-proxy \\ 12 --config=/etc/kubernetes/kube-proxy-config.yaml \\ 13 --logtostderr=true \\ 14 --v=2 15 restart=on-failure 16 restartsec=5 17 limitnofile=65536 18 19 [install] 20 wantedby=multi-user.target 21 eof
1.8 分发kube-proxy systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_name in ${all_names[@]} 4 do 5 echo ">>> ${all_name}" 6 scp kube-proxy.service root@${all_name}:/etc/systemd/system/ 7 done #分发system
二 启动并验证
2.1 启动kube-proxy 服务
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_ip in ${all_ips[@]} 4 do 5 echo ">>> ${all_ip}" 6 ssh root@${all_ip} "mkdir -p ${k8s_dir}/kube-proxy" 7 ssh root@${all_ip} "modprobe ip_vs_rr" 8 ssh root@${all_ip} "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy" 9 done #启动服务前必须先创建工作目录
2.2 检查kube-proxy 服务
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# for all_ip in ${all_ips[@]} 3 do 4 echo ">>> ${all_ip}" 5 ssh root@${all_ip} "systemctl status kube-proxy|grep active" 6 done
2.3 查看监听端口
kube-proxy 监听 10249 和 10256 端口:
- 10249:对外提供 /metrics;
- 10256:对外提供 /healthz 的访问。
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_ip in ${all_ips[@]} 4 do 5 echo ">>> ${all_ip}" 6 ssh root@${all_ip} "sudo netstat -lnpt|grep kube-prox" 7 done
2.4 查看ipvs 路由规则
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for all_ip in ${all_ips[@]} 4 do 5 echo ">>> ${all_ip}" 6 ssh root@${all_ip} "/usr/sbin/ipvsadm -ln" 7 done
可见所有通过 https 访问 k8s svc kubernetes 的请求都转发到 kube-apiserver 节点的 6443 端口。