欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  科技

010.Kubernetes二进制部署kube-controller-manager

程序员文章站 2022-08-20 08:11:16
一 部署高可用kube-controller-manager 1.1 高可用kube-controller-manager介绍 本实验部署一个三实例 kube-controller-manager 的集群,启动后将通过竞争选举机制产生一个 leader 节点,其它节点为阻塞状态。当 leader 节 ......

一 部署高可用kube-controller-manager

1.1 高可用kube-controller-manager介绍

本实验部署一个三实例 kube-controller-manager 的集群,启动后将通过竞争选举机制产生一个 leader 节点,其它节点为阻塞状态。当 leader 节点不可用时,阻塞的节点将再次进行选举产生新的 leader 节点,从而保证服务的可用性。
为保证通信安全,本文档先生成 x509 证书和私钥,kube-controller-manager 在如下两种情况下使用该证书:
  • 与 kube-apiserver 的安全端口通信;
  • 在安全端口(https,10252) 输出 prometheus 格式的 metrics。

1.2 创建kube-controller-manager证书和私钥

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# cat > kube-controller-manager-csr.json <<eof
  3 {
  4   "cn": "system:kube-controller-manager",
  5   "hosts": [
  6     "127.0.0.1",
  7     "172.24.8.71",
  8     "172.24.8.72",
  9     "172.24.8.73"
 10   ],
 11   "key": {
 12     "algo": "rsa",
 13     "size": 2048
 14   },
 15   "names": [
 16     {
 17       "c": "cn",
 18       "st": "shanghai",
 19       "l": "shanghai",
 20       "o": "system:kube-controller-manager",
 21       "ou": "system"
 22     }
 23   ]
 24 }
 25 eof
 26 #创建kube-controller-manager的ca证书请求文件
解释:
hosts 列表包含所有 kube-controller-manager 节点 ip;
cn 和 o 均为 system:kube-controller-manager,kubernetes 内置的 clusterrolebindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
  3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \
  4 -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager	#生成ca密钥(ca-key.pem)和证书(ca.pem)

1.3 分发证书和私钥

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp kube-controller-manager*.pem root@${master_ip}:/etc/kubernetes/cert/
  7   done

1.4 创建和分发kubeconfig

kube-controller-manager 使用 kubeconfig 文件访问 apiserver,该文件提供了 apiserver 地址、嵌入的 ca 证书和 kube-controller-manager 证书:
  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# kubectl config set-cluster kubernetes \
  4   --certificate-authority=/opt/k8s/work/ca.pem \
  5   --embed-certs=true \
  6   --server=${kube_apiserver} \
  7   --kubeconfig=kube-controller-manager.kubeconfig
  8 
  9 [root@k8smaster01 work]# kubectl config set-credentials system:kube-controller-manager \
 10   --client-certificate=kube-controller-manager.pem \
 11   --client-key=kube-controller-manager-key.pem \
 12   --embed-certs=true \
 13   --kubeconfig=kube-controller-manager.kubeconfig
 14 
 15 [root@k8smaster01 work]# kubectl config set-context system:kube-controller-manager \
 16   --cluster=kubernetes \
 17   --user=system:kube-controller-manager \
 18   --kubeconfig=kube-controller-manager.kubeconfig
 19 
 20 [root@k8smaster01 work]# kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
 21 
 22 [root@k8smaster01 ~]# cd /opt/k8s/work
 23 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
 24 [root@k8smaster01 work]# for master_ip in ${master_ips[@]}
 25   do
 26     echo ">>> ${master_ip}"
 27     scp kube-controller-manager.kubeconfig root@${master_ip}:/etc/kubernetes/
 28   done

1.5 创建kube-controller-manager的systemd

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# cat > kube-controller-manager.service.template <<eof
  4 [unit]
  5 description=kubernetes controller manager
  6 documentation=https://github.com/googlecloudplatform/kubernetes
  7 
  8 [service]
  9 workingdirectory=${k8s_dir}/kube-controller-manager
 10 execstart=/opt/k8s/bin/kube-controller-manager \\
 11   --profiling \\
 12   --cluster-name=kubernetes \\
 13   --controllers=*,bootstrapsigner,tokencleaner \\
 14   --kube-api-qps=1000 \\
 15   --kube-api-burst=2000 \\
 16   --leader-elect \\
 17   --use-service-account-credentials\\
 18   --concurrent-service-syncs=2 \\
 19   --bind-address=##master_ip## \\
 20   --secure-port=10252 \\
 21   --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\
 22   --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\
 23   --port=0 \\
 24   --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
 25   --client-ca-file=/etc/kubernetes/cert/ca.pem \\
 26   --requestheader-allowed-names="" \\
 27   --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
 28   --requestheader-extra-headers-prefix="x-remote-extra-" \\
 29   --requestheader-group-headers=x-remote-group \\
 30   --requestheader-username-headers=x-remote-user \\
 31   --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
 32   --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\
 33   --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\
 34   --experimental-cluster-signing-duration=8760h \\
 35   --horizontal-pod-autoscaler-sync-period=10s \\
 36   --concurrent-deployment-syncs=10 \\
 37   --concurrent-gc-syncs=30 \\
 38   --node-cidr-mask-size=24 \\
 39   --service-cluster-ip-range=${service_cidr} \\
 40   --pod-eviction-timeout=6m \\
 41   --terminated-pod-gc-threshold=10000 \\
 42   --root-ca-file=/etc/kubernetes/cert/ca.pem \\
 43   --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\
 44   --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
 45   --logtostderr=true \\
 46   --v=2
 47 restart=on-failure
 48 restartsec=5
 49 
 50 [install]
 51 wantedby=multi-user.target
 52 eof

1.6 分发systemd

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for (( i=0; i < 3; i++ ))
  4   do
  5     sed -e "s/##master_name##/${master_names[i]}/" -e "s/##master_ip##/${master_ips[i]}/" kube-controller-manager.service.template > kube-controller-manager-${master_ips[i]}.service
  6   done						#修正相应ip
  7 [root@k8smaster01 work]# ls kube-controller-manager*.service
  8 [root@k8smaster01 ~]# cd /opt/k8s/work
  9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
 10 [root@k8smaster01 work]# for master_ip in ${master_ips[@]}
 11   do
 12     echo ">>> ${master_ip}"
 13     scp kube-controller-manager-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-controller-manager.service
 14   done						#分发system

二 启动并验证

2.1 启动kube-controller-manager 服务

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "mkdir -p ${k8s_dir}/kube-controller-manager"
  7     ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
  8   done

2.2 检查kube-controller-manager 服务

  1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh
  2 [root@k8smaster01 ~]# for master_ip in ${master_ips[@]}
  3   do
  4     echo ">>> ${master_ip}"
  5     ssh root@${master_ip} "systemctl status kube-controller-manager|grep active"
  6   done
010.Kubernetes二进制部署kube-controller-manager

2.3 查看输出的 metrics

  1 [root@k8smaster01 ~]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.24.8.71:10252/metrics |head
注意:以上命令在 kube-controller-manager 节点上执行。

2.4 查看权限

  1 [root@k8smaster01 ~]# kubectl describe clusterrole system:kube-controller-manager
010.Kubernetes二进制部署kube-controller-manager
clusterole system:kube-controller-manager 的权限很小,只能创建 secret、serviceaccount 等资源对象,各 controller 的权限分散到 clusterrole system:controller:xxx 中。
当在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数,这样 main controller 会为各 controller 创建对应的 serviceaccount xxx-controller。内置的 clusterrolebinding system:controller:xxx 将赋予各 xxx-controller serviceaccount 对应的 clusterrole system:controller:xxx 权限。
  1 [root@k8smaster01 ~]# kubectl get clusterrole|grep controller
010.Kubernetes二进制部署kube-controller-manager
如deployment controller:
  1 [root@k8smaster01 ~]# kubectl describe clusterrole system:controller:deployment-controller
010.Kubernetes二进制部署kube-controller-manager

2.5 查看当前leader

  1 [root@k8smaster01 ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml
010.Kubernetes二进制部署kube-controller-manager
kubelet 认证和授权:https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization