009.Kubernetes二进制部署kube-apiserver
程序员文章站
2022-04-21 18:08:24
一 部署master节点 1.1 master节点服务 kubernetes master 节点运行如下组件: kube-apiserver kube-scheduler kube-controller-manager kube-nginx kube-apiserver、kube-scheduler ......
一 部署master节点
1.1 master节点服务
kubernetes master 节点运行如下组件:
- kube-apiserver
- kube-scheduler
- kube-controller-manager
- kube-nginx
kube-apiserver、kube-scheduler 和 kube-controller-manager 均以多实例模式运行:
kube-scheduler 和 kube-controller-manager 会自动选举产生一个 leader 实例,其它实例处于阻塞模式,当 leader 挂了后,重新选举产生新的 leader,从而保证服务可用性。
kube-apiserver 是无状态的,需要通过 kube-nginx 进行代理访问,从而保证服务可用性。
1.2 安装kubernetes
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# wget https://dl.k8s.io/v1.14.2/kubernetes-server-linux-amd64.tar.gz 3 [root@k8smaster01 work]# tar -xzvf kubernetes-server-linux-amd64.tar.gz 4 [root@k8smaster01 work]# cd kubernetes 5 [root@k8smaster01 kubernetes]# tar -xzvf kubernetes-src.tar.gz
1.3 分发kubernetes
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp kubernetes/server/bin/{apiextensions-apiserver,cloud-controller-manager,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${master_ip}:/opt/k8s/bin/ 7 ssh root@${master_ip} "chmod +x /opt/k8s/bin/*" 8 done
二 部署高可用kube-apiserver
2.1 高可用apiserver介绍
本实验部署一个三实例 kube-apiserver 集群的步骤,它们通过 kube-nginx 进行代理访问,从而保证服务可用性。
2.2 创建kubernetes证书和私钥
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kubernetes-csr.json <<eof 3 { 4 "cn": "kubernetes", 5 "hosts": [ 6 "127.0.0.1", 7 "172.24.8.71", 8 "172.24.8.72", 9 "172.24.8.73", 10 "${cluster_kubernetes_svc_ip}", 11 "kubernetes", 12 "kubernetes.default", 13 "kubernetes.default.svc", 14 "kubernetes.default.svc.cluster", 15 "kubernetes.default.svc.cluster.local." 16 ], 17 "key": { 18 "algo": "rsa", 19 "size": 2048 20 }, 21 "names": [ 22 { 23 "c": "cn", 24 "st": "shanghai", 25 "l": "shanghai", 26 "o": "k8s", 27 "ou": "system" 28 } 29 ] 30 } 31 eof 32 #创建kubernetes的ca证书请求文件
解释:
hosts 字段指定授权使用该证书的 ip 和域名列表,这里列出了 master 节点 ip、kubernetes 服务的 ip 和域名;
kubernetes 服务 ip 是 apiserver 自动创建的,一般是 --service-cluster-ip-range 参数指定的网段的第一个ip,后续可以通过下面命令获取:
1 # kubectl get svc kubernetes
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \ 3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \ 4 -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes #生成ca密钥(ca-key.pem)和证书(ca.pem)
2.3 分发证书和私钥
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 4 do 5 echo ">>> ${master_ip}" 6 ssh root@${master_ip} "mkdir -p /etc/kubernetes/cert" 7 scp kubernetes*.pem root@${master_ip}:/etc/kubernetes/cert/ 8 done
2.4 创建加密配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# cat > encryption-config.yaml <<eof 4 kind: encryptionconfig 5 apiversion: v1 6 resources: 7 - resources: 8 - secrets 9 providers: 10 - aescbc: 11 keys: 12 - name: key1 13 secret: ${encryption_key} 14 - identity: {} 15 eof
2.5 分发加密配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp encryption-config.yaml root@${master_ip}:/etc/kubernetes/ 7 done
2.6 创建审计策略文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# cat > audit-policy.yaml <<eof 4 apiversion: audit.k8s.io/v1beta1 5 kind: policy 6 rules: 7 # the following requests were manually identified as high-volume and low-risk, so drop them. 8 - level: none 9 resources: 10 - group: "" 11 resources: 12 - endpoints 13 - services 14 - services/status 15 users: 16 - 'system:kube-proxy' 17 verbs: 18 - watch 19 20 - level: none 21 resources: 22 - group: "" 23 resources: 24 - nodes 25 - nodes/status 26 usergroups: 27 - 'system:nodes' 28 verbs: 29 - get 30 31 - level: none 32 namespaces: 33 - kube-system 34 resources: 35 - group: "" 36 resources: 37 - endpoints 38 users: 39 - 'system:kube-controller-manager' 40 - 'system:kube-scheduler' 41 - 'system:serviceaccount:kube-system:endpoint-controller' 42 verbs: 43 - get 44 - update 45 46 - level: none 47 resources: 48 - group: "" 49 resources: 50 - namespaces 51 - namespaces/status 52 - namespaces/finalize 53 users: 54 - 'system:apiserver' 55 verbs: 56 - get 57 58 # don't log hpa fetching metrics. 59 - level: none 60 resources: 61 - group: metrics.k8s.io 62 users: 63 - 'system:kube-controller-manager' 64 verbs: 65 - get 66 - list 67 68 # don't log these read-only urls. 69 - level: none 70 nonresourceurls: 71 - '/healthz*' 72 - /version 73 - '/swagger*' 74 75 # don't log events requests. 76 - level: none 77 resources: 78 - group: "" 79 resources: 80 - events 81 82 # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes 83 - level: request 84 omitstages: 85 - requestreceived 86 resources: 87 - group: "" 88 resources: 89 - nodes/status 90 - pods/status 91 users: 92 - kubelet 93 - 'system:node-problem-detector' 94 - 'system:serviceaccount:kube-system:node-problem-detector' 95 verbs: 96 - update 97 - patch 98 99 - level: request 100 omitstages: 101 - requestreceived 102 resources: 103 - group: "" 104 resources: 105 - nodes/status 106 - pods/status 107 usergroups: 108 - 'system:nodes' 109 verbs: 110 - update 111 - patch 112 113 # deletecollection calls can be large, don't log responses for expected namespace deletions 114 - level: request 115 omitstages: 116 - requestreceived 117 users: 118 - 'system:serviceaccount:kube-system:namespace-controller' 119 verbs: 120 - deletecollection 121 122 # secrets, configmaps, and tokenreviews can contain sensitive & binary data, 123 # so only log at the metadata level. 124 - level: metadata 125 omitstages: 126 - requestreceived 127 resources: 128 - group: "" 129 resources: 130 - secrets 131 - configmaps 132 - group: authentication.k8s.io 133 resources: 134 - tokenreviews 135 # get repsonses can be large; skip them. 136 - level: request 137 omitstages: 138 - requestreceived 139 resources: 140 - group: "" 141 - group: admissionregistration.k8s.io 142 - group: apiextensions.k8s.io 143 - group: apiregistration.k8s.io 144 - group: apps 145 - group: authentication.k8s.io 146 - group: authorization.k8s.io 147 - group: autoscaling 148 - group: batch 149 - group: certificates.k8s.io 150 - group: extensions 151 - group: metrics.k8s.io 152 - group: networking.k8s.io 153 - group: policy 154 - group: rbac.authorization.k8s.io 155 - group: scheduling.k8s.io 156 - group: settings.k8s.io 157 - group: storage.k8s.io 158 verbs: 159 - get 160 - list 161 - watch 162 163 # default level for known apis 164 - level: requestresponse 165 omitstages: 166 - requestreceived 167 resources: 168 - group: "" 169 - group: admissionregistration.k8s.io 170 - group: apiextensions.k8s.io 171 - group: apiregistration.k8s.io 172 - group: apps 173 - group: authentication.k8s.io 174 - group: authorization.k8s.io 175 - group: autoscaling 176 - group: batch 177 - group: certificates.k8s.io 178 - group: extensions 179 - group: metrics.k8s.io 180 - group: networking.k8s.io 181 - group: policy 182 - group: rbac.authorization.k8s.io 183 - group: scheduling.k8s.io 184 - group: settings.k8s.io 185 - group: storage.k8s.io 186 187 # default level for all other requests. 188 - level: metadata 189 omitstages: 190 - requestreceived 191 eof
2.7 分发策略文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp audit-policy.yaml root@${master_ip}:/etc/kubernetes/audit-policy.yaml 7 done
2.8 创建访问 metrics-server的证书和密钥
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > proxy-client-csr.json <<eof 3 { 4 "cn": "aggregator", 5 "hosts": [], 6 "key": { 7 "algo": "rsa", 8 "size": 2048 9 }, 10 "names": [ 11 { 12 "c": "cn", 13 "st": "shanghai", 14 "l": "shanghai", 15 "o": "k8s", 16 "ou": "system" 17 } 18 ] 19 } 20 eof 21 #创建metrics-server的ca证书请求文件
解释:
cn 名称需要位于 kube-apiserver 的 --requestheader-allowed-names 参数中,否则后续访问 metrics 时会提示权限不足。
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \ 3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \ 4 -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client #生成ca密钥(ca-key.pem)和证书(ca.pem)
2.9 分发证书和私钥
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp proxy-client*.pem root@${master_ip}:/etc/kubernetes/cert/ 7 done
2.10 创建kube-apiserver的systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# cat > kube-apiserver.service.template <<eof 4 [unit] 5 description=kubernetes api server 6 documentation=https://github.com/googlecloudplatform/kubernetes 7 after=network.target 8 9 [service] 10 workingdirectory=${k8s_dir}/kube-apiserver 11 execstart=/opt/k8s/bin/kube-apiserver \\ 12 --advertise-address=##master_ip## \\ 13 --default-not-ready-toleration-seconds=360 \\ 14 --default-unreachable-toleration-seconds=360 \\ 15 --feature-gates=dynamicauditing=true \\ 16 --max-mutating-requests-inflight=2000 \\ 17 --max-requests-inflight=4000 \\ 18 --default-watch-cache-size=200 \\ 19 --delete-collection-workers=2 \\ 20 --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\ 21 --etcd-cafile=/etc/kubernetes/cert/ca.pem \\ 22 --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\ 23 --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\ 24 --etcd-servers=${etcd_endpoints} \\ 25 --bind-address=##master_ip## \\ 26 --secure-port=6443 \\ 27 --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\ 28 --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\ 29 --insecure-port=0 \\ 30 --audit-dynamic-configuration \\ 31 --audit-log-maxage=15 \\ 32 --audit-log-maxbackup=3 \\ 33 --audit-log-maxsize=100 \\ 34 --audit-log-mode=batch \\ 35 --audit-log-truncate-enabled \\ 36 --audit-log-batch-buffer-size=20000 \\ 37 --audit-log-batch-max-size=2 \\ 38 --audit-log-path=${k8s_dir}/kube-apiserver/audit.log \\ 39 --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\ 40 --profiling \\ 41 --anonymous-auth=false \\ 42 --client-ca-file=/etc/kubernetes/cert/ca.pem \\ 43 --enable-bootstrap-token-auth \\ 44 --requestheader-allowed-names="aggregator" \\ 45 --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\ 46 --requestheader-extra-headers-prefix="x-remote-extra-" \\ 47 --requestheader-group-headers=x-remote-group \\ 48 --requestheader-username-headers=x-remote-user \\ 49 --service-account-key-file=/etc/kubernetes/cert/ca.pem \\ 50 --authorization-mode=node,rbac \\ 51 --runtime-config=api/all=true \\ 52 --enable-admission-plugins=noderestriction \\ 53 --allow-privileged=true \\ 54 --apiserver-count=3 \\ 55 --event-ttl=168h \\ 56 --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\ 57 --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\ 58 --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\ 59 --kubelet-https=true \\ 60 --kubelet-timeout=10s \\ 61 --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\ 62 --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\ 63 --service-cluster-ip-range=${service_cidr} \\ 64 --service-node-port-range=${node_port_range} \\ 65 --logtostderr=true \\ 66 --v=2 67 restart=on-failure 68 restartsec=10 69 type=notify 70 limitnofile=65536 71 72 [install] 73 wantedby=multi-user.target 74 eof
解释:
- --advertise-address:apiserver 对外通告的 ip(kubernetes 服务后端节点 ip);
- --default-*-toleration-seconds:设置节点异常相关的阈值;
- --max-*-requests-inflight:请求相关的最大阈值;
- --etcd-*:访问 etcd 的证书和 etcd 服务器地址;
- --experimental-encryption-provider-config:指定用于加密 etcd 中 secret 的配置;
- --bind-address: https 监听的 ip,不能为 127.0.0.1,否则外界不能访问它的安全端口 6443;
- --secret-port:https 监听端口;
- --insecure-port=0:关闭监听 http 非安全端口(8080);
- --tls-*-file:指定 apiserver 使用的证书、私钥和 ca 文件;
- --audit-*:配置审计策略和审计日志文件相关的参数;
- --client-ca-file:验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书;
- --enable-bootstrap-token-auth:启用 kubelet bootstrap 的 token 认证;
- --requestheader-*:kube-apiserver 的 aggregator layer 相关的配置参数,proxy-client & hpa 需要使用;
- --requestheader-client-ca-file:用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书;在启用了 metric aggregator 时使用;
- --requestheader-allowed-names:不能为空,值为逗号分割的 --proxy-client-cert-file 证书的 cn 名称,这里设置为 "aggregator";
- --service-account-key-file:签名 serviceaccount token 的公钥文件,kube-controller-manager 的 --service-account-private-key-file 指定私钥文件,两者配对使用;
- --runtime-config=api/all=true: 启用所有版本的 apis,如 autoscaling/v2alpha1;
- --authorization-mode=node,rbac、--anonymous-auth=false: 开启 node 和 rbac 授权模式,拒绝未授权的请求;
- --enable-admission-plugins:启用一些默认关闭的 plugins;
- --allow-privileged:运行执行 privileged 权限的容器;
- --apiserver-count=3:指定 apiserver 实例的数量;
- --event-ttl:指定 events 的保存时间;
- --kubelet-*:如果指定,则使用 https 访问 kubelet apis;需要为证书对应的用户(上面 kubernetes*.pem 证书的用户为 kubernetes) 用户定义 rbac 规则,否则访问 kubelet api 时提示未授权;
- --proxy-client-*:apiserver 访问 metrics-server 使用的证书;
- --service-cluster-ip-range: 指定 service cluster ip 地址段;
- --service-node-port-range: 指定 nodeport 的端口范围。
提示:如果 kube-apiserver 机器没有运行 kube-proxy,则还需要添加 --enable-aggregator-routing=true 参数。
注意:requestheader-client-ca-file 指定的 ca 证书,必须具有 client auth and server auth;
如果 --requestheader-allowed-names 为空,或者 --proxy-client-cert-file 证书的 cn 名称不在 allowed-names 中,则后续查看 node 或 pods 的 metrics 失败,提示:
1 [root@zhangjun-k8s01 1.8+]# kubectl top nodes 2 error from server (forbidden): nodes.metrics.k8s.io is forbidden: user "aggregator" cannot list resource "nodes" in api group "metrics.k8s.io" at the cluster scope 3
2.11 分发systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for (( i=0; i < 3; i++ )) 4 do 5 sed -e "s/##master_name##/${master_names[i]}/" -e "s/##master_ip##/${master_ips[i]}/" kube-apiserver.service.template > kube-apiserver-${master_ips[i]}.service 6 done 7 [root@k8smaster01 work]# ls kube-apiserver*.service #替换相应的ip 8 [root@k8smaster01 ~]# cd /opt/k8s/work 9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 10 [root@k8smaster01 work]# for master_ip in ${master_ips[@]} 11 do 12 echo ">>> ${master_ip}" 13 scp kube-apiserver-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-apiserver.service 14 done #分发systemd
三 启动并验证
3.1 启动kube-apiserver服务
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# for master_ip in ${master_ips[@]} 3 do 4 echo ">>> ${master_ip}" 5 ssh root@${master_ip} "mkdir -p ${k8s_dir}/kube-apiserver" 6 ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver" 7 done
3.2 检查kube-apiserver服务
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# for master_ip in ${master_ips[@]} 3 do 4 echo ">>> ${master_ip}" 5 ssh root@${master_ip} "systemctl status kube-apiserver |grep 'active:'" 6 done
3.3 查看kube-apiserver写入etcd的数据
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# etcdctl_api=3 etcdctl \ 3 --endpoints=${etcd_endpoints} \ 4 --cacert=/opt/k8s/work/ca.pem \ 5 --cert=/opt/k8s/work/etcd.pem \ 6 --key=/opt/k8s/work/etcd-key.pem \ 7 get /registry/ --prefix --keys-only
3.4 检查集群信息
1 [root@k8smaster01 ~]# kubectl cluster-info 2 [root@k8smaster01 ~]# kubectl get all --all-namespaces 3 [root@k8smaster01 ~]# kubectl get componentstatuses 4 [root@k8smaster01 ~]# sudo netstat -lnpt|grep kube #检查 kube-apiserver 监听的端口
提示:
如果执行 kubectl 命令式时输出如下错误信息,则说明使用的 ~/.kube/config 文件不对,请切换到正确的账户后再执行该命令:
the connection to the server localhost:8080 was refused - did you specify the right host or port?
执行 kubectl get componentstatuses 命令时,apiserver 默认向 127.0.0.1 发送请求。当 controller-manager、scheduler 以集群模式运行时,有可能和 kube-apiserver 不在一台机器上,这时 controller-manager 或 scheduler 的状态为 unhealthy,但实际上它们工作正常;
6443: 接收 https 请求的安全端口,对所有请求做认证和授权;
由于关闭了非安全端口,故没有监听 8080。
3.5 授权
授予 kube-apiserver 访问 kubelet api 的权限。
在执行 kubectl exec、run、logs 等命令时,apiserver 会将请求转发到 kubelet 的 https 端口。本实验定义 rbac 规则,授权 apiserver 使用的证书(kubernetes.pem)用户名(cn:kuberntes)访问 kubelet api 的权限:
1 [root@k8smaster01 ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
推荐阅读
-
Centos7.5安装mysql5.7.24二进制包方式部署
-
009.Kubernetes二进制部署kube-apiserver
-
012.Kubernetes二进制部署worker节点Flannel
-
003.Kubernetes二进制部署准备
-
004.Kubernetes二进制部署创建证书
-
006.Kubernetes二进制部署ETCD
-
005.Kubernetes二进制部署kubectl
-
010.Kubernetes二进制部署kube-controller-manager
-
014.Kubernetes二进制部署docker
-
013.Kubernetes二进制部署worker节点Nginx实现高可用