搭建k8s集群
程序员文章站
2022-06-12 15:33:06
...
搭建k8s集群
安装虚拟机
- 在Windows物理机上安装3台CentOS虚拟机
- 软件:VMware Workstation Pro
- 镜像:CentOS-7-x86_64-DVD-2003.iso
- 规划:一个master节点+两个node节点
- 步骤参考博客,推荐阅读
1.虚拟机引导设置
2.CentOS设置
-
地区选上海
-
磁盘设置
-
查看VMWare配置
-
查看物理机网关
-
配置IPv4
-
IP获取方式从DHCP改为Manual;
-
然后添加ip地址,注意自定义的ip要和上面NAT模式下的子网ip同个网段;
-
最后添加DNS服务器,设置为物理机的网关即可,不设置的话安装后无法解析域名。
-
进入系统
-
查看网络设置
cat /etc/sysconfig/network-scripts/ifcfg-ens33
-
测试网络
ping www.baidu.com
-
安装必要工具
yum install net-tools yum install vim
-
安装docker:参考官网指引
-
配置阿里云容器镜像服务的镜像加速器(需要开通阿里云账号)
sudo mkdir -p /etc/docker sudo tee /etc/docker/daemon.json <<-'EOF' { "registry-mirrors": ["https://xxxxxxx.mirror.aliyuncs.com"] } EOF sudo systemctl daemon-reload sudo systemctl restart docker
-
设置docker开启启动
systemctl enable docker usermod -aG docker your_username reboot
3.通过克隆创建两个node节点
-
master节点先关机
-
IP地址改为151
vim /etc/sysconfig/network-scripts/ifcfg-ens33
-
修改主机名为node1
hostnamectl set-hostname node1
-
node2节点同理
通过kubeadm部署k8s
- 参考博客,推荐阅读
1.环境配置
- 所有节点执行以下命令
systemctl stop firewalld
systemctl disable firewalld
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
swapoff -a # 临时关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab #永久关闭
echo \
"192.168.65.150 master
192.168.65.151 node1
192.168.65.152 node2"\
>> /etc/hosts
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
ntpdate -u cn.pool.ntp.org
2.安装kubeadm,kubelet和kubectl
- 所有节点执行以下命令
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet
- 可以看到安装的kubernetes版本是19.3.0
Installed: kubeadm.x86_64 0:1.19.3-0 kubectl.x86_64 0:1.19.3-0 kubelet.x86_64 0:1.19.3-0
Dependency Installed: conntrack-tools.x86_64 0:1.4.4-7.el7
cri-tools.x86_64 0:1.13.0-0
kubernetes-cni.x86_64 0:0.8.7-0
libnetfilter_cthelper.x86_64 0:1.0.0-11.el7
libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7
libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7
3.部署master节点
- 在192.168.65.150上执行以下命令,注意更换address和version
kubeadm init \
--apiserver-advertise-address=192.168.65.150 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.19.3 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16
- 看到下面的信息表示初始化成功
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.65.150:6443 --token qggque.d86p6jbt8j8sfa83 \
--discovery-token-ca-cert-hash sha256:c6087fe38da78e9fe20cfa29e7d9a36601acc028c0833e80c4f60f51512cb92d
- 按照上面的提示执行
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 安装Pod网络插件(CNI)
kubectl apply -f kube-flannel.yml
- kube-flannel.yml文件可以从这个链接下载,修改改文件,添加网卡 iface=ens33
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.10.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=ens33
- 具体内容如下
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.13.0-rc2
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.13.0-rc2
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=ens33
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
4.部署node节点
- 拉取镜像
docker pull lizhenliang/flannel:v0.11.0-amd64
- node节点加入集群,命令从上面kubeadm init的输出提示复制
kubeadm join 192.168.65.150:6443 --token qggque.d86p6jbt8j8sfa83 \
--discovery-token-ca-cert-hash sha256:c6087fe38da78e9fe20cfa29e7d9a36601acc028c0833e80c4f60f51512cb92d
- 看见下面的提示即成功加入
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
5.准备完成
kubectl get pods -n kube-system
[aaa@qq.com manifests]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6d56c8448f-d7pjq 1/1 Running 0 43m
coredns-6d56c8448f-mbbbh 1/1 Running 0 43m
etcd-master 1/1 Running 0 43m
kube-apiserver-master 1/1 Running 0 43m
kube-controller-manager-master 1/1 Running 0 13m
kube-flannel-ds-7frtd 1/1 Running 1 27m
kube-flannel-ds-nk97s 1/1 Running 0 27m
kube-flannel-ds-wt4bp 1/1 Running 0 32m
kube-proxy-8km29 1/1 Running 2 27m
kube-proxy-phz9c 1/1 Running 0 27m
kube-proxy-xxtqp 1/1 Running 0 43m
kube-scheduler-master 1/1 Running 0 12m
6.测试集群
# 创建nginx容器
kubectl create deployment nginx --image=nginx
# 暴露对外端口
kubectl expose deployment nginx --port=80 --type=NodePort
# 查看nginx是否运行成功
kubectl get pod,svc
[aaa@qq.com manifests]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[aaa@qq.com manifests]# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
[aaa@qq.com manifests]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-6799fc88d8-v7pzq 0/1 ContainerCreating 0 13s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 47m
service/nginx NodePort 10.1.220.85 <none> 80:30966/TCP 6s
[aaa@qq.com manifests]#
- 可以看到Nginx对外服务的端口是30966,在浏览器分别访问:http://192.168.65.150:30966/,http://192.168.65.151:30966/,http://192.168.65.152:30966/,均可以访问到nginx
- 再测试一下扩容
[aaa@qq.com manifests]# kubectl scale deployment nginx --replicas=3
deployment.apps/nginx scaled
[aaa@qq.com manifests]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-rdv2f 0/1 ContainerCreating 0 4s
nginx-6799fc88d8-v7pzq 1/1 Running 0 4m49s
nginx-6799fc88d8-w6xbh 0/1 ContainerCreating 0 4s
kubectl get cs 报错
[aaa@qq.com k8s]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused
etcd-0 Healthy {"health":"true"}
- 解决方法:参考博客,修改master节点配置文件
vim /etc/kubernetes/manifests/kube-controller-manager.yaml
# 注释掉port=0这一行
vim /etc/kubernetes/manifests/kube-scheduler.yaml
# 注释掉port=0这一行
- 三个节点重启kubelet
systemctl restart kubelet.service
- 主节点再次执行
kubectl get cs
[aaa@qq.com manifests]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
kubectl get node报错
- 有节点not ready
[aaa@qq.com k8s]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready master 24m v1.19.3
node1 NotReady <none> 7m59s v1.19.3
node2 Ready <none> 8m2s v1.19.3
- 可能是防火墙的问题,参考博客
- 在问题节点上执行
systemctl status firewalld
systemctl stop firewalld
systemctl enable firewalld
systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet
- 主节点再次执行
kubectl get node
[aaa@qq.com manifests]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready master 39m v1.19.3
node1 Ready <none> 22m v1.19.3
node2 Ready <none> 22m v1.19.3
部署Dashboard
kubectl apply -f kubernetes-dashboard.yaml
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
nodePort: 30000
selector:
k8s-app: kubernetes-dashboard
type: NodePort
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kubernetes-dashboard
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
namespace: kubernetes-dashboard
type: Opaque
data:
csrf: ""
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-key-holder
namespace: kubernetes-dashboard
type: Opaque
---
kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
namespace: kubernetes-dashboard
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
rules:
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster", "dashboard-metrics-scraper"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: kubernetesui/dashboard:v2.0.0-beta4
imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
- --namespace=kubernetes-dashboard
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: dashboard-metrics-scraper
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
spec:
containers:
- name: dashboard-metrics-scraper
image: kubernetesui/metrics-scraper:v1.0.1
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
serviceAccountName: kubernetes-dashboard
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}
- 查看暴露的端口
[aaa@qq.com k8s]# kubectl get pods,svc -n kube-system
NAME READY STATUS RESTARTS AGE
pod/coredns-6d56c8448f-d7pjq 1/1 Running 0 75m
pod/coredns-6d56c8448f-mbbbh 1/1 Running 0 75m
pod/etcd-master 1/1 Running 0 75m
pod/kube-apiserver-master 1/1 Running 0 75m
pod/kube-controller-manager-master 1/1 Running 0 44m
pod/kube-flannel-ds-7frtd 1/1 Running 1 58m
pod/kube-flannel-ds-nk97s 1/1 Running 0 58m
pod/kube-flannel-ds-wt4bp 1/1 Running 0 64m
pod/kube-proxy-8km29 1/1 Running 2 58m
pod/kube-proxy-phz9c 1/1 Running 0 58m
pod/kube-proxy-xxtqp 1/1 Running 0 75m
pod/kube-scheduler-master 1/1 Running 0 44m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.1.0.10 <none> 53/UDP,53/TCP,9153/TCP 75m
- 创建service account并绑定默认cluster-admin管理员集群角色
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
-
记住token
-
浏览器访问https://192.168.65.150:30000/,发现因为证书问题,谷歌浏览器无法访问,而火狐浏览器可以,参考博客
-
输入token,成功访问
-
重新安装dashboard
kubectl delete -f kubernetes-dashboard.yaml
kubectl create -f kubernetes-dashboard.yaml
其他踩过的坑
- https://blog.csdn.net/woay2008/article/details/93250137
- https://blog.csdn.net/weixin_44723434/article/details/94583457
- https://*.com/questions/61305498/kubernetes-couldnt-able-to-join-master-node-error-execution-phase-preflight