欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识！
您现在的位置是： 首页

K8S之路(一) -- Kubeadm搭建K8S集群

程序员文章站 2022-03-12 16:54:39
...

一：摘要概述

Kubernetes简称K8S，8指代中间八个字母，希腊语舵手。大众理解的K8S都是对容器进行管理，Docker另外含义集装箱，命名K8S的初衷就是希望可以对这些集装箱进行管理。有伟大愿景的组织才能做出伟大的贡献，K8S的划时代后续正文中慢慢体会，本文将帮助小白新手搭建K8S集群。别被集群吓到，直白点就是K8S运行环境，只不过多加点节点机器

二：软硬件准备

硬件的相关要求官网上有明确指导，最低要求2核16G。其中Node即工作节点推荐4核，为了一帆风顺本文将使用超高标准操作。但是如果真是小白更建议用虚拟机操作，干净的配置环境对于搭建K8S集群来讲会少很多困扰

*** 操作系统 CPU 内存 IP地址
1 CentOS7 8 128 192.168.15.12
2 CentOS7 8 128 192.168.15.13
3 CentOS7 8 128 192.168.15.14

软件准备无非就是安装包，但是这里只会阐述相关软件版本，下载前请先执行后续步骤再下载。

*** 组件名称 版本 说明
1 kubeadm 1.16.0 初始化Master节点，仅仅需要Master节点安装
2 kubelet 1.16.0 Node节点运行管理，Master与Node都需要安装
3 kubectl 1.16.0 K8S命令工具，Master与Node都可以装上

安装K8S之前每台服务器保证安装Docker Engine，推荐版本18.09。安装完成后修改daemon.json文件，详细的Docker安装请移步Docker安装。修改内容如下，因为Docker默认采用systemd驱动，但是K8S采用cgroupfs驱动，需要将两者统一。统一的方式可以修改Docker，也可以修改K8S。K8S咱还没装好，就统一修改Docker

// 执行命令
vim /etc/docker/daemon.json

// 修改内容
 "exec-opts": [
        "native.cgroupdriver=cgroupfs"
    ]

K8S之路(一) -- Kubeadm搭建K8S集群

三：环境准备

K8S搭建需要做许多相关配置，请务必按要求操作。不要存在侥幸心理，作者的经历就是血淋淋的教训

3.1 关闭防火墙

为这事儿个人绞尽脑汁想保留防火墙，按照公司规定这是必须的防护。但是，随着K8S安装后的应用扩展你会发现天真写你脑门上，还是关了吧。端口太多且有些应用组件通信端口不好找

// CentOS7检查firewall防火墙
systemctl status firewalld.service

// 关闭防火墙
systemctl stop firewalld.service

// 解除开机自启动
systemctl disable firewalld.service
3.2 主机名解析

保证每台机器可以根据主机名字解析出相应的IP地址，首先修改/etc/hostname文件为机器命名，然后修改/etc/hosts文件绑定主机名与IP

*** IP地址 主机名
1 192.168.15.12 k8s-master
2 192.168.15.13 k8s-node1
3 192.168.15.14 k8s-node2 
// 方式1：文件修改
vim /etc/hostname

// 方式2：命令修改,CentOS7提供hostnamectl系列命令。下述命令执行完后查看/etc/hostname文件你就明白了
hostnamectl set-hostname k8s-master

// 最后结果查询,主机名有静态名、瞬态名等，自行了解
方式一：hostname
方拾二：hostnamectl status
3.3 关闭Swap

执行命令free -h可以查看主机内存使用情况，控制台输出形式如下：Mem是真正的物理内存，当这个内存达到设定使用最高比例后开始使用Swap内存，Swap内存是磁盘虚拟内存，详解请自行学习。这里操作就是必须关闭释放禁止使用Swap
K8S之路(一) -- Kubeadm搭建K8S集群

// 临时关闭,开机失效
swapoff -a

// 永久关闭，但都是重启生效。机器不能重启请先使用临时命令操作
方式一：sed -i 's/.*swap.*/#&/' /etc/fstab
方式二：修改文件/etc/fstab，将Swap那行注释
3.4 关闭Selinux

Selinux增强型Linux，2.6集成为Linux模块，Linux的安全子系统。打开可能对后续目录挂载产生影响，部署时将其关闭

// 临时关闭
setenforce 0

// 永久关闭
方式一：sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
	   sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

方式二：vim /etc/selinux/config 和文件 vim /etc/sysconfig/selinux 将SELINUX参数设置为disabled
3.5 关闭Dnsmasp

Dnsmasp一个网络工具，可能导致Docker容器无法解析域名

// 关闭服务
systemctl stop dnsmasp

// 禁止开机自启动
systemctl disable dnsmasp
3.6 设置系统参数
// 执行命令
vim /etc/sysctl.d/k8s.conf 

// 文件内容
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100

// 查看效果
sysctl --system

K8S之路(一) -- Kubeadm搭建K8S集群

3.7 设置系统时钟
// 调整系统时区
timedatectl set-timezone Asia/Shanghai

// UTC时钟与硬件一致
timedatectl set-local-rtc 0
3.8 配置阿里云K8S yum源

K8S默认的下载源是google，国外的肯定墙了过不去。所以yum下载前需要配置国内的yum源

// 执行命令 
vim /etc/yum.repos.d/kubernetes.repo

// 文件内容
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

四：下载安装

请仔细检查是否按照要求在所有主机上执行了三步骤，确保！确保！确保！很重要的事情提示三遍。上述三步骤一定要在所有主机上执行

4.1 Master主机操作

yum 下载安装相关组件

// 安装kubeadm、kubectl、kubelet
yum install -y kubeadm-1.16.0-0 kubectl-1.16.0-0 kubelet-1.16.0-0

启动kublet应用

systemctl enable kubelet && systemctl start kubelet

初始化Master

kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.16.0 --apiserver-advertise-address 自己的Mater主机IP地址 --pod-network-cidr=10.244.0.0/16 --token-ttl 0

后续操作，为了可以使用kubectl客户端命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装通信网络组件flanneld，因为官方的yml下载源在国外，国内请查看第六节的yml文件。这个应用组件用于通信，此时未安装使用命令kubectl get nodes 查看节点状态发现其为NotReady，装完后将会变成Ready才是正确的状态

// 任意位置创建文件
touch kube-flanneld.yml

// 将第五节内容复制粘贴到文件中，并执行下述命令
kubectl apply -f kube-flanneld.yml
4.2 Node操作

yum 下载安装相关组件

// 安装kubeadm、kubectl、kubelet
yum install -y kubectl-1.16.0-0 kubelet-1.16.0-0

加入集群，此时加入集群的命令可以去Master节点上获取。其实初始化完毕后的Master也会进行输出，如果没有记录就去查询即可

// Master节点执行命令，查询加入集群命令
kubeadm token create --print-join-command

// Node节点执行Master节点控制台输出内容即可

如果想在Node节点上操作kubectl命令请将Master节点的$HOME/.kube目录下的config文件拷贝到Node节点的相同位置即可，不然会提示8080什么的相关信息
K8S之路(一) -- Kubeadm搭建K8S集群

五：踩坑小结

5.1 Node节点NotReady

做完所有操作后发现Node节点状态任然是NotReady，此时执行命令kubectl get pods -n kube-system发现有的状态显示不是Running，此时去Node节点查看并没有对应的Docker镜像，处理方式就是将Master节点下载好的相关镜像save然后Node节点load完毕后即可恢复Ready状态。因为本人已经安装好，所以下图所有都是Running
K8S之路(一) -- Kubeadm搭建K8S集群

5.2 网络错误

错误输出

ROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1

修改方法

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

六：kube-flanneld.yml文件

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
    - configMap
    - secret
    - emptyDir
    - hostPath
  allowedHostPaths:
    - pathPrefix: "/etc/cni/net.d"
    - pathPrefix: "/etc/kube-flannel"
    - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups: ['extensions']
    resources: ['podsecuritypolicies']
    verbs: ['use']
    resourceNames: ['psp.flannel.unprivileged']
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - amd64
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-arm64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - arm64
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-arm
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - arm
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-ppc64le
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - ppc64le
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-ppc64le
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-ppc64le
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-s390x
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - s390x
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-s390x
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-s390x
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

上一篇： k8s 1.9二进制版集群+ipvs+coredns

下一篇： 做Excel上传踩的坑

推荐阅读