欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

k8s学习(一) centos7下搭建k8s集群

程序员文章站 2022-03-01 13:01:44
...

一 环境准备

1 三台机器,还需要一台docker镜像服务器

 	master   192.168.100.89  
	node2    192.168.100.91   
	node3    192.168.100.92  
	registry 192.168.100.89

2 所有机器都关闭selinux

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

3 三台机器都配置好hostname

hostnamectl set-hostname master[或者node2/node3]
echo "192.168.100.89 master" >> /etc/hosts
echo "192.168.100.91 node2" >> /etc/hosts
echo "192.168.100.92 node3" >> /etc/hosts

把kubeadm init时查找的仓库地址配置为本地docker镜像仓库地址,这样被墙了也能从本地拉倒镜像

echo "192.168.100.89 quay.io k8s.gcr.io gcr.io"  >> /etc/hosts

4 关闭swap

swapoff -a
编辑 /etc/fstab,注释掉包含swap的那一行
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=20ca01ff-c5eb-47bc-99a0-6527b8cb246e /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap 

使用top命令查看结果

5 配置yum源

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo 
yum makecache

6 配置docker镜像仓库

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache

7 安装docker并启动

yum install docker-ce -y
systemctl start docker & systemctl enable docker

8 关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

9 在registry上创建docker本地镜像仓库
这里registry和master公用的一个服务器

docker pull registry
docker run --restart=always -d -p 80:5000 --hostname=my-registry --name my-registry -v /mnt/data/registry:/var/lib/registry registry

10 配置各节点系统内核参数使流过网桥的流量也进入iptables/netfilter框架中

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
vm.swappiness                       = 0
EOF

执行命令使配置生效

sysctl -p /etc/sysctl.d/k8s.conf

11 确认iptables的FORWARD规则
Docker不知啥时候开始会将iptables filter链的FORWARD规则默认设置为DROP

[[email protected] ~]# iptables -vnL | grep FORWARD
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
[[email protected] ~]# systemctl start docker
[[email protected] ~]# iptables -vnL | grep FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)

设置FORWARD规则为ACCEPT

iptables -P FORWARD ACCEPT

将该操作固化到开机流程中,而且得在docker服务启动之后,因此我们添加一个systemd开机服务

cat > /usr/lib/systemd/system/forward-accept.service <<EOF
[Unit]
Description=set forward accept
After=docker.service
 
[Service]
ExecStart=/usr/sbin/iptables -P FORWARD ACCEPT
 
[Install]
WantedBy=multi-user.target
EOF
systemctl enable forward-accept && systemctl start forward-accept

12 安装ntp服务并启动
保证集群间的时间一致,否则会有各种未知问题。

yum install -y ntp
systemctl start ntpd;systemctl enable ntpd

13、使docker和kubelet的cgroup driver一致,并配置本地镜像以及kubeadm初始化默认读取的镜像仓库
kubelet默认是systemd,docker是cgroupfs

查看docker的cgroup

docker info | grep "Cgroup Driver"
Cgroup Driver: cgroupfs

cat << EOF > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "insecure-registries":["192.168.100.89:80", "quay.io", "k8s.gcr.io", "gcr.io"]
}
EOF
systemctl restart docker

都配置完后最好重启机器

二 准备镜像
kubeadm init的时候会从默认仓库下载镜像,我们先准备好对应版本的镜像
下面的操作在一台机器执行就可以了

1 查看最先版本的镜像依赖
当前版本为v1.15.2
查看镜像依赖

这里可能会kubeadm找不到命令,如果想查看版本,可以先执行以下三-1步骤,安装以下再查

kubeadm config images list --kubernetes-version=v1.15.2

结果为:

k8s.gcr.io/kube-apiserver:v1.15.2
k8s.gcr.io/kube-controller-manager:v1.15.2
k8s.gcr.io/kube-scheduler:v1.15.2
k8s.gcr.io/kube-proxy:v1.15.2
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1

2 下载各个镜像

docker pull mirrorgooglecontainers/kube-apiserver:v1.15.2
docker pull mirrorgooglecontainers/kube-proxy:v1.15.2
docker pull mirrorgooglecontainers/kube-controller-manager:v1.15.2
docker pull mirrorgooglecontainers/kube-scheduler:v1.15.2
docker pull coredns/coredns:1.3.1
docker pull mirrorgooglecontainers/etcd:3.3.10
docker pull mirrorgooglecontainers/pause:3.1
获取flannel镜像
docker pull quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64

3 为镜像打上自己的tag

docker tag mirrorgooglecontainers/kube-apiserver:v1.15.2 192.168.100.89:80/kube-apiserver:v1.15.2
docker tag mirrorgooglecontainers/kube-proxy:v1.15.2 192.168.100.89:80/kube-proxy:v1.15.2
docker tag mirrorgooglecontainers/kube-controller-manager:v1.15.2 192.168.100.89:80/kube-controller-manager:v1.15.2
docker tag mirrorgooglecontainers/kube-scheduler:v1.15.2 192.168.100.89:80/kube-scheduler:v1.15.2
docker tag coredns/coredns:1.3.1 192.168.100.89:80/coredns:1.3.1
docker tag mirrorgooglecontainers/etcd:3.3.10 192.168.100.89:80/etcd:3.3.10
docker tag mirrorgooglecontainers/pause:3.1 192.168.100.89:80/pause:3.1
docker tag quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64 192.168.100.89:80/coreos/flannel:v0.11.0-amd64

4 push到本地仓库

docker push 192.168.100.89:80/kube-apiserver:v1.15.2
docker push 192.168.100.89:80/kube-proxy:v1.15.2
docker push 192.168.100.89:80/kube-controller-manager:v1.15.2
docker push 192.168.100.89:80/kube-scheduler:v1.15.2
docker push 192.168.100.89:80/coredns:1.3.1
docker push 192.168.100.89:80/etcd:3.3.10
docker push 192.168.100.89:80/pause:3.1
docker push 192.168.100.89:80/coreos/flannel:v0.11.0-amd64

三、安装kubelet

1 使用阿里云repo源安装

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet

2 master节点一键部署集群

kubeadm init --pod-network-cidr 10.244.0.0/16

切记,如果集群通信采用flannel的话,执行时一定要带 --pod-network-cidr 参数,并且网络段要和后面步骤中使用的flannel yaml文件中定义的保持一致。

kubeadm init --pod-network-cidr 10.244.0.0/16

执行后界面

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.89:6443 --token w5yoxp.a4g7fokmf4co1otq \
--discovery-token-ca-cert-hash sha256:351e1e5113e9b2c672280c4bc4f57a6c2defb6d289d03c94590d0710d2033873  

拷贝最后join的信息,待用

3 配置让非root用户可以使用kubelet

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

4 将其他两个节点加入master

kubeadm join 192.168.100.89:6443 --token w5yoxp.a4g7fokmf4co1otq \
--discovery-token-ca-cert-hash sha256:351e1e5113e9b2c672280c4bc4f57a6c2defb6d289d03c94590d0710d2033873 

执行后界面

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

5 配置通信组件flannel
在master节点查看状态
kubectl get nodes

NAME STATUS ROLES AGE VERSION
node2 NotReady 14s v1.15.2
master NotReady master 172m v1.15.2
node3 NotReady 11s v1.15.2

状态为NotReady,执行命令:
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml

这里也可以先把文件下载再执行命令,保证网段与init时的参数一致

 net-conf.json: |
{
  "Network": "10.244.0.0/16",
  "Backend": {
    "Type": "vxlan"
  }
}

过一会再看到状态就是ready了

四 常见错误处理

1 清理kubelet

kubeadm reset
rm -rf $HOME/.kube/
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ip link delete cni0
ip link delete flannel.1
systemctl restart docker