kubernetes实践之一：kubernetes二进制包安装
kubernetes二进制部署
1、环境规划
|
软件 |
版本 |
|
linux操作系统 |
centos linux release 7.6.1810 (core) |
|
kubernetes |
1.9 |
|
docker |
18.09.3 |
|
etcd |
3.3.10 |
|
角色 |
ip |
组件 |
推荐配置 |
|
k8s_master etcd01 |
192.168.1.153 |
kube-apiserver kube-controller-manager kube-scheduler etcd |
cpu 2核+ 2g内存+ |
|
k8s_node01 etcd02 |
192.168.1.154 |
kubelet kube-proxy docker flannel etcd |
|
|
k8s_node02 etcd03 |
192.168.1.155 |
kubelet kube-proxy docker flannel etcd |
|
2、 单master集群架构
3、 系统常规参数配置
3.1 关闭selinux
sed -i 's/selinux=enforcing/selinux=disabled/' /etc/selinux/config
setenforce 0
3.2 文件数调整
sed -i '/* soft nproc 4096/d' /etc/security/limits.d/20-nproc.conf
echo '* - nofile 65536' >> /etc/security/limits.conf
echo '* soft nofile 65535' >> /etc/security/limits.conf
echo '* hard nofile 65535' >> /etc/security/limits.conf
echo 'fs.file-max = 65536' >> /etc/sysctl.conf
3.3 防火墙关闭
systemctl disable firewalld.service
systemctl stop firewalld.service
3.4 常用工具安装及时间同步
yum -y install vim telnet iotop openssh-clients openssh-server ntp net-tools.x86_64 wget
ntpdate time.windows.com
3.5 hosts文件配置(3个节点)
vim /etc/hosts
192.168.1.153 k8s_master
192.168.1.154 k8s_node01
192.168.1.155 k8s_node02
3.6 服务器之间免密钥登录
ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.1.154
ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.1.155
4、 自签ssl证书
4.1 etcd生成证书
|
cfssl.sh |
etcd-cert.sh |
etcd.sh |
4.1.1 安装cfssl工具（cfssl.sh）
cd /home/k8s_install/ssl_etcd
chmod +x cfssl.sh
./cfssl.sh
内容如下：
curl -l https://pkg.cfssl.org/r1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -l https://pkg.cfssl.org/r1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -l https://pkg.cfssl.org/r1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
4.1.2 生成etcd 自签ca证书（etcd-cert.sh）
chmod +x etcd-cert.sh
./etcd-cert.sh
内容如下：
cat > ca-config.json <<eof
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
eof
cat > ca-csr.json <<eof
{
"cn": "etcd ca",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"c": "cn",
"l": "beijing",
"st": "beijing"
}
]
}
eof
cfssl gencert -initca ca-csr.json | cfssljson -bare ca –
#-----------------------
cat > server-csr.json <<eof
{
"cn": "etcd",
"hosts": [
"192.168.1.153",
"192.168.1.154",
"192.168.1.155",
"192.168.1.156",
"192.168.1.157"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"c": "cn",
"l": "beijing",
"st": "beijing"
}
]
}
eof
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
注意：hosts一定要包含所有节点，可以多部署几个预留节点以便后续扩容，否则还需要重新生成
4.1.3 etcd二进制包安装
#存放配置文件，可执行文件，证书文件
mkdir /opt/etcd/{cfg,bin,ssl} -p
#ssl 证书切记复制到/opt/etcd/ssl/
cp {ca,server-key,server}.pem /opt/etcd/ssl/
#部署etcd以及增加etcd服务(etcd.sh)
cd /home/k8s_install/soft/
tar -zxvf etcd-v3.3.10-linux-amd64.tar.gz
cd etcd-v3.3.10-linux-amd64
mv etcd etcdctl /opt/etcd/bin/
cd /home/k8s_install/ssl_etcd
chmod +x etcd.sh
参数说明:1.etcd名称 2.本机ip 3.其他两个etcd名称以及地址
./etcd.sh etcd01 192.168.1.153 etcd02=https://192.168.1.154:2380,etcd03=https://192.168.1.155:2380
执行后会卡住实际是在等待其他两个节点加入
其他两个node节点部署etcd:
scp -r /opt/etcd/ k8s_node01:/opt/
scp -r /opt/etcd/ k8s_node02:/opt/
scp /usr/lib/systemd/system/etcd.service k8s_node01:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service k8s_node02:/usr/lib/systemd/system/
#修改node节点配置文件(2个节点都需要更改)
ssh k8s_node01
vim /opt/etcd/cfg/etcd
etcd_name以及ip地址
systemctl daemon-reload
systemctl start etcd.service
etcd.sh脚本内容如下：
#!/bin/bash
etcd_name=$1
etcd_ip=$2
etcd_cluster=$3
work_dir=/opt/etcd
cat <<eof >$work_dir/cfg/etcd
#[member]
etcd_name="${etcd_name}"
etcd_data_dir="/var/lib/etcd/default.etcd"
etcd_listen_peer_urls="https://${etcd_ip}:2380"
etcd_listen_client_urls="https://${etcd_ip}:2379"
#[clustering]
etcd_initial_advertise_peer_urls="https://${etcd_ip}:2380"
etcd_advertise_client_urls="https://${etcd_ip}:2379"
etcd_initial_cluster="etcd01=https://${etcd_ip}:2380,${etcd_cluster}"
etcd_initial_cluster_token="etcd-cluster"
etcd_initial_cluster_state="new"
eof
cat <<eof >/usr/lib/systemd/system/etcd.service
[unit]
description=etcd server
after=network.target
after=network-online.target
wants=network-online.target
[service]
type=notify
environmentfile=${work_dir}/cfg/etcd
execstart=${work_dir}/bin/etcd \
--name=\${etcd_name} \
--data-dir=\${etcd_data_dir} \
--listen-peer-urls=\${etcd_listen_peer_urls} \
--listen-client-urls=\${etcd_listen_client_urls},http://127.0.0.1:2379 \
--advertise-client-urls=\${etcd_advertise_client_urls} \
--initial-advertise-peer-urls=\${etcd_initial_advertise_peer_urls} \
--initial-cluster=\${etcd_initial_cluster} \
--initial-cluster-token=\${etcd_initial_cluster_token} \
--initial-cluster-state=new \
--cert-file=${work_dir}/ssl/server.pem \
--key-file=${work_dir}/ssl/server-key.pem \
--peer-cert-file=${work_dir}/ssl/server.pem \
--peer-key-file=${work_dir}/ssl/server-key.pem \
--trusted-ca-file=${work_dir}/ssl/ca.pem \
--peer-trusted-ca-file=${work_dir}/ssl/ca.pem
restart=on-failure
limitnofile=65536
[install]
wantedby=multi-user.target
eof
systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd
4.1.4 查看etcd集群健康情况
cd /opt/etcd/ssl
/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.1.153:2379,https://192.168.1.154:2379,https://192.168.1.155:2379" cluster-health
5、 安装docker(node 节点)
5.1 安装依赖包
yum install -y yum-utils \ device-mapper-persistent-data \ lvm2
5.2 配置官方源（替换为阿里源）
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
5.3 更新并安装docker-ce
yum makecache fast
yum install docker-ce -y
5.4 配置docker加速器
curl -ssl https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
5.5 启动docker
systemctl restart docker.service
systemctl enable docker.service
6、部署flannel网络
overlay network:覆盖网络，在基础网络上叠加的一种虚拟网络技术模式，该网络中的主机通过虚拟链路连接起来。 vxlan:将源数据包封装到udp中，并使用基础网络的ip/mac作为外层报文头进行封装，然后在以太网上传输，到达目的地后由隧道端点解封装并将数据发送给目标地址。 flannel:是overlay网络的一种，也是将源数据包封装在另一种网络包里面进行路由转发和通信，目前已经支持udp、vxlan、aws vpc和gce路由等数据转发方式。 多主机容器网络通信其他主流方案:隧道方案( weave、openvswitch )，路由方案(calico)等。
6.1 写入分配的子网段到etcd，供flanneld使用(master)
/opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints=https://192.168.1.153:2379,https://192.168.1.154:2379,https://192.168.1.155:2379 set /coreos.com/network/config '{ "network": "172.17.0.0/16", "backend": {"type": "vxlan"}}'
6.2 二进制包安装flannel(node节点 flannel.sh)
下载地址：
#
mkdir /opt/kubernetes/{bin,cfg,ssl} -p
cd /home/k8s_install/flannel_install/
tar -zxvf flannel-v0.10.0-linux-amd64.tar.gz
mv {flanneld,mk-docker-opts.sh} /opt/kubernetes/bin/
cd /home/k8s_install/flannel_install
chmod +x flannel.sh
chmod +x /opt/kubernetes/bin/{flanneld,mk-docker-opts.sh}
./flannel.sh https://192.168.1.153:2379,https://192.168.1.154:2379,https://192.168.1.155:2379
脚本内容如下：
#!/bin/bash
etcd_endpoints=${1:-"http://127.0.0.1:2379"}
cat <<eof >/opt/kubernetes/cfg/flanneld
flannel_options="--etcd-endpoints=${etcd_endpoints} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"
eof
cat <<eof >/usr/lib/systemd/system/flanneld.service
[unit]
description=flanneld overlay address etcd agent
after=network-online.target network.target
before=docker.service
[service]
type=notify
environmentfile=/opt/kubernetes/cfg/flanneld
execstart=/opt/kubernetes/bin/flanneld --ip-masq \$flannel_options
execstartpost=/opt/kubernetes/bin/mk-docker-opts.sh -k docker_network_options -d /run/flannel/subnet.env
restart=on-failure
[install]
wantedby=multi-user.target
eof
cat <<eof >/usr/lib/systemd/system/docker.service
[unit]
description=docker application container engine
documentation=https://docs.docker.com
after=network-online.target firewalld.service
wants=network-online.target
[service]
type=notify
environmentfile=/run/flannel/subnet.env
execstart=/usr/bin/dockerd \$docker_network_options
execreload=/bin/kill -s hup \$mainpid
limitnofile=infinity
limitnproc=infinity
limitcore=infinity
timeoutstartsec=0
delegate=yes &nbs