欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

kubernetes(四)二进制安装-flannel安装

程序员文章站 2024-03-12 11:17:14
...

部署 flannel 网络(在master节点上执行)

kubernetes组件kubelet服务依赖docker服务,docker网络需要用flannel来配置docker0网桥的ip地址,所以需要先安装flannel网络组建

flannel 使用 vxlan 技术为各节点创建一个可以互通的 Pod 网络,使用的端口为 UDP 8472(需要开放该端口,如公有云 AWS 等)。

flanneld 第一次启动时,从 etcd 获取配置的 Pod 网段信息,为本节点分配一个未使用的地址段,然后创建 flannedl.1 网络接口(也可能是其它名称,如 flannel1 等)。

flannel 将分配给自己的 Pod 网段信息写入 /run/flannel/docker 文件,docker 后续使用这个文件中的环境变量设置 docker0 网桥,从而从这个地址段为本节点的所有 Pod 容器分配 IP

  1. 下载和安装flanneld 二进制文件

    
    cd /opt/k8s/work
    mkdir flannel
    wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
    tar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel
    
    cp flannel/{flanneld,mk-docker-opts.sh} /opt/k8s/bin/
    
    export node_ip=192.168.0.114
    scp flannel/{flanneld,mk-docker-opts.sh} [email protected]${192.168.0.114}:/opt/k8s/bin/
    
  2. 创建 flanneld 证书和私钥

    flanneld 从 etcd 集群存取网段分配信息,而 etcd 集群启用了双向 x509 证书认证,所以需要为 flanneld 生成证书和私钥。

    1. 创建证书签名请求

      cd /opt/k8s/work
      cat > flanneld-csr.json <<EOF
      {
        "CN": "flanneld",
        "hosts": [],
        "key": {
          "algo": "rsa",
          "size": 2048
        },
        "names": [
          {
            "C": "CN",
            "ST": "NanJing",
            "L": "NanJing",
            "O": "k8s",
            "OU": "system"
          }
        ]
      }
      EOF
      
      
    2. 生成证书和私钥

      cfssl gencert -ca=/opt/k8s/work/ca.pem \
        -ca-key=/opt/k8s/work/ca-key.pem \
        -config=/opt/k8s/work/ca-config.json \
        -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
      ls flanneld*pem
      
    3. 将生成的证书和私钥分发到所有节点

      cd /opt/k8s/work
      mkdir -p /etc/flanneld/cert
      cp flanneld*.pem /etc/flanneld/cert
      
      export node_ip=192.168.0.114
      ssh [email protected]${node_ip} "mkdir -p /etc/flanneld/cert"
      scp flanneld*.pem [email protected]${node_ip}:/etc/flanneld/cert
      
      
  3. 向 etcd 写入集群 Pod 网段信息

    cd /opt/k8s/work
    
    export FLANNEL_ETCD_PREFIX="/kubernetes/network"
    export ETCD_ENDPOINTS="https://192.168.0.107:2379"
    
    etcdctl \
      --endpoints=${ETCD_ENDPOINTS} \
      --ca-file=/opt/k8s/work/ca.pem \
      --cert-file=/opt/k8s/work/flanneld.pem \
      --key-file=/opt/k8s/work/flanneld-key.pem \
      mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
    
    
    • 写入的 Pod 网段 Network 网络段对应的数值(如 /16)必须小于 SubnetLen对应的值(如24)
  4. 创建 flanneld 服务的启动文件

    
    cd /opt/k8s/work
    export FLANNEL_ETCD_PREFIX="/kubernetes/network"
    export ETCD_ENDPOINTS="https://192.168.0.107:2379"
    
    cat > flanneld.service << EOF
    [Unit]
    Description=Flanneld overlay address etcd agent
    After=network.target
    After=network-online.target
    Wants=network-online.target
    After=etcd.service
    Before=docker.service
    
    [Service]
    Type=notify
    ExecStart=/opt/k8s/bin/flanneld \\
      -etcd-cafile=/etc/kubernetes/cert/ca.pem \\
      -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
      -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
      -etcd-endpoints=${ETCD_ENDPOINTS} \\
      -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
      -ip-masq
    ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
    Restart=always
    RestartSec=5
    StartLimitInterval=0
    
    [Install]
    WantedBy=multi-user.target
    RequiredBy=docker.service
    EOF
    
    
    • mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网段信息,通过-d参数写入 /run/flannel/docker 文件,后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥, -k 参数控制生成文件中变量的名称,下面docker启动时会用到这个变量;
    • flanneld 使用系统缺省路由所在的接口与其它节点通信,对于有多个网络接口(如内网和公网)的节点,可以用 -iface 参数指定通信接口;
    • -ip-masq: flanneld 为访问 Pod 网络外的流量设置 SNAT 规则,同时将传递给 Docker 的变量 --ip-masq(/run/flannel/docker 文件中)设置为 false,这样 Docker 将不再创建 SNAT 规则; Docker 的 --ip-masq 为 true 时,创建的 SNAT 规则比较“暴力”:将所有本节点 Pod 发起的、访问非 docker0 接口的请求做 SNAT,这样访问其他节点 Pod 的请求来源 IP 会被设置为 flannel.1 接口的 IP,导致目的 Pod 看不到真实的来源 Pod IP。 flanneld 创建的 SNAT 规则比较温和,只对访问非 Pod 网段的请求做 SNAT
  5. 分发flanneld服务

    cd /opt/k8s/work
    
    cp flanneld.service /etc/systemd/system/
    
    export node_ip=192.168.0.114
    scp flanneld.service [email protected]${node_ip}:/etc/systemd/system/
    
    
  6. 启动flanneld服务

    systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld
    
    ssh [email protected]${node_ip) "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"
    
    
  7. 检查启动结果

    systemctl status flanneld|grep Active
    
    export node_ip=192.168.0.114
    ssh [email protected]${node_ip} "systemctl status flanneld|grep Active"
    
    • 确保状态为 active (running),否则查看日志,确认原因

    • 如果出现异常,通过如下命令查看

      journalctl -u flanneld
      
  8. 检查分配给各 flanneld 的 Pod 网段信息

    export FLANNEL_ETCD_PREFIX="/kubernetes/network"
    export ETCD_ENDPOINTS="https://192.168.0.107:2379"
    
    
    etcdctl \
      --endpoints=${ETCD_ENDPOINTS} \
      --ca-file=/etc/kubernetes/cert/ca.pem \
      --cert-file=/etc/flanneld/cert/flanneld.pem \
      --key-file=/etc/flanneld/cert/flanneld-key.pem \
      get ${FLANNEL_ETCD_PREFIX}/config
    

    输出结果

    {"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}
    
  9. 查看已分配的 Pod 子网段列表

    export FLANNEL_ETCD_PREFIX="/kubernetes/network"
    export ETCD_ENDPOINTS="https://192.168.0.107:2379"
    
    etcdctl \
      --endpoints=${ETCD_ENDPOINTS} \
      --ca-file=/etc/kubernetes/cert/ca.pem \
      --cert-file=/etc/flanneld/cert/flanneld.pem \
      --key-file=/etc/flanneld/cert/flanneld-key.pem \
      ls ${FLANNEL_ETCD_PREFIX}/subnets
    

    输出结果

    /kubernetes/network/subnets/172.30.22.0-24
    /kubernetes/network/subnets/172.30.78.0-24
    
  10. 检查节点 flannel 网络信息

    [email protected]:/opt/k8s/work# ip addr show
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
        link/ether 04:92:26:13:92:2b brd ff:ff:ff:ff:ff:ff
    3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
        link/ether d0:c5:d3:57:73:01 brd ff:ff:ff:ff:ff:ff
        inet 192.168.0.107/24 brd 192.168.0.255 scope global dynamic noprefixroute wlp3s0
           valid_lft 6385sec preferred_lft 6385sec
        inet6 fe80::1fda:e90a:207a:67e4/64 scope link noprefixroute
           valid_lft forever preferred_lft forever
    4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
        link/ether 12:cb:66:43:de:36 brd ff:ff:ff:ff:ff:ff
        inet 172.30.22.0/32 scope global flannel.1
           valid_lft forever preferred_lft forever
        inet6 fe80::10cb:66ff:fe43:de36/64 scope link
           valid_lft forever preferred_lft forever
      
    [email protected]:/opt/k8s/work# ip route show |grep flannel.1
    172.30.78.0/24 via 172.30.78.0 dev flannel.1 onlink 
    
    
  11. 验证各节点能通过 Pod 网段互通

    [email protected]:/opt/k8s/work# ip addr show flannel.1 |grep -w inet
        inet 172.30.22.0/32 scope global flannel.1
    [email protected]:/opt/k8s/work# ssh 192.168.0.114 "/sbin/ip addr show flannel.1|grep -w inet"
        inet 172.30.78.0/32 scope global flannel.1
    [email protected]:/opt/k8s/work# ping -c 1 172.30.78.0
    PING 172.30.78.0 (172.30.78.0) 56(84) bytes of data.
    64 bytes from 172.30.78.0: icmp_seq=1 ttl=64 time=80.7 ms
    
    --- 172.30.78.0 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 80.707/80.707/80.707/0.000 ms
    [email protected]:/opt/k8s/work# ssh 192.168.0.114 "ping -c 1 172.30.22.0"
    PING 172.30.22.0 (172.30.22.0) 56(84) bytes of data.
    64 bytes from 172.30.22.0: icmp_seq=1 ttl=64 time=4.09 ms
    
    --- 172.30.22.0 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 4.094/4.094/4.094/0.000 ms
    
    
    
  12. 生成文件内容,注意DOCKER_NETWORK_OPTIONS的值

    [email protected]:/opt/k8s/work# cat /run/flannel/subnet.env
    FLANNEL_NETWORK=172.30.0.0/16
    FLANNEL_SUBNET=172.30.22.1/24
    FLANNEL_MTU=1450
    FLANNEL_IPMASQ=true
    [email protected]:/opt/k8s/work# cat /run/flannel/docker
    DOCKER_OPT_BIP="--bip=172.30.22.1/24"
    DOCKER_OPT_IPMASQ="--ip-masq=false"
    DOCKER_OPT_MTU="--mtu=1450"
    DOCKER_NETWORK_OPTIONS=" --bip=172.30.22.1/24 --ip-masq=false --mtu=1450"
    
相关标签: 技术