k8s iptables 转ipvs模式

检查是否有ipvs 所需模块

ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4

查看是否被加载

[[email protected]-master1 ~]# ls /usr/lib/modules/3.10.0-514.el7.x86_64/kernel/net/netfilter/ipvs/ |grep  -e ip_vs
ip_vs_dh.ko
ip_vs_ftp.ko
ip_vs.ko
ip_vs_lblc.ko
ip_vs_lblcr.ko
ip_vs_lc.ko
ip_vs_nq.ko
ip_vs_pe_sip.ko
ip_vs_rr.ko
ip_vs_sed.ko
ip_vs_sh.ko
ip_vs_wlc.ko
ip_vs_wrr.ko

[[email protected]-master1 ~]# ls /usr/lib/modules/3.10.0-514.el7.x86_64/kernel/net/ipv4/netfilter/ |grep nf_conntrack_ipv4
nf_conntrack_ipv4.ko

或
lsmod | grep -e ip_vs -e nf_conntrack_ipv4

或
cut -f1 -d " "  /proc/modules | grep -e ip_vs -e nf_conntrack_ipv4

没有这些模块，使用以下命令加载

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4

在使用IPVS模式之前，还应在节点上安装ipset等软件包。

默认情况下，Kube-proxy在以kubeadm部署的集群中以iptables模式运行。查看日志如下

[[email protected]-master1 ~]# kubectl logs kube-proxy-5hrkc -n kube-system
W1231 02:00:36.140360       1 server_others.go:559] Unknown proxy mode "", assuming iptables proxy
I1231 02:00:36.153776       1 node.go:136] Successfully retrieved node IP: 172.23.5.158
I1231 02:00:36.153813       1 server_others.go:186] Using iptables Proxier.
I1231 02:00:36.154102       1 server.go:583] Version: v1.18.0
I1231 02:00:36.154495       1 conntrack.go:52] Setting nf_conntrack_max to 655360
I1231 02:00:36.154906       1 config.go:133] Starting endpoints config controller
I1231 02:00:36.154940       1 config.go:315] Starting service config controller
I1231 02:00:36.154962       1 shared_informer.go:223] Waiting for caches to sync for service config
I1231 02:00:36.154971       1 shared_informer.go:223] Waiting for caches to sync for endpoints config
I1231 02:00:36.255118       1 shared_informer.go:230] Caches are synced for service config 
I1231 02:00:36.255126       1 shared_informer.go:230] Caches are synced for endpoints config 

修改ConfigMap的kube-system/kube-proxy中的config.conf

[[email protected]-master1 ~]#  kubectl edit cm kube-proxy -n kube-system
configmap/kube-proxy edited

#修改如下
kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha1
...
ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: ""
      syncPeriod: 30s
    kind: KubeProxyConfiguration
    metricsBindAddress: 127.0.0.1:10249
    mode: "ipvs"                  #修改

...

之后重启各个节点上的kube-proxy（删除后会自动重新创建）

[[email protected]-master1 ~]# kubectl get pod -n kube-system | grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'
pod "kube-proxy-7dstj" deleted
pod "kube-proxy-lx887" deleted
pod "kube-proxy-nfsb9" deleted
pod "kube-proxy-pkj44" deleted

[[email protected]-master1 ~]#  kubectl get pod -n kube-system | grep kube-proxy 
kube-proxy-47dh9                           1/1     Running   0          13s
kube-proxy-64qnx                           1/1     Running   0          17s
kube-proxy-cbm26                           1/1     Running   0          20s
kube-proxy-xnpnn                           1/1     Running   0          15s

再次查看日志，可以看到ipvs已经启用了

[[email protected]-master ~]# kubectl logs kube-proxy-lgvmc -n kube-system
I0202 08:34:36.899095       1 node.go:172] Successfully retrieved node IP: 192.168.74.130
I0202 08:34:36.899216       1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.74.130), assume IPv4 operation
I0202 08:34:36.980583       1 server_others.go:258] Using ipvs Proxier.
E0202 08:34:36.980985       1 proxier.go:389] can't set sysctl net/ipv4/vs/conn_reuse_mode, kernel version must be at least 4.1
W0202 08:34:36.981086       1 proxier.go:445] IPVS scheduler not specified, use rr by default
I0202 08:34:36.981362       1 server.go:650] Version: v1.20.2
I0202 08:34:36.983439       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0202 08:34:36.983812       1 config.go:224] Starting endpoint slice config controller
I0202 08:34:36.983876       1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
I0202 08:34:36.983903       1 config.go:315] Starting service config controller
I0202 08:34:36.983906       1 shared_informer.go:240] Waiting for caches to sync for service config
I0202 08:34:37.084471       1 shared_informer.go:247] Caches are synced for endpoint slice config 
I0202 08:34:37.084603       1 shared_informer.go:247] Caches are synced for service config 

异常

E0202 08:49:36.828051       1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[10 244 5 151 0 0 0 0 0 0 0 0 0 0 0 0]
E0202 08:49:36.828067       1 proxier.go:1533] Failed to sync endpoint for service: 10.244.4.0:443/TCP, err: parseIP Error ip=[10 244 5 151 0 0 0 0 0 0 0 0 0 0 0 0]
E0202 08:49:36.828170       1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[172 23 6 215 0 0 0 0 0 0 0 0 0 0 0 0]
E0202 08:49:36.828198       1 proxier.go:1192] Failed to sync endpoint for service: 10.1.21.242:9283/TCP, err: parseIP Error ip=[172 23 6 215 0 0 0 0 0 0 0 0 0 0 0 0]

通过 kube-proxy Pod 的日志可以看到，里面有很多 Error 级别的日志信息，根据关键字 IPVS、parseIP Error 可知，可能是由于 IPVS 模块对 IP 进行格式化导致出现问题。

因为这个问题是升级到 kubernetes 1.18 版本才出现的，所以去 Kubernetes Github 查看相关 issues，发现有人在升级 Kubernetes 版本到 1.18 后，也遇见了相同的问题，经过 issue 中 Kubernetes 维护人员讨论，分析出原因可能为新版 Kubernetes 使用的 IPVS 模块是比较新的，需要系统内核版本支持，本人使用的是 CentOS 系统，内核版本为 3.10，里面的 IPVS 模块比较老旧，缺少新版 Kubernetes IPVS 所需的依赖。

根据该 issue 讨论结果，解决该问题的办法是，更新内核为新的版本。

参考文章

异常处理： http://www.mydlq.club/article/78/
kubernetes的ipvs模式和iptables模式： https://blog.csdn.net/fanren224/article/details/86548398