记录 - k8s入门4 - StatefulSet basic (1.16.0)
文章目录
环境
基于记录 - k8s 入门搭建 (1.16.0, helloweb) 搭建的环境。三个节点：192.168.199.200 (master, k8s0)、201 (ks81)、202 (k8s2)。
StatefulSet 用来管理有状态应用（的一组pod）。本篇主要参考 https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set，介绍基础sts。
hostPath 存储
在k8s里，容器与容器、pod与pod、节点与节点之间常常需要共享存储（storage, volume, PersistentVolume）。aws,azure,gce提供商用存储。本篇练习使用hostPath 类型的存储，参考 https://kubernetes.io/docs/concepts/storage/volumes/#hostpath。
hostPath 存储将节点(主机) 的文件/目录映射到pod里，例如将主机的 “/myapp/html” 目录映射为pod里的 “/usr/share/nginx/html”。hostPath 最主要的问题是需要维护在不同主机上同一目录内容的异同。
在三个节点创建测试目录、及建立一套相同内容的存储目录：
mkdir -p /test/k8s/basic_stateful_set
cd /test/k8s/basic_stateful_set
mkdir -p mnt/data
echo 'dat-0' > mnt/data/index.html
mkdir -p mnt/data2
echo 'dat-1' > mnt/data2/index.html
mkdir -p mnt/data3
echo 'dat-2' > mnt/data3/index.html
mkdir -p mnt/data4
echo 'dat-3' > mnt/data4/index.html
mkdir -p mnt/data5
echo 'dat-4' > mnt/data5/index.html
以下在k8s0 操作。
创建pv1.yaml：
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv1
labels:
type: local
spec:
# storageClassName: manual
capacity:
storage: 1Mi
accessModes:
- ReadWriteOnce
hostPath:
path: "/test/k8s/basic_stateful_set/mnt/data"
类似再创建pv2~pv5.yaml，文件内容里相应修改pv2->data2, …, pv5->data5。
创建5个持久存储（PersistentVolume）：
[[email protected] basic_stateful_set]# for i in 1 2 3 4 5; do kc create -f pv$i.yaml; done
persistentvolume/pv1 created
persistentvolume/pv2 created
persistentvolume/pv3 created
persistentvolume/pv4 created
persistentvolume/pv5 created
[[email protected] basic_stateful_set]# kc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv1 1Mi RWO Retain Available 29s
pv2 1Mi RWO Retain Available 28s
pv3 1Mi RWO Retain Available 28s
pv4 1Mi RWO Retain Available 27s
pv5 1Mi RWO Retain Available 27s
RWO (ReadWriteOnce) 意思是存储仅可被单个节点读写。
创建StatefulSet
创建web.yaml:
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
serviceName: "nginx"
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Mi
一个sts 创建2个有状态pod (replicas=2)。pod 里image=nginx 容器有映射的存储目录"/usr/share/nginx/html"。通过"name=www" 对应到pvc (volumeClaim)。
pod并不是直接引用pv，而是pod -> pvc -> pv。pvc 申明要使用pv。创建pvc时kubernetes 检查如果有符合条件的pv 则自动绑定该pv。
[[email protected] basic_stateful_set]# #创建
[[email protected] basic_stateful_set]# kc apply -f web.yaml
service/nginx created
statefulset.apps/web created
[[email protected] basic_stateful_set]#
[[email protected] basic_stateful_set]# kc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 15d
nginx ClusterIP None <none> 80/TCP 92s
[[email protected] basic_stateful_set]# #sts "web" 的2个pod：web-0, web-1 - 序号自动递增
[[email protected] basic_stateful_set]# kc get sts
NAME READY AGE
web 2/2 96s
[[email protected] basic_stateful_set]# kc get pod
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 118s
web-1 1/1 Running 0 113s
[[email protected] basic_stateful_set]# #pvc 的名称也是自动创建
[[email protected] basic_stateful_set]# kc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
www-web-0 Bound pv1 1Mi RWO 53s
www-web-1 Bound pv5 1Mi RWO 49s
[[email protected] basic_stateful_set]# #pv1,pv5绑定了，剩下三个可用
[[email protected] basic_stateful_set]# kc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv1 1Mi RWO Retain Bound default/www-web-0 2m20s
pv2 1Mi RWO Retain Available 2m19s
pv3 1Mi RWO Retain Available 2m19s
pv4 1Mi RWO Retain Available 2m18s
pv5 1Mi RWO Retain Bound default/www-web-1 2m18s
上面web-0,web-1 就组成一个StatefulSet。
sts里的pod
如上，sts 里pod 有固定、唯一的标志：web-0、web-1。
每个pod 也有固定的hostname：
[[email protected] basic_stateful_set]# for i in 0 1; do kc exec web-$i -- sh -c 'hostname'; done
web-0
web-1
web-0使用pv1 - “dat-0”，web-1使用pv5 - “dat-4”：
[[email protected] basic_stateful_set]# kc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
www-web-0 Bound pv1 1Mi RWO 5m28s
www-web-1 Bound pv5 1Mi RWO 5m24s
[[email protected] basic_stateful_set]# kc get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-0 1/1 Running 0 5m35s 10.100.109.103 k8s2 <none> <none>
web-1 1/1 Running 0 5m31s 10.100.166.206 k8s1 <none> <none>
[[email protected] basic_stateful_set]# curl 10.100.109.103
dat-0
[[email protected] basic_stateful_set]# curl 10.100.166.206
dat-4
执行“kc get pod -w -l app=nginx” 监视pod。
另开shell-2 执行“kc delete pod -l app=nginx”。
在原shell 可以看到2个pod 删除后，sts又自动重建了它们。
[[email protected] basic_stateful_set]# kc get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-0 1/1 Running 1 12m 10.100.109.105 k8s2 <none> <none>
web-1 1/1 Running 0 8m40s 10.100.166.255 k8s1 <none> <none>
[[email protected] basic_stateful_set]# #hostname 保持着
[[email protected] basic_stateful_set]# for i in 0 1; do kc exec web-$i -- sh -c 'hostname'; done
web-0
web-1
[[email protected] basic_stateful_set]# curl 10.100.109.105
dat-0
[[email protected] basic_stateful_set]# curl 10.100.166.255
dat-4
[[email protected] basic_stateful_set]# kc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
www-web-0 Bound pv1 1Mi RWO 22m
www-web-1 Bound pv5 1Mi RWO 22m
删除pod时pvc不会自动删除，下次再创建pod时sts会重用原来的pvc。就是在正常情况下pod即使重建 也仍使用原pvc -> 原pv。但是注意：
- 例如k8s2停机，下次pod 只能重建在k8s1上。所以要保证每台机器有一套相同内容的存储
- 如果删除了pvc，下次重建时可能绑定不同pv
伸缩sts
指通过kc scale、kc patch 命令改变replicas数。
扩容
监视：kc get pods -w -l app=nginx
在shell-2：kc scale sts web --replicas=5
监视结果：
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 1 31m
web-1 1/1 Running 0 27m
web-2 0/1 Pending 0 0s
web-2 0/1 Pending 0 0s
web-2 0/1 Pending 0 2s
web-2 0/1 ContainerCreating 0 2s
web-2 0/1 ContainerCreating 0 3s
web-2 1/1 Running 0 7s
web-3 0/1 Pending 0 0s
web-3 0/1 Pending 0 0s
web-3 0/1 Pending 0 2s
web-3 0/1 ContainerCreating 0 2s
web-3 0/1 ContainerCreating 0 3s
web-3 1/1 Running 0 2m29s
web-4 0/1 Pending 0 0s
web-4 0/1 Pending 0 0s
web-4 0/1 Pending 0 1s
web-4 0/1 ContainerCreating 0 1s
web-4 0/1 ContainerCreating 0 3s
web-4 1/1 Running 0 5s
sts会按顺序扩容，例如只有web-3 Running之后才会创建web-4。
缩容
同样：kc get pods -w -l app=nginx
在shell-2：kc patch sts web -p ‘{“spec”:{“replicas”:3}}’
监视结果：
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 1 40m
web-1 1/1 Running 0 36m
web-2 1/1 Running 0 9m37s
web-3 1/1 Running 0 9m30s
web-4 1/1 Running 0 7m1s
web-4 1/1 Terminating 0 7m58s
web-4 0/1 Terminating 0 7m59s
web-4 0/1 Terminating 0 8m
web-4 0/1 Terminating 0 8m3s
web-4 0/1 Terminating 0 8m3s
web-3 1/1 Terminating 0 10m
web-3 0/1 Terminating 0 10m
web-3 0/1 Terminating 0 10m
web-3 0/1 Terminating 0 10m
web-4 Terminating 之后才会删除web-3。
[[email protected] basic_stateful_set]# kc get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-0 1/1 Running 1 44m 10.100.109.105 k8s2 <none> <none>
web-1 1/1 Running 0 40m 10.100.166.255 k8s1 <none> <none>
web-2 1/1 Running 0 13m 10.100.166.193 k8s1 <none> <none>
[[email protected] basic_stateful_set]# kc get pvc -l app=nginx
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
www-web-0 Bound pv1 1Mi RWO 53m
www-web-1 Bound pv5 1Mi RWO 53m
www-web-2 Bound pv2 1Mi RWO 13m
www-web-3 Bound pv3 1Mi RWO 13m
www-web-4 Bound pv4 1Mi RWO 11m
当sts的pod 被删除或缩容而删时，其pvc 仍旧会保留。
更新sts
Kubernetes 1.7之后，sts支持自动化更新。可以用来升级容器镜像、资源限制、label、annotation。spec.updateStrategy 可以指定两种更新策略：RollingUpdate (默认) 和OnDelete。
RollingUpdate策略
RollingUpdate 策略会反方向（reverse ordinal order）更新sts的pod。
设置：
kc patch statefulset web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'
监视：kc get po -l app=nginx -w
在shell-2：
kc patch sts web --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"YYY/helloweb"}]'
更新为入门搭建篇的helloweb 镜像，注意“YYY” 替换为docker hub 账户名。
监视结果：
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 1 64m
web-1 1/1 Running 0 60m
web-2 1/1 Running 0 94s
web-2 1/1 Terminating 0 2m15s
web-2 0/1 Terminating 0 2m16s
web-2 0/1 Terminating 0 2m19s
web-2 0/1 Terminating 0 2m19s
web-2 0/1 Pending 0 0s
web-2 0/1 Pending 0 0s
web-2 0/1 ContainerCreating 0 0s
web-2 0/1 ContainerCreating 0 2s
web-2 1/1 Running 0 6s
web-1 1/1 Terminating 0 61m
web-1 0/1 Terminating 0 61m
web-1 0/1 Terminating 0 61m
web-1 0/1 Terminating 0 61m
web-1 0/1 Pending 0 1s
web-1 0/1 Pending 0 1s
web-1 0/1 ContainerCreating 0 1s
web-1 0/1 ContainerCreating 0 2s
web-1 1/1 Running 0 4s
web-0 1/1 Terminating 1 65m
web-0 0/1 Terminating 1 65m
web-0 0/1 Terminating 1 65m
web-0 0/1 Terminating 1 65m
web-0 0/1 Pending 0 0s
web-0 0/1 Pending 0 0s
web-0 0/1 ContainerCreating 0 0s
web-0 0/1 ContainerCreating 0 1s
web-0 1/1 Running 0 2s
可见是按2、1、0的反序逐一更新。
检查：
[[email protected] basic_stateful_set]# for p in 0 1 2; do kc get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
YYY/helloweb
YYY/helloweb
YYY/helloweb
分段更新 (Staging an Update)
设置partition = 3：
kc patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":3}}}}'
更新sts 镜像=nginx：
kc patch sts web --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"nginx"}]'
检查镜像仍是helloweb：
[[email protected] basic_stateful_set]# for p in 0 1 2; do kc get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
YYY/helloweb
YYY/helloweb
YYY/helloweb
监视：kc get po -l app=nginx -w
在shell-2：kc delete po web-2
监视结果：
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 15m
web-1 1/1 Running 0 15m
web-2 1/1 Running 0 15m
web-2 1/1 Terminating 0 16m
web-2 0/1 Terminating 0 16m
web-2 0/1 Terminating 0 16m
web-2 0/1 Terminating 0 16m
web-2 0/1 Pending 0 0s
web-2 0/1 Pending 0 0s
web-2 0/1 ContainerCreating 0 0s
web-2 0/1 ContainerCreating 0 1s
web-2 1/1 Running 0 2s
web-2 重建后镜像仍是helloweb：
[[email protected] basic_stateful_set]# kc get po web-2 --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'
YYY/helloweb
因为web-0、1、2的ordinal 都小于partition 3，所以它们在更新时仍保留旧设置 (restore original container)。
金丝雀发布 (Rolling Out a Canary)
通过少量减少partition 来测试程序。
监视：kc get po -l app=nginx -w
在shell-2 减小partition = 2：
kc patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'
监视结果：
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 32m
web-1 1/1 Running 0 32m
web-2 1/1 Running 0 16m
web-2 1/1 Terminating 0 16m
web-2 0/1 Terminating 0 16m
web-2 0/1 Terminating 0 16m
web-2 0/1 Terminating 0 16m
web-2 0/1 Pending 0 0s
web-2 0/1 Pending 0 0s
web-2 0/1 ContainerCreating 0 0s
web-2 0/1 ContainerCreating 0 1s
web-2 1/1 Running 0 3s
由于web-2的ordinal 2 现在>= partition 2，所以它自动更新了。
检查：
[[email protected] ~]# kc get po web-2 --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'
nginx
其镜像修改成nginx。
继续监视：kc get po -l app=nginx -w
在shell-2删除web-1：kc delete po web-1
监视结果：
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 39m
web-1 1/1 Running 0 39m
web-2 1/1 Running 0 6m21s
web-1 1/1 Terminating 0 39m
web-1 0/1 Terminating 0 39m
web-1 0/1 Terminating 0 39m
web-1 0/1 Terminating 0 39m
web-1 0/1 Terminating 0 39m
web-1 0/1 Pending 0 0s
web-1 0/1 Pending 0 0s
web-1 0/1 ContainerCreating 0 1s
web-1 0/1 ContainerCreating 0 2s
web-1 1/1 Running 0 5s
web-1重建了。检查其镜像：
[[email protected] basic_stateful_set]# kc get po web-1 --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'
YYY/helloweb
web-1 镜像仍没修改，因为它的ordinal 1 < partition 2。
当设置了partition后，修改.spec.template 只会修改ordinal >= partition 的pod。Ordinal < partition 的pod 删除/重建时仍保留旧设置 (restore original configuration)。
分段发布 (Phased Roll Outs)
目前partition = 2。
监视：kc get po -l app=nginx -w
在shell-2 设置partition = 0：
kc patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":0}}}}'
监视结果：
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 54m
web-1 1/1 Running 0 14m
web-2 1/1 Running 0 21m
web-1 1/1 Terminating 0 14m
web-1 0/1 Terminating 0 14m
web-1 0/1 Terminating 0 14m
web-1 0/1 Terminating 0 14m
web-1 0/1 Pending 0 0s
web-1 0/1 Pending 0 0s
web-1 0/1 ContainerCreating 0 0s
web-1 0/1 ContainerCreating 0 2s
web-1 1/1 Running 0 7m55s
web-0 1/1 Terminating 1 62m
web-0 0/1 Terminating 1 62m
web-0 0/1 Terminating 1 62m
web-0 0/1 Terminating 1 62m
web-0 0/1 Pending 0 0s
web-0 0/1 Pending 0 0s
web-0 0/1 ContainerCreating 0 0s
web-0 0/1 ContainerCreating 0 1s
web-0 1/1 Running 0 2s
看到按1、0的反序重建了pod。检查镜像：
[[email protected] basic_stateful_set]# for p in 0 1 2; do kc get po web-$p --template '{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'; echo; done
nginx
nginx
nginx
看到所有pod都已更新。
OnDelete策略
Kubernetes 1.6及之前的行为。当更新.spec.template 时，不会自动更新pod。
删除sts
Non-Cascading Delete
非级联删除：删除sts时不删除pod。
web-0、1、2。
非级联删除：kc delete sts web --cascade=false
三个pod仍在。
删除web-0：kc delete pod web-0
只剩web-1、2。
重新执行：kc apply -f web.yaml
监视到先重建web-0、再删除web-2 (因为yaml里replicas=2)。
检查：
[[email protected] basic_stateful_set]# for i in 0 1; do kc exec -it web-$i -- cat /usr/share/nginx/html/index.html; done
dat-0
dat-4
web-0、1使用的pv没变（只要节点还是原来的），因为删除sts/pod 时，不会自动删除pv。
Cascading Delete
级联删除：删除sts时也删除pod。
web-0、1。
级联删除：kc delete statefulset web
2个pod 均删除了。
注意级联删除还留下nginx service没有删除。删除它：
kc delete service nginx
再次重建：
[[email protected] basic_stateful_set]# kc apply -f web.yaml
service/nginx created
statefulset.apps/web created
检查：
[[email protected] basic_stateful_set]# kc get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-0 1/1 Running 0 38s 10.100.166.214 k8s1 <none> <none>
web-1 1/1 Running 0 36s 10.100.109.113 k8s2 <none> <none>
[[email protected] basic_stateful_set]# for i in 0 1; do kc exec -it web-$i -- cat /usr/share/nginx/html/index.html; done
dat-0
dat-4
[[email protected] basic_stateful_set]# curl 10.100.166.214
dat-0
[[email protected] basic_stateful_set]# curl 10.100.109.113
dat-4
[[email protected] basic_stateful_set]# for i in 0 1; do kc exec -it web-$i -- sh -c 'hostname'; done
web-0
web-1
如前，因为pvc & pv 没有删除，所以重建后应用的行为通常不会改变。
终于，删除sts和svc：
[[email protected] basic_stateful_set]# kc delete svc nginx
service "nginx" deleted
[[email protected] basic_stateful_set]# kc delete sts web
statefulset.apps "web" deleted
[[email protected] basic_stateful_set]# kc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
www-web-0 Bound pv1 1Mi RWO 3h3m
www-web-1 Bound pv5 1Mi RWO 3h3m
www-web-2 Bound pv2 1Mi RWO 144m
www-web-3 Bound pv3 1Mi RWO 143m
www-web-4 Bound pv4 1Mi RWO 141m
[[email protected] basic_stateful_set]# kc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv1 1Mi RWO Retain Bound default/www-web-0 3h4m
pv2 1Mi RWO Retain Bound default/www-web-2 3h4m
pv3 1Mi RWO Retain Bound default/www-web-3 3h4m
pv4 1Mi RWO Retain Bound default/www-web-4 3h4m
pv5 1Mi RWO Retain Bound default/www-web-1 3h4m
（注：期间经过多次重试，所以5个pvc和pv均已绑定，刚才实际在用的只有2个）
Pod Management Policy
.spec.podManagementPolicy 属性：
- “OrderedReady”：默认。上面所述严格按ordinal 顺序/反序来创建/删除 pod
- “Parallel”：不严格遵照ordinal 顺序
Cleanup
删除本教程创建的pvc和hostPath pv：
kc delete pvc -l app=nginx
for i in 1 2 3 4 5; do kc delete pv pv$i; done
至于三个节点上实际的存储目录(/test/k8s/basic_stateful_set/mnt/data*) 请自行处理。
上一篇： PHPstorm快捷键(分享)
下一篇： kubectl命令