测试StatefulSet资源时，pod卡死在ContainerCreating
程序员文章站
2024-03-11 15:38:55
...
背景：
创建一个名为myapp的StatefulSet资源，通过Pod模板创建了3个Pod资源副本，并基于volumeClaimTemplates向前面创建的PV进行了请求大小为2Gi的专用存储卷。
问题描述：
没有像预计的那样创建三个pod副本，只创建了一个myapp-0副本，并且卡死在ContainerCreating状态。NAME READY STATUS RESTARTS AGE
myapp-0 0/1 ContainerCreating 0 2m57s
原因分析：
查看pv和pvc的情况如下，发现只有一个pv和pvc进行了绑定，并没有像预计的那样有三对绑定的pv和pvc。[[email protected] volumes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv001 1Gi RWO,RWX Retain Available 14m
pv002 2Gi RWO Retain Bound default/myappdata-myapp-0 14m
pv003 2Gi RWO,RWX Retain Available 14m
pv004 2Gi RWO,RWX Retain Available 14m
pv005 2Gi RWO,RWX Retain Available 14m
[[email protected] volumes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
myappdata-myapp-0 Bound pv002 2Gi RWO 13m
解决方案：
通过kubectl describe pod myapp-0查看此pod的情况，如下：
[[email protected] volumes]# kubectl describe pod myapp-0
Name: myapp-0
Namespace: default
Priority: 0
Node: k8s-node2/192.168.139.145
Start Time: Tue, 16 Feb 2021 21:10:44 +0800
Labels: app=myapp-pod
controller-revision-hash=myapp-8598dd746f
statefulset.kubernetes.io/pod-name=myapp-0
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/myapp
Containers:
myapp:
Container ID:
Image: ikubernetes/myapp:v1
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from myappdata (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dcwl2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
myappdata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: myappdata-myapp-0
ReadOnly: false
default-token-dcwl2:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-dcwl2
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/myapp-0 to k8s-node2
Warning FailedMount 12m kubelet, k8s-node2 MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107550.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 12m kubelet, k8s-node2 MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107554.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 11m kubelet, k8s-node2 MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107577.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 11m kubelet, k8s-node2 MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107625.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 11m kubelet, k8s-node2 MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107642.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 11m kubelet, k8s-node2 MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107673.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 11m kubelet, k8s-node2 MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107771.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 10m kubelet, k8s-node2 MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107919.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 9m58s kubelet, k8s-node2 Unable to attach or mount volumes: unmounted volumes=[myappdata], unattached volumes=[default-token-dcwl2 myappdata]: timed out waiting for the condition
Warning FailedMount 104s (x6 over 9m53s) kubelet, k8s-node2 (combined from similar events): MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-110486.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
Warning FailedMount 52s (x3 over 5m24s) kubelet, k8s-node2 Unable to attach or mount volumes: unmounted volumes=[myappdata], unattached volumes=[myappdata default-token-dcwl2]: timed out waiting for the condition
发现问题描述在于mount.nfs: Failed to resolve server store: Name or service not known mount.nfs: Operation already in progress Warning FailedMount 52s (x3 over 5m24s) kubelet, k8s-node2 Unable to attach or mount volumes: unmounted volumes=[myappdata], unattached volumes=[myappdata default-token-dcwl2]: timed out waiting for the condition，大概意思是k8s-node2节点无法识别store（部署了nfs的主机名），因此无法挂载nfs卷。
查看k8s-node2节点的/etc/hosts配置，发现忘记配置了store的ip映射关系，故加上store 192.168.139.189，然后重新部署StatefulSet资源成功。
结果如下：
[[email protected] statefulSetDemo]# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myapp-0 1/1 Running 0 11s 10.244.1.27 k8s-node2 <none> <none>
myapp-1 1/1 Running 0 10s 10.244.0.17 k8s-node1 <none> <none>
myapp-2 1/1 Running 0 5s 10.244.1.28 k8s-node2 <none> <none>
mydemo01pod-694474c9bd-5pwlc 1/1 Running 0 2d6h 10.244.0.4 k8s-node1 <none> <none>
mydemo02pod-7cb5c8c875-vh9gv 1/1 Running 0 34h 10.244.0.9 k8s-node1 <none> <none>
[[email protected] statefulSetDemo]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pv001 1Gi RWO,RWX Retain Available 27m
persistentvolume/pv002 2Gi RWO Retain Bound default/myappdata-myapp-0 27m
persistentvolume/pv003 2Gi RWO,RWX Retain Bound default/myappdata-myapp-1 27m
persistentvolume/pv004 2Gi RWO,RWX Retain Bound default/myappdata-myapp-2 27m
persistentvolume/pv005 2Gi RWO,RWX Retain Available 27m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/myappdata-myapp-0 Bound pv002 2Gi RWO 26m
persistentvolumeclaim/myappdata-myapp-1 Bound pv003 2Gi RWO,RWX 26s
persistentvolumeclaim/myappdata-myapp-2 Bound pv004 2Gi RWO,RWX 21s
If you miss the train I’m on
You will know that l am gone
You can hear the whistle blow a hundred miles…
上一篇： vue安装less