欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

测试StatefulSet资源时,pod卡死在ContainerCreating

程序员文章站 2024-03-11 15:38:55
...

背景:

创建一个名为myapp的StatefulSet资源,通过Pod模板创建了3个Pod资源副本,并基于volumeClaimTemplates向前面创建的PV进行了请求大小为2Gi的专用存储卷。


问题描述:

没有像预计的那样创建三个pod副本,只创建了一个myapp-0副本,并且卡死在ContainerCreating状态。
NAME                           READY   STATUS              RESTARTS   AGE
myapp-0                        0/1     ContainerCreating   0          2m57s


原因分析:

查看pv和pvc的情况如下,发现只有一个pv和pvc进行了绑定,并没有像预计的那样有三对绑定的pv和pvc。
[[email protected] volumes]# kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
pv001   1Gi        RWO,RWX        Retain           Available                                                       14m
pv002   2Gi        RWO            Retain           Bound       default/myappdata-myapp-0                           14m
pv003   2Gi        RWO,RWX        Retain           Available                                                       14m
pv004   2Gi        RWO,RWX        Retain           Available                                                       14m
pv005   2Gi        RWO,RWX        Retain           Available                                                       14m

[[email protected] volumes]# kubectl get pvc
NAME                STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myappdata-myapp-0   Bound    pv002    2Gi        RWO                           13m


解决方案:

通过kubectl describe pod myapp-0查看此pod的情况,如下:

[[email protected] volumes]# kubectl describe pod myapp-0
Name:           myapp-0
Namespace:      default
Priority:       0
Node:           k8s-node2/192.168.139.145
Start Time:     Tue, 16 Feb 2021 21:10:44 +0800
Labels:         app=myapp-pod
                controller-revision-hash=myapp-8598dd746f
                statefulset.kubernetes.io/pod-name=myapp-0
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/myapp
Containers:
  myapp:
    Container ID:   
    Image:          ikubernetes/myapp:v1
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/nginx/html from myappdata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dcwl2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  myappdata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  myappdata-myapp-0
    ReadOnly:   false
  default-token-dcwl2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dcwl2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age        From                Message
  ----     ------       ----       ----                -------
  Normal   Scheduled    <unknown>  default-scheduler   Successfully assigned default/myapp-0 to k8s-node2
  Warning  FailedMount  12m        kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107550.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  12m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107554.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107577.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107625.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107642.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107673.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107771.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  10m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107919.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  9m58s                 kubelet, k8s-node2  Unable to attach or mount volumes: unmounted volumes=[myappdata], unattached volumes=[default-token-dcwl2 myappdata]: timed out waiting for the condition
  Warning  FailedMount  104s (x6 over 9m53s)  kubelet, k8s-node2  (combined from similar events): MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-110486.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  52s (x3 over 5m24s)  kubelet, k8s-node2  Unable to attach or mount volumes: unmounted volumes=[myappdata], unattached volumes=[myappdata default-token-dcwl2]: timed out waiting for the condition

发现问题描述在于mount.nfs: Failed to resolve server store: Name or service not known mount.nfs: Operation already in progress Warning FailedMount 52s (x3 over 5m24s) kubelet, k8s-node2 Unable to attach or mount volumes: unmounted volumes=[myappdata], unattached volumes=[myappdata default-token-dcwl2]: timed out waiting for the condition,大概意思是k8s-node2节点无法识别store(部署了nfs的主机名),因此无法挂载nfs卷。

查看k8s-node2节点的/etc/hosts配置,发现忘记配置了store的ip映射关系,故加上store 192.168.139.189,然后重新部署StatefulSet资源成功。
结果如下:

[[email protected] statefulSetDemo]# kubectl get pod -owide
NAME                           READY   STATUS    RESTARTS   AGE    IP            NODE        NOMINATED NODE   READINESS GATES
myapp-0                        1/1     Running   0          11s    10.244.1.27   k8s-node2   <none>           <none>
myapp-1                        1/1     Running   0          10s    10.244.0.17   k8s-node1   <none>           <none>
myapp-2                        1/1     Running   0          5s     10.244.1.28   k8s-node2   <none>           <none>
mydemo01pod-694474c9bd-5pwlc   1/1     Running   0          2d6h   10.244.0.4    k8s-node1   <none>           <none>
mydemo02pod-7cb5c8c875-vh9gv   1/1     Running   0          34h    10.244.0.9    k8s-node1   <none>           <none>
[[email protected] statefulSetDemo]# kubectl get pv,pvc
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
persistentvolume/pv001   1Gi        RWO,RWX        Retain           Available                                                       27m
persistentvolume/pv002   2Gi        RWO            Retain           Bound       default/myappdata-myapp-0                           27m
persistentvolume/pv003   2Gi        RWO,RWX        Retain           Bound       default/myappdata-myapp-1                           27m
persistentvolume/pv004   2Gi        RWO,RWX        Retain           Bound       default/myappdata-myapp-2                           27m
persistentvolume/pv005   2Gi        RWO,RWX        Retain           Available                                                       27m

NAME                                      STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/myappdata-myapp-0   Bound    pv002    2Gi        RWO                           26m
persistentvolumeclaim/myappdata-myapp-1   Bound    pv003    2Gi        RWO,RWX                       26s
persistentvolumeclaim/myappdata-myapp-2   Bound    pv004    2Gi        RWO,RWX                       21s


If you miss the train I’m on

You will know that l am gone

You can hear the whistle blow a hundred miles…