docker报错rpc error: code = 14 desc = grpc: the connection is unavailable

实验环境

操作系统：CentOS Linux release 7.4.1708 (Core)
docker版本：1.13.1

场景还原

在尝试强制删除正在运行的容器时报错：

[[email protected] ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                      NAMES
3e22bd0b6a40        mongo:3.2           "docker-entrypoint..."   19 hours ago        Up 19 hours         0.0.0.0:20002->27017/tcp   zen_mclean
[[email protected] ~]# docker rm 3e22bd0b6a40 -f
Error response from daemon: Cannot kill container 3e22bd0b6a40c85d2af45b5d65fb3648acab7e0ad05fa909201051a8f00a3d15: rpc error: code = 14 desc = grpc: the connection is unavailable

尝试关闭容器，进入容器操作界面也报相同错误：

[[email protected] ~]# docker stop 3e22bd0b6a40
Error response from daemon: Cannot stop container 3e22bd0b6a40: Cannot kill container 3e22bd0b6a40c85d2af45b5d65fb3648acab7e0ad05fa909201051a8f00a3d15: rpc error: code = 14 desc = grpc: the connection is unavailable
[[email protected] ~]# docker exec -it 3e22bd0b6a40 /bin/bash
rpc error: code = 14 desc = grpc: the connection is unavailable

尝试使用start命令启动容器，再删除容器，依然无效：

[[email protected] ~]# docker start zen_mclean
zen_mclean
[[email protected] ~]# docker rm -f zen_mclean 
Error response from daemon: Could not kill running container 3e22bd0b6a40c85d2af45b5d65fb3648acab7e0ad05fa909201051a8f00a3d15, cannot remove - Cannot kill container 3e22bd0b6a40c85d2af45b5d65fb3648acab7e0ad05fa909201051a8f00a3d15: rpc error: code = 14 desc = grpc: the connection is unavailable
[[email protected] ~]# docker exec -it 3e22bd0b6a40 /bin/bash
rpc error: code = 14 desc = grpc: the connection is unavailable

最后使用docker-containerd命令以debug模式调试容器：

[[email protected] ~]# docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --debug
WARN[0000] containerd: low RLIMIT_NOFILE changing to max  current=1024 max=4096
DEBU[0000] containerd: read past events                  count=2
ERRO[0000] containerd: notify OOM events                 error=open /proc/4153/cgroup: no such file or directory
DEBU[0000] containerd: container restored                id=3e22bd0b6a40c85d2af45b5d65fb3648acab7e0ad05fa909201051a8f00a3d15
DEBU[0000] containerd: supervisor running                cpus=1 memory=975 runtime=docker-runc runtimeArgs=[] stateDir=/var/run/docker/libcontainerd/containerd
DEBU[0000] containerd: process exited                    id=3e22bd0b6a40c85d2af45b5d65fb3648acab7e0ad05fa909201051a8f00a3d15 pid=init status=137 systemPid=4153
DEBU[0000] containerd: grpc api on /var/run/docker/libcontainerd/docker-containerd.sock 
ERRO[0000] containerd: deleting container                error=exit status 1: "container 3e22bd0b6a40c85d2af45b5d65fb3648acab7e0ad05fa909201051a8f00a3d15 does not exist\none or more of the container deletions failed\n"





^CINFO[1971] stopping containerd after receiving interrupt 

调试后发现容器状态变为了未开启，尝试删除容器，成功

[root@localhost ~]# docker exec -it 3e22bd0b6a40 /bin/bash
Error response from daemon: Container 3e22bd0b6a40c85d2af45b5d65fb3648acab7e0ad05fa909201051a8f00a3d15 is not running
[root@localhost ~]# docker rm -f zen_mclean 
zen_mclean
[root@localhost ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

总结

这次错误是由于偷懒，直接关闭开启中的容器导致的。以后要先关闭容器，再删除。