[问题未处理]-docker报错stream copy error- reading from a closed fifo导致服务器oom并cpu飙升后无响应
程序员文章站
2022-06-24 22:48:54
问题及原因告警触发 服务器探活失败 最后宕机告警。现象:系统oom cpu负载爆高 服务器连接不上 读写等爆炸 阿里云控制台vnc进入也显示超时总共想通业务的机子有5台 其余4台服务器都没有问题。关键错误日志Sep 27 08:37:54 docker005 dockerd: time="2020-09-27T08:37:54.525450264+08:00" level=error msg="stream copy error: reading from a closed fifo"...
问题及原因
告警触发 服务器探活失败 最后宕机告警。
现象:系统oom cpu负载爆高 服务器连接不上 读写等爆炸 阿里云控制台vnc进入也显示超时
总共想通业务的机子有5台 其余4台服务器都没有问题。
关键错误日志
Sep 27 08:37:54 docker005 dockerd: time="2020-09-27T08:37:54.525450264+08:00" level=error msg="stream copy error: reading from a closed fifo"
Sep 27 08:37:54 docker005 dockerd: time="2020-09-27T08:37:54.525450309+08:00" level=error msg="stream copy error: reading from a closed fifo"
Sep 27 08:37:55 docker005 dockerd: time="2020-09-27T08:37:55.652626099+08:00" level=warning msg="Health check for container 9b351669f228259d22736b4a1097ddbfc26225ae2978a702a4b5149ff6e30f7d error: context deadline exceeded: unknown"
Sep 27 08:39:03 docker005 dockerd: time="2020-09-27T08:39:03.580218917+08:00" level=warning msg="Health check for container 9b351669f228259d22736b4a1097ddbfc26225ae2978a702a4b5149ff6e30f7d error: context deadline exceeded: unknown"
Sep 27 08:39:03 docker005 dockerd: time="2020-09-27T08:39:03.580277907+08:00" level=error msg="exit event" container=9b351669f228259d22736b4a1097ddbfc26225ae2978a702a4b5149ff6e30f7d error="no such process" module=libcontainerd namespace=moby process=fa63ec2efcfb6f5d03069b5f4775e08da32274272b1f35bc646dbfd84f2d70de
Sep 27 08:39:36 docker005 dockerd: time="2020-09-27T08:39:36.605897377+08:00" level=error msg="stream copy error: reading from a closed fifo"
Sep 27 08:39:36 docker005 dockerd: time="2020-09-27T08:39:36.605898957+08:00" level=error msg="stream copy error: reading from a closed fifo"
暂时未确认引起宕机的具体原因 猜测是绑定在这台机子上的定时任务引起的 不排除阿里云的问题。
类似issues
https://github.com/docker/for-linux/issues/402
本文地址:https://blog.csdn.net/xujiamin0022016/article/details/108859174