深入分析Tomcat无响应问题及解决方法
问题描述
生产环境下有几台tomcat,但突然某个时候发现所有的请求都不能响应了,由于我们的web server使用的是nginx,会将请求反向到tomcat上,所以起初怀疑是nginx就没有收到请求,但查看日志后发现,nginx中大量出现499的返回,这说明问题还是出在tomcat上.
问题排查
首先我想到的是不是cpu跑满了,虽说cpu没有报警但还是本能的top命令看下系统负载,发现系统只有0.x的负载,cpu,内存消耗都是正常的.
由于cpu没有出现异常,所以应该不是gc出现了问题,但还是检查了下gc log,果然gc也没问题
此时必须让jstack上场了,果然在使用jstack后发现很多线程都是waiting状态
"http-nio-127.0.0.1-801-exec-498" daemon prio=10 tid=0x00002ada7c14f800 nid=0x16a6 waiting on condition [0x00002ada9c905000] java.lang.thread.state: waiting (parking) at sun.misc.unsafe.park(native method) - parking to wait for <0x00000007873e6990> (a java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject) at java.util.concurrent.locks.locksupport.park(locksupport.java:186) at java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject.await(abstractqueuedsynchronizer.java:2043) at org.apache.http.pool.poolentryfuture.await(poolentryfuture.java:133) at org.apache.http.pool.abstractconnpool.getpoolentryblocking(abstractconnpool.java:282) at org.apache.http.pool.abstractconnpool.access$000(abstractconnpool.java:64) at org.apache.http.pool.abstractconnpool$2.getpoolentry(abstractconnpool.java:177) at org.apache.http.pool.abstractconnpool$2.getpoolentry(abstractconnpool.java:170) at org.apache.http.pool.poolentryfuture.get(poolentryfuture.java:102) at org.apache.http.impl.conn.poolinghttpclientconnectionmanager.leaseconnection(poolinghttpclientconnectionmanager.java:240) at org.apache.http.impl.conn.poolinghttpclientconnectionmanager$1.get(poolinghttpclientconnectionmanager.java:227) at org.apache.http.impl.execchain.mainclientexec.execute(mainclientexec.java:173) at org.apache.http.impl.execchain.protocolexec.execute(protocolexec.java:195) at org.apache.http.impl.execchain.retryexec.execute(retryexec.java:85) at org.apache.http.impl.execchain.redirectexec.execute(redirectexec.java:108) at org.apache.http.impl.client.internalhttpclient.doexecute(internalhttpclient.java:186) at org.apache.http.impl.client.closeablehttpclient.execute(closeablehttpclient.java:82) at org.apache.http.impl.client.closeablehttpclient.execute(closeablehttpclient.java:106) at com.weimai.utils.httpclientutil.doget(httpclientutil.java:105) at com.weimai.utils.httpclientutil.doget(httpclientutil.java:87) at com.weimai.utils.weiboutil.checkuser(weiboutil.java:214) at com.weimai.web.userinfocontroller.newweibologin(userinfocontroller.java:1223) at sun.reflect.generatedmethodaccessor390.invoke(unknown source) at sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) at java.lang.reflect.method.invoke(method.java:606)
此时意识到问题应该出现http连接上,马上用netstat查看下801端口的连接状态,果然发现很多请求都是close_wait,这里简单解释下close_wait状态,如果我们的client程序处于close_wait状态的话,说明套接字是被动关闭的,整个流程应该是这样
因为如果是server端主动断掉当前连接的话,那么双方关闭这个tcp连接共需要四个packet
server -> fin -> client
server <- ack <- client
这时候server端处于fin_wait_2状态,而我们的程序处于close_wait状态
server <- fin <- client
这时client发送fin给server,client就置为last_ack状态。
server -> ack -> client
server回应了ack,那么client的套接字才会真正置为closed状态
我们的请求处于close_wait状态,而不是last_ack状态,说明还没有发fin给server,那么很简单,去看httpclientutil中如何处理就知道了,果然在查看httpclientutil代码中发现对于非正常关闭的http连接没有做abort,补充完善好try catch finally块后问题得到解决.