dbcp 连接池不合理的锁导致连接耗尽解决方案

dbcp 连接池不合理的锁导致连接耗尽解决方案

应用报错,表象来看是连接池爆满了。

org.springframework.transaction.cannotcreatetransactionexception: could not open jdbc connection for transaction; nested exception is org.apache.commons.dbcp.sqlnestedexception: cannot get a connection, pool exhausted
    at org.springframework.jdbc.datasource.datasourcetransactionmanager.dobegin(datasourcetransactionmanager.java:241) ~[spring-jdbc-3.2.2.release.jar:3.2.2.release]
    at com.alibaba.dubbo.remoting.transport.dispatcher.channeleventrunnable.run(channeleventrunnable.java:82) [dubbo-2.5.3.jar:2.5.3]
    at java.util.concurrent.threadpoolexecutor$worker.runtask(threadpoolexecutor.java:886) [na:1.6.0_33]
    at java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:908) [na:1.6.0_33]
    at java.lang.thread.run(thread.java:662) [na:1.6.0_33]
caused by: org.apache.commons.dbcp.sqlnestedexception: cannot get a connection, pool exhausted
    at org.apache.commons.dbcp.poolingdatasource.getconnection(poolingdatasource.java:103) ~[commons-dbcp.jar:1.2.1]
    at org.apache.commons.dbcp.basicdatasource.getconnection(basicdatasource.java:540) ~[commons-dbcp.jar:1.2.1]
    at com.eshore.crmpub.jdbc.datasource.multidatasource.getconnection(multidatasource.java:74) ~[crmpub-jdbc-1.0.jar:1.0]
    at org.springframework.jdbc.datasource.datasourcetransactionmanager.dobegin(datasourcetransactionmanager.java:203) ~[spring-jdbc-3.2.2.release.jar:3.2.2.release]
    ... 32 common frames omitted
caused by: java.util.nosuchelementexception: timeout waiting for idle object
    at org.apache.commons.pool.impl.genericobjectpool.borrowobject(genericobjectpool.java:1174) ~[commons-pool-1.6.jar:1.6]
    at org.apache.commons.dbcp.abandonedobjectpool.borrowobject(abandonedobjectpool.java:74) ~[commons-dbcp.jar:1.2.1]
    at org.apache.commons.dbcp.poolingdatasource.getconnection(poolingdatasource.java:95) ~[commons-dbcp.jar:1.2.1]
    ... 35 common frames omitted

应用堆栈，几十条线程都阻塞到dbcp池的poolableconnectionfactory.makeobject方法了，此方法在等待0x0000000709a638a0对象锁，但0x0000000709a638a0对象一直被其中一条线程执行poolableconnectionfactory.makeobject时锁了，而且这条线程获取到锁后还阻塞住了，所以导致后面几十条线程都blocked了。

"dubboserverhandler-132.121.95.136:28101-thread-300" daemon prio=10 tid=0x00007f02b4136800 nid=0x4954 waiting for monitor entry [0x00007f0286018000]
  java.lang.thread.state: blocked (on object monitor)
  at org.apache.commons.dbcp.poolableconnectionfactory.makeobject(poolableconnectionfactory.java:290)
  - waiting to lock <0x0000000709a638a0> (a org.apache.commons.dbcp.poolableconnectionfactory)
  at org.apache.commons.pool.impl.genericobjectpool.borrowobject(genericobjectpool.java:1188)
  at org.apache.commons.dbcp.abandonedobjectpool.borrowobject(abandonedobjectpool.java:74)
  at org.apache.commons.dbcp.poolingdatasource.getconnection(poolingdatasource.java:95)
  at org.apache.commons.dbcp.basicdatasource.getconnection(basicdatasource.java:540)
  at com.eshore.crmpub.jdbc.datasource.multidatasource.getconnection(multidatasource.java:74)
  at org.springframework.jdbc.datasource.datasourcetransactionmanager.dobegin(datasourcetransactionmanager.java:203)
  at org.springframework.transaction.support.abstractplatformtransactionmanager.gettransaction(abstractplatformtransactionmanager.java:372)
  at org.springframework.transaction.interceptor.transactionaspectsupport.createtransactionifnecessary(transactionaspectsupport.java:417)
  at org.springframework.transaction.interceptor.transactionaspectsupport.invokewithintransaction(transactionaspectsupport.java:255)
  at org.springframework.transaction.interceptor.transactioninterceptor.invoke(transactioninterceptor.java:94)
  at org.springframework.aop.framework.reflectivemethodinvocation.proceed(reflectivemethodinvocation.java:161)
  at org.springframework.aop.aspectj.methodinvocationproceedingjoinpoint.proceed(methodinvocationproceedingjoinpoint.java:80)
  at com.eshore.crm.service.common.pub.aspect.datasourcerouteaspect.around(datasourcerouteaspect.java:74)
  at sun.reflect.generatedmethodaccessor253.invoke(unknown source)
  at sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:25)
  at java.lang.reflect.method.invoke(method.java:597)
  at org.springframework.aop.aspectj.abstractaspectjadvice.invokeadvicemethodwithgivenargs(abstractaspectjadvice.java:621)
  at org.springframework.aop.aspectj.abstractaspectjadvice.invokeadvicemethod(abstractaspectjadvice.java:610)
  at org.springframework.aop.aspectj.aspectjaroundadvice.invoke(aspectjaroundadvice.java:65)
  at org.springframework.aop.framework.reflectivemethodinvocation.proceed(reflectivemethodinvocation.java:161)
  at org.springframework.aop.interceptor.exposeinvocationinterceptor.invoke(exposeinvocationinterceptor.java:91)
  at org.springframework.aop.framework.reflectivemethodinvocation.proceed(reflectivemethodinvocation.java:172)
  at org.springframework.aop.framework.cglibaopproxy$dynamicadvisedinterceptor.intercept(cglibaopproxy.java:631)
  at com.eshore.crm.service.sysmgr.sca.impl.sysmgrserviceimpl$$enhancerbycglib$$bfd71326.qrybulletinlist(<generated>)
  at com.alibaba.dubbo.common.bytecode.wrapper101.invokemethod(wrapper101.java)
  at com.alibaba.dubbo.rpc.proxy.javassist.javassistproxyfactory$1.doinvoke(javassistproxyfactory.java:46)
  at com.alibaba.dubbo.rpc.proxy.abstractproxyinvoker.invoke(abstractproxyinvoker.java:72)
  at com.alibaba.dubbo.rpc.protocol.invokerwrapper.invoke(invokerwrapper.java:53)
  at com.eshore.crmpub.sca.dubbo.filter.basefilter.invoke(basefilter.java:90)
  at com.alibaba.dubbo.rpc.protocol.protocolfilterwrapper$1.invoke(protocolfilterwrapper.java:91)
  at com.alibaba.dubbo.rpc.protocol.dubbo.dubboprotocol$1.reply(dubboprotocol.java:108)
  at com.alibaba.dubbo.remoting.exchange.support.header.headerexchangehandler.handlerequest(headerexchangehandler.java:84)
  at com.alibaba.dubbo.remoting.exchange.support.header.headerexchangehandler.received(headerexchangehandler.java:170)
  at com.alibaba.dubbo.remoting.transport.decodehandler.received(decodehandler.java:52)
  at com.alibaba.dubbo.remoting.transport.dispatcher.channeleventrunnable.run(channeleventrunnable.java:82)
  at java.util.concurrent.threadpoolexecutor$worker.runtask(threadpoolexecutor.java:886)
  at java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:908)
  at java.lang.thread.run(thread.java:662)

  locked ownable synchronizers:
  - <0x000000070a120f00> (a java.util.concurrent.locks.reentrantlock$nonfairsync)

"dubboserverhandler-132.121.95.136:28101-thread-290" daemon prio=10 tid=0x00007f03203da800 nid=0x4948 runnable [0x00007f0286a22000]
  java.lang.thread.state: runnable
  at java.net.socketinputstream.socketread0(native method)
  at java.net.socketinputstream.read(socketinputstream.java:129)
  at com.mysql.jdbc.util.readaheadinputstream.fill(readaheadinputstream.java:112)
  at com.mysql.jdbc.util.readaheadinputstream.readfromunderlyingstreamifnecessary(readaheadinputstream.java:159)
  at com.mysql.jdbc.util.readaheadinputstream.read(readaheadinputstream.java:187)
  - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.readaheadinputstream)
  at com.mysql.jdbc.mysqlio.readfully(mysqlio.java:3140)
  at com.mysql.jdbc.mysqlio.readpacket(mysqlio.java:597)
  at com.mysql.jdbc.mysqlio.dohandshake(mysqlio.java:1085)
  at com.mysql.jdbc.connectionimpl.coreconnect(connectionimpl.java:2494)
  at com.mysql.jdbc.connectionimpl.connectonetryonly(connectionimpl.java:2527)
  at com.mysql.jdbc.connectionimpl.createnewio(connectionimpl.java:2309)
  - locked <0x00000007159d1898> (a com.mysql.jdbc.jdbc4connection)
  at com.mysql.jdbc.connectionimpl.<init>(connectionimpl.java:834)
  at com.mysql.jdbc.jdbc4connection.<init>(jdbc4connection.java:46)
  at sun.reflect.generatedconstructoraccessor78.newinstance(unknown source)
  at sun.reflect.delegatingconstructoraccessorimpl.newinstance(delegatingconstructoraccessorimpl.java:27)
  at java.lang.reflect.constructor.newinstance(constructor.java:513)
  at com.mysql.jdbc.util.handlenewinstance(util.java:408)
  at com.mysql.jdbc.connectionimpl.getinstance(connectionimpl.java:419)
  at com.mysql.jdbc.nonregisteringdriver.connect(nonregisteringdriver.java:344)
  at org.apache.commons.dbcp.driverconnectionfactory.createconnection(driverconnectionfactory.java:37)
  at org.apache.commons.dbcp.poolableconnectionfactory.makeobject(poolableconnectionfactory.java:290)
  - locked <0x0000000709a638a0> (a org.apache.commons.dbcp.poolableconnectionfactory)
  at org.apache.commons.pool.impl.genericobjectpool.borrowobject(genericobjectpool.java:1188)
  at org.apache.commons.dbcp.abandonedobjectpool.borrowobject(abandonedobjectpool.java:74)
  at org.apache.commons.dbcp.poolingdatasource.getconnection(poolingdatasource.java:95)
  at org.apache.commons.dbcp.basicdatasource.getconnection(basicdatasource.java:540)
  at com.eshore.crmpub.jdbc.datasource.multidatasource.getconnection(multidatasource.java:74)
  at org.springframework.jdbc.datasource.datasourcetransactionmanager.dobegin(datasourcetransactionmanager.java:203)
  at org.springframework.transaction.support.abstractplatformtransactionmanager.gettransaction(abstractplatformtransactionmanager.java:372)
  at org.springframework.transaction.interceptor.transactionaspectsupport.createtransactionifnecessary(transactionaspectsupport.java:417)
  at org.springframework.transaction.interceptor.transactionaspectsupport.invokewithintransaction(transactionaspectsupport.java:255)
  at org.springframework.transaction.interceptor.transactioninterceptor.invoke(transactioninterceptor.java:94)
  at org.springframework.aop.framework.reflectivemethodinvocation.proceed(reflectivemethodinvocation.java:161)
  at org.springframework.aop.aspectj.methodinvocationproceedingjoinpoint.proceed(methodinvocationproceedingjoinpoint.java:80)
  at com.eshore.crm.service.common.pub.aspect.datasourcerouteaspect.around(datasourcerouteaspect.java:74)
  at sun.reflect.generatedmethodaccessor253.invoke(unknown source)
  at sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:25)
  at java.lang.reflect.method.invoke(method.java:597)
  at org.springframework.aop.aspectj.abstractaspectjadvice.invokeadvicemethodwithgivenargs(abstractaspectjadvice.java:621)
  at org.springframework.aop.aspectj.abstractaspectjadvice.invokeadvicemethod(abstractaspectjadvice.java:610)
  at org.springframework.aop.aspectj.aspectjaroundadvice.invoke(aspectjaroundadvice.java:65)
  at org.springframework.aop.framework.reflectivemethodinvocation.proceed(reflectivemethodinvocation.java:161)
  at org.springframework.aop.interceptor.exposeinvocationinterceptor.invoke(exposeinvocationinterceptor.java:91)
  at org.springframework.aop.framework.reflectivemethodinvocation.proceed(reflectivemethodinvocation.java:172)
  at org.springframework.aop.framework.cglibaopproxy$dynamicadvisedinterceptor.intercept(cglibaopproxy.java:631)
  at com.eshore.crm.service.cachemgr.sca.common.dictcacheserviceimpl$$enhancerbycglib$$e27e8e25.getdict(<generated>)
  at com.alibaba.dubbo.common.bytecode.wrapper93.invokemethod(wrapper93.java)
  at com.alibaba.dubbo.rpc.proxy.javassist.javassistproxyfactory$1.doinvoke(javassistproxyfactory.java:46)
  at com.alibaba.dubbo.rpc.proxy.abstractproxyinvoker.invoke(abstractproxyinvoker.java:72)
  at com.alibaba.dubbo.rpc.protocol.invokerwrapper.invoke(invokerwrapper.java:53)
  at com.eshore.crmpub.sca.dubbo.filter.basefilter.invoke(basefilter.java:90)
  at com.alibaba.dubbo.rpc.protocol.protocolfilterwrapper$1.invoke(protocolfilterwrapper.java:91)
  at com.alibaba.dubbo.rpc.protocol.dubbo.dubboprotocol$1.reply(dubboprotocol.java:108)
  at com.alibaba.dubbo.remoting.exchange.support.header.headerexchangehandler.handlerequest(headerexchangehandler.java:84)
  at com.alibaba.dubbo.remoting.exchange.support.header.headerexchangehandler.received(headerexchangehandler.java:170)
  at com.alibaba.dubbo.remoting.transport.decodehandler.received(decodehandler.java:52)
  at com.alibaba.dubbo.remoting.transport.dispatcher.channeleventrunnable.run(channeleventrunnable.java:82)
  at java.util.concurrent.threadpoolexecutor$worker.runtask(threadpoolexecutor.java:886)
  at java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:908)
  at java.lang.thread.run(thread.java:662)

  locked ownable synchronizers:
  - <0x000000070a0f9218> (a java.util.concurrent.locks.reentrantlock$nonfairsync)

可以看到其实获取锁的线程是阻塞在这里了，做mysql握手包的时候一直在等待读，阻塞在网络io了。

 java.lang.thread.state: runnable
  at java.net.socketinputstream.socketread0(native method)
  at java.net.socketinputstream.read(socketinputstream.java:129)
  at com.mysql.jdbc.util.readaheadinputstream.fill(readaheadinputstream.java:112)
  at com.mysql.jdbc.util.readaheadinputstream.readfromunderlyingstreamifnecessary(readaheadinputstream.java:159)
  at com.mysql.jdbc.util.readaheadinputstream.read(readaheadinputstream.java:187)
  - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.readaheadinputstream)
  at com.mysql.jdbc.mysqlio.readfully(mysqlio.java:3140)
  at com.mysql.jdbc.mysqlio.readpacket(mysqlio.java:597)
  at com.mysql.jdbc.mysqlio.dohandshake(mysqlio.java:1085)
  at com.mysql.jdbc.connectionimpl.coreconnect(connectionimpl.java:2494)
  at com.mysql.jdbc.connectionimpl.connectonetryonly(connectionimpl.java:2527)
  at com.mysql.jdbc.connectionimpl.createnewio(connectionimpl.java:2309)

这个是dbcp1.x版本在创建新的连接对象时会把整个连接工厂类对象锁了，一旦创建过程中发生阻塞就会导致整个池都死掉。这个问题很大。

解决方案，升级dbcp2.x版本或tomcat jdbc或druid连接池，推荐druid连接池，功能强大自带监控。

感谢阅读，希望能帮助到大家，谢谢大家对本站的支持！