weblogic 11g StuckThreadMaxTime 问题解决以及线程池、数据库连接池参数调优

程序员文章站 2022-04-29 08:50:33

...

问题背景

Java程序中使用原生POI 进行excel导出，数据量可能比较大。

问题日志

<2018-4-24 下午03时56分09秒 CST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "626" seconds working on the request "aaa@qq.com[
GET /xxx/xxx/xxx.do HTTP/1.1
Connection: keep-alive
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Referer: http://localhost:7001/zx.html
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.8
Cookie: WLS_HTTP_BRIDGE=bJZHhpZbGJMjt5qd9Mxpcgxb1FBxDZLYp4BTt7sgFTGYhN2GMbz9!-1353724554

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-21 "[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    org.apache.xmlbeans.impl.store.Saver$TextSaver.resize(Saver.java:1674)
    org.apache.xmlbeans.impl.store.Saver$TextSaver.preEmit(Saver.java:1286)
    org.apache.xmlbeans.impl.store.Saver$TextSaver.emit(Saver.java:1185)
    org.apache.xmlbeans.impl.store.Saver$TextSaver.emitElement(Saver.java:957)
    org.apache.xmlbeans.impl.store.Saver.processElement(Saver.java:396)
    org.apache.xmlbeans.impl.store.Saver.process(Saver.java:296)
    org.apache.xmlbeans.impl.store.Saver$TextSaver.saveToString(Saver.java:1854)
    org.apache.xmlbeans.impl.store.Cursor._xmlText(Cursor.java:544)
    org.apache.xmlbeans.impl.store.Cursor.xmlText(Cursor.java:2426)
    ^-- Holding lock: org.apache.xmlbeans.implaaa@qq.com149d860[thin lock]
    org.apache.xmlbeans.impl.values.XmlObjectBase.xmlText(XmlObjectBase.java:1485)
    org.apache.xmlbeans.impl.values.XmlObjectBase.toString(XmlObjectBase.java:1472)
    ^-- Holding lock: org.apache.xmlbeans.implaaa@qq.com149d860[thin lock]
    org.apache.poi.xssf.usermodel.extensions.XSSFCellBorder.equals(XSSFCellBorder.java:180)
    java.util.ArrayList.indexOf(ArrayList.java:210)
    org.apache.poi.xssf.model.StylesTable.putBorder(StylesTable.java:448)
    org.apache.poi.xssf.usermodel.XSSFCellStyle.setBottomBorderColor(XSSFCellStyle.java:946)
    org.apache.poi.xssf.usermodel.XSSFCellStyle.setBottomBorderColor(XSSFCellStyle.java:935)

问题分析：

此问题是由于处理请求超时引起的，系统配置的处理时间是600s，但是该线程处理了626后，仍然没将请求释放，所以报了这个错误。如果发送该请求较多，很有可能会导致weblogic的线程阻塞，严重会引起weblogic挂起现象。

问题解决：

1）修改StuckThreadMaxTime参数，将默认的600s改成1200s，或者其它适合的值。
weblogic 11g StuckThreadMaxTime 问题解决以及线程池、数据库连接池参数调优

2）增大线程数，防止线程阻塞问题。
3）优化程序，减少处理时间。

其他解决方案（从sql优化入手）

该异常出现的原因是资源请求的时间超出了weblogic设定的600s，造成资源排队请求，如果类似的操作很多的话，那么会造成大面积的资源请求队列，从而引起weblogic无法正常提供服务，严重时引起weblogic崩溃。那么这种原因是如何导致的呢？
首先，我们从测试服务器上发现，出现这种情况的原因是因为该请求的时间过长，于是从该请求的数据处理过程入手进行分析，发现该请求的sql语句，在sql/plus下执行时间过长，如下：
如表大概10W数据，在sql/plus下执行时间超长，造成请求weblogic反应时间超出默认值，从而引起资源排队请求的问题，引起服务器不稳定运行。那么出现了这种问题，怎么解决呢？我们的解决方法是对该sql语句进行优化处理：
1:对INFO_SIGN,PPMC等字段建立规范表，从数据库中进行查询，尽量减少in的使用
2:对<>等操作符不使用,使用> or <等方式来代替
3:尽量减少排序order by，rownum的使用，只在关键时刻进行使用，其他时刻能够不使用的就不进行使用。
通过以上方式来减少资源请求时间，从而减少以上异常的发生，来保证服务器的正常运行。

线程池、数据库连接池调优

线程池

修改config.xml中的<self-tuning-thread-pool-size-min>

如下：

  <server>
    <name>AdminServer</name>
    <reverse-dns-allowed>false</reverse-dns-allowed>
    <native-io-enabled>true</native-io-enabled>
    <thread-pool-percent-socket-readers>33</thread-pool-percent-socket-readers>
    <ssl>
      <login-timeout-millis>25000</login-timeout-millis>
    </ssl>
    <max-open-sock-count>-1</max-open-sock-count>
    <stuck-thread-max-time>1200</stuck-thread-max-time>
    <stuck-thread-timer-interval>60</stuck-thread-timer-interval>
    <self-tuning-thread-pool-size-min>50</self-tuning-thread-pool-size-min>
    <self-tuning-thread-pool-size-max>300</self-tuning-thread-pool-size-max>
    <listen-address></listen-address>
    <accept-backlog>300</accept-backlog>
    <login-timeout-millis>5000</login-timeout-millis>
    <low-memory-time-interval>3600</low-memory-time-interval>
    <low-memory-sample-size>10</low-memory-sample-size>
    <low-memory-granularity-level>5</low-memory-granularity-level>
    <low-memory-gc-threshold>5</low-memory-gc-threshold>
  </server>

数据库连接池：

在你的domin下，config下，jdbc下，对应的数据源xml配置中
增加

 <jdbc-connection-pool-params>
    <initial-capacity>2</initial-capacity>
    <max-capacity>15</max-capacity>
    <capacity-increment>1</capacity-increment>
    <test-table-name>SQL SELECT 1</test-table-name>
    <statement-cache-size>10</statement-cache-size>
    <statement-cache-type>LRU</statement-cache-type>
  </jdbc-connection-pool-params>

在控制台中如下操作：
weblogic 11g StuckThreadMaxTime 问题解决以及线程池、数据库连接池参数调优

——————————————日志分隔符————————————————————————

完整相关日志如下：

Tue Apr 24 16:04:44 2018
BEA JRockit(R) R27.6.5-32_o-121899-1.6.0_14-20091001-2107-windows-ia32

"Main Thread" id=1 idx=0x4 tid=5492 prio=5 alive, in native, waiting
    -- Waiting for notification on: weblogic/t3/srvr/aaa@qq.com[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:849)
    ^-- Lock released while waiting: weblogic/t3/srvr/aaa@qq.com[fat lock]
    at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:465)
    at weblogic/Server.main(Server.java:67)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"(Signal Handler)" id=2 idx=0x8 tid=7444 prio=5 alive, in native, daemon

"(GC Main Thread)" id=3 idx=0xc tid=9156 prio=5 alive, in native, daemon

"(GC Worker Thread 1)" id=? idx=0x10 tid=8700 prio=5 alive, in native, daemon

"(GC Worker Thread 2)" id=? idx=0x14 tid=9708 prio=5 alive, in native, daemon

"(GC Worker Thread 3)" id=? idx=0x18 tid=1280 prio=5 alive, in native, daemon

"(GC Worker Thread 4)" id=? idx=0x1c tid=7756 prio=5 alive, in native, daemon

"(Code Generation Thread 1)" id=4 idx=0x20 tid=8888 prio=5 alive, in native, native_waiting, daemon

"(Code Optimization Thread 1)" id=5 idx=0x24 tid=492 prio=5 alive, in native, native_waiting, daemon

"(VM Periodic Task)" id=6 idx=0x28 tid=4764 prio=10 alive, in native, daemon

"(Attach Listener)" id=7 idx=0x2c tid=4772 prio=5 alive, in native, daemon

"Finalizer" id=8 idx=0x30 tid=9244 prio=8 alive, in native, native_waiting, daemon
    at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
    at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
    at jrockit/memory/Finalizer$4.run(Finalizer.java:159)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Reference Handler" id=9 idx=0x34 tid=1936 prio=10 alive, in native, native_waiting, daemon
    at java/lang/ref/Reference.waitForActivatedQueue()Ljava/lang/ref/Reference;(Native Method)
    at java/lang/ref/Reference.access$100(Reference.java:11)
    at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:79)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"(Sensor Event Thread)" id=10 idx=0x38 tid=8836 prio=5 alive, in native, daemon

"JDWP Transport Listener: dt_Socket" id=11 idx=0x3c tid=8400 prio=10 alive, in native, daemon

"JDWP Event Helper Thread" id=12 idx=0x40 tid=452 prio=10 alive, in native, native_waiting, daemon

"Timer-0" id=15 idx=0x44 tid=9472 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/aaa@qq.com[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at java/util/TimerThread.mainLoop(Timer.java:483)
    ^-- Lock released while waiting: java/util/aaa@qq.com[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Timer-1" id=16 idx=0x48 tid=6184 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/aaa@qq.com[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/util/TimerThread.mainLoop(Timer.java:509)
    ^-- Lock released while waiting: java/util/aaa@qq.com[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=17 idx=0x4c tid=9948 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/aaa@qq.com[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/aaa@qq.com[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"weblogic.time.TimeEventGenerator" id=18 idx=0x50 tid=9340 prio=9 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/time/common/internal/aaa@qq.com[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at weblogic/time/common/internal/TimeTable.snooze(TimeTable.java:286)
    ^-- Lock released while waiting: weblogic/time/common/internal/aaa@qq.com[fat lock]
    at weblogic/time/common/internal/TimeEventGenerator.run(TimeEventGenerator.java:117)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"JMAPI event thread" id=19 idx=0x54 tid=8376 prio=5 alive, in native, native_waiting, daemon

"weblogic.timers.TimerThread" id=20 idx=0x58 tid=1064 prio=9 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/timers/internal/aaa@qq.com[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at weblogic/timers/internal/TimerThread$Thread.run(TimerThread.java:267)
    ^-- Lock released while waiting: weblogic/timers/internal/aaa@qq.com[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-7" id=24 idx=0x60 tid=1112 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x02E3B590
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)
    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)
    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x64 tid=1844 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x68 tid=484 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x6c tid=932 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=28 idx=0x70 tid=5992 prio=5 alive, in native, daemon
    at weblogic/socket/NTSocketMuxer.getIoCompletionResult(Lweblogic/socket/NTSocketMuxer$IoCompletionData;)Z(Native Method)
    at weblogic/socket/NTSocketMuxer.processSockets(NTSocketMuxer.java:81)
    at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
    at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
    at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"VDE Transaction Processor Thread" id=31 idx=0x74 tid=3728 prio=2 alive, in native, waiting, daemon
    -- Waiting for notification on: com/octetstring/vde/backend/standard/TransactionProcessor@0x03F42670[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at com/octetstring/vde/backend/standard/TransactionProcessor.waitChange(TransactionProcessor.java:367)
    ^-- Lock released while waiting: com/octetstring/vde/backend/standard/TransactionProcessor@0x03F42670[fat lock]
    at com/octetstring/vde/backend/standard/TransactionProcessor.run(TransactionProcessor.java:212)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"DoSManager" id=33 idx=0x7c tid=9032 prio=6 alive, in native, sleeping, native_waiting, daemon
    at java/lang/Thread.sleep(J)V(Native Method)
    at com/octetstring/vde/DoSManager.run(DoSManager.java:433)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"LDAPConnThread-0 ldap://10.20.8.67:13001" id=34 idx=0x80 tid=8316 prio=5 alive, in native, daemon
    at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
    at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)
    at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
    at java/net/SocketInputStream.read(SocketInputStream.java:129)
    at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)
    at java/io/BufferedInputStream.read(BufferedInputStream.java:237)
    ^-- Holding lock: java/io/BufferedInputStream@0x04343BF0[thin lock]
    at netscape/ldap/ber/stream/BERElement.getElement(BERElement.java:101)
    at netscape/ldap/LDAPConnThread.run(LDAPConnThread.java:538)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-11" id=35 idx=0x84 tid=1972 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x042C5FC8
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)
    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)
    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-12" id=36 idx=0x88 tid=5996 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x04DF2408
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at weblogic/utils/concurrent/JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
    at weblogic/store/internal/PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:567)
    at weblogic/store/internal/PersistentStoreImpl.run(PersistentStoreImpl.java:615)
    at weblogic/store/internal/PersistentStoreImpl$2.run(PersistentStoreImpl.java:383)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=37 idx=0x8c tid=6408 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x052046C8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x052046C8[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ConnectionStateManager-0" id=38 idx=0x90 tid=7476 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x046F7BE8
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
    at org/apache/curator/framework/state/ConnectionStateManager.processEvents(ConnectionStateManager.java:175)
    at org/apache/curator/framework/state/ConnectionStateManager.access$000(ConnectionStateManager.java:42)
    at org/apache/curator/framework/state/ConnectionStateManager$1.call(ConnectionStateManager.java:108)
    at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303)
    at java/util/concurrent/FutureTask.run(FutureTask.java:138)
    at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (10.20.8.46:2181)" id=39 idx=0x94 tid=3140 prio=5 alive, in native, daemon
    at sun/nio/ch/PollArrayWrapper.poll0(JIJ)I(Native Method)
    at sun/nio/ch/PollArrayWrapper.poll(PollArrayWrapper.java:74)
    at sun/nio/ch/WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:63)
    at sun/nio/ch/SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    ^-- Holding lock: sun/nio/ch/Util$1@0x0478EFE8[thin lock]
    ^-- Holding lock: java/util/Collections$UnmodifiableSet@0x0478EF60[thin lock]
    ^-- Holding lock: sun/nio/ch/WindowsSelectorImpl@0x0478E6E8[thin lock]
    at sun/nio/ch/SelectorImpl.select(SelectorImpl.java:80)
    at org/apache/zookeeper/ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    at org/apache/zookeeper/ClientCnxn$SendThread.run(ClientCnxn.java:1081)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'-EventThread" id=40 idx=0x98 tid=8732 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x03DC06A0
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/apache/zookeeper/ClientCnxn$EventThread.run(ClientCnxn.java:494)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"CuratorFramework-0" id=41 idx=0x9c tid=7176 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x046E6908
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:160)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:39)
    at org/apache/curator/framework/imps/CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:652)
    at org/apache/curator/framework/imps/CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:55)
    at org/apache/curator/framework/imps/CuratorFrameworkImpl$3.call(CuratorFrameworkImpl.java:243)
    at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:303)
    at java/util/concurrent/FutureTask.run(FutureTask.java:138)
    at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-13" id=42 idx=0xa0 tid=10248 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: org/apache/commons/httpclient/util/IdleConnectionTimeoutThread@0x03DC9830[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/apache/commons/httpclient/util/IdleConnectionTimeoutThread.run(IdleConnectionTimeoutThread.java:108)
    ^-- Lock released while waiting: org/apache/commons/httpclient/util/IdleConnectionTimeoutThread@0x03DC9830[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-14" id=43 idx=0xa4 tid=10356 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x091D05C0
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:63)
    at kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:33)
    at kafka/utils/IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
    at kafka/utils/IteratorTemplate.hasNext(IteratorTemplate.scala:58)
    at com/paic/groupOperateAnalysis/service/impl/SearchEngine$1.run(SearchEngine.java:75)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"metrics-meter-tick-thread-1" id=45 idx=0xa8 tid=10372 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:164)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"metrics-meter-tick-thread-2" id=46 idx=0xac tid=10376 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:164)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ZkClient-EventThread-48-30.4.139.159:35450,30.4.139.160:35450,30.4.139.161:35450" id=48 idx=0xb4 tid=10388 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x093C0D98
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/I0Itec/zkclient/ZkEventThread.run(ZkEventThread.java:67)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-14-SendThread(30.4.139.161:35450)" id=49 idx=0xb8 tid=10392 prio=5 alive, in native, daemon
    at sun/nio/ch/PollArrayWrapper.poll0(JIJ)I(Native Method)
    at sun/nio/ch/PollArrayWrapper.poll(PollArrayWrapper.java:74)
    at sun/nio/ch/WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:63)
    at sun/nio/ch/SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    ^-- Holding lock: sun/nio/ch/Util$1@0x093C1800[thin lock]
    ^-- Holding lock: java/util/Collections$UnmodifiableSet@0x093C17F0[thin lock]
    ^-- Holding lock: sun/nio/ch/WindowsSelectorImpl@0x093C15E0[thin lock]
    at sun/nio/ch/SelectorImpl.select(SelectorImpl.java:80)
    at org/apache/zookeeper/ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    at org/apache/zookeeper/ClientCnxn$SendThread.run(ClientCnxn.java:1081)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-14-EventThread" id=50 idx=0xbc tid=10396 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x0B306AA8
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/apache/zookeeper/ClientCnxn$EventThread.run(ClientCnxn.java:494)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"kafka-consumer-scheduler-0" id=51 idx=0xc0 tid=10404 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:164)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-18" id=52 idx=0xc4 tid=10416 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x066516C0
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:63)
    at kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:33)
    at kafka/utils/IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
    at kafka/utils/IteratorTemplate.hasNext(IteratorTemplate.scala:58)
    at com/paic/groupOperateAnalysis/service/impl/SearchEngine$1.run(SearchEngine.java:75)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"dshbrd-search-engine-dev_IQSZ-L3025-1524554155067-8b86727d_watcher_executor" id=53 idx=0xc8 tid=10420 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2054)
    at kafka/consumer/ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:544)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"ZkClient-EventThread-54-30.4.139.159:35450,30.4.139.160:35450,30.4.139.161:35450" id=54 idx=0xcc tid=10424 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x06163D10
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/I0Itec/zkclient/ZkEventThread.run(ZkEventThread.java:67)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-18-SendThread(30.4.139.159:35450)" id=55 idx=0xd0 tid=10428 prio=5 alive, in native, daemon
    at sun/nio/ch/PollArrayWrapper.poll0(JIJ)I(Native Method)
    at sun/nio/ch/PollArrayWrapper.poll(PollArrayWrapper.java:74)
    at sun/nio/ch/WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:63)
    at sun/nio/ch/SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    ^-- Holding lock: sun/nio/ch/Util$1@0x0664C770[thin lock]
    ^-- Holding lock: java/util/Collections$UnmodifiableSet@0x0664C760[thin lock]
    ^-- Holding lock: sun/nio/ch/WindowsSelectorImpl@0x0664C550[thin lock]
    at sun/nio/ch/SelectorImpl.select(SelectorImpl.java:80)
    at org/apache/zookeeper/ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    at org/apache/zookeeper/ClientCnxn$SendThread.run(ClientCnxn.java:1081)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-18-EventThread" id=56 idx=0xd4 tid=10432 prio=5 alive, in native, parked, daemon
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x0664E310
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java/util/concurrent/LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org/apache/zookeeper/ClientCnxn$EventThread.run(ClientCnxn.java:494)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"kafka-consumer-scheduler-0" id=57 idx=0xd8 tid=10436 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java/util/concurrent/DelayQueue.take(DelayQueue.java:164)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"dshbrd-search-engine-dev_IQSZ-L3025-1524554158815-ae234a20_watcher_executor" id=58 idx=0xdc tid=10440 prio=5 alive, in native, parked, daemon
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:198)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2054)
    at kafka/consumer/ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:544)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-1" id=60 idx=0xe4 tid=10564 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3AF970[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3AF970[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-2" id=61 idx=0xe8 tid=10568 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B01A0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B01A0[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-3" id=62 idx=0xec tid=10572 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B0978[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B0978[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-4" id=63 idx=0xf0 tid=10576 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B1150[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B1150[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-5" id=64 idx=0xf4 tid=10580 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B1928[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B1928[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-6" id=65 idx=0xf8 tid=10584 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B2100[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B2100[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-7" id=66 idx=0xfc tid=10588 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B28D8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0B3B28D8[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-8" id=67 idx=0x100 tid=10592 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x060A8A38[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x060A8A38[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-9" id=68 idx=0x104 tid=10596 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0668B9A0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0668B9A0[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_Worker-10" id=69 idx=0x108 tid=10600 prio=5 alive, in native, waiting
    -- Waiting for notification on: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0668C4B0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
    ^-- Lock released while waiting: org/quartz/simpl/SimpleThreadPool$WorkerThread@0x0668C4B0[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"schedulerFactory_QuartzSchedulerThread" id=70 idx=0x10c tid=10604 prio=5 alive, in native, waiting
    -- Waiting for notification on: java/lang/Object@0x06798AF8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at org/quartz/core/QuartzSchedulerThread.run(QuartzSchedulerThread.java:245)
    ^-- Lock released while waiting: java/lang/Object@0x06798AF8[fat lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Timer-2" id=71 idx=0x110 tid=10608 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: java/util/TaskQueue@0x04931AE0[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/util/TimerThread.mainLoop(Timer.java:509)
    ^-- Lock released while waiting: java/util/TaskQueue@0x04931AE0[fat lock]
    at java/util/TimerThread.run(Timer.java:462)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=83 idx=0x140 tid=10740 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x0B2D7AC8[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0B2D7AC8[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=86 idx=0x14c tid=8948 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/work/ExecuteThread@0x0F235B30[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Object.wait(Object.java:485)
    at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:157)
    ^-- Lock released while waiting: weblogic/work/ExecuteThread@0x0F235B30[fat lock]
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:178)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"dshbrd-search-engine-dev_IQSZ-L3025-1524554158815-ae234a20-leader-finder-thread" id=87 idx=0x150 tid=10124 prio=5 alive, in native, parked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x0664F210
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at kafka/consumer/ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:61)
    at kafka/utils/ShutdownableThread.run(ShutdownableThread.scala:60)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"dshbrd-search-engine-dev_IQSZ-L3025-1524554155067-8b86727d-leader-finder-thread" id=88 idx=0x154 tid=9944 prio=5 alive, in native, parked
    -- Parking to wait for: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x030F3BA8
    at jrockit/vm/Locks.park0(J)V(Native Method)
    at jrockit/vm/Locks.park(Locks.java:2517)
    at sun/misc/Unsafe.park(ZJ)V(Native Method)
    at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)
    at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at kafka/consumer/ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:61)
    at kafka/utils/ShutdownableThread.run(ShutdownableThread.scala:60)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=89 idx=0x158 tid=10460 prio=5 alive, in native, waiting, daemon
    -- Waiting for notification on: weblogic/t3/srvr/T3Srvr$2@0x03E62930[fat lock]
    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
    at java/lang/Object.wait(J)V(Native Method)
    at java/lang/Thread.join(Thread.java:1143)
    ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr$2@0x03E62930[fat lock]
    at java/lang/Thread.join(Thread.java:1196)
    at java/lang/ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
    at java/lang/ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
    at java/lang/Shutdown.runHooks(Shutdown.java:79)
    at java/lang/Shutdown.sequence(Shutdown.java:123)
    at java/lang/Shutdown.exit(Shutdown.java:168)
    ^-- Holding lock: java/lang/Class@0x02C28598[thin lock]
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-1" id=14 idx=0x15c tid=4976 prio=5 alive, in native, native_blocked
    at jrockit/vm/RNI.generateVirtualCode(Ljava/lang/Object;II)I(Native Method)
    at weblogic/application/internal/flow/HeadWorkContextFlow.unprepare(HeadWorkContextFlow.java:0)
    at weblogic/application/internal/BaseDeployment$1.previous(BaseDeployment.java:1233)
    at weblogic/application/utils/StateMachineDriver.previousState(StateMachineDriver.java:329)
    at weblogic/application/utils/StateMachineDriver.previousState(StateMachineDriver.java:309)
    at weblogic/application/internal/BaseDeployment.unprepare(BaseDeployment.java:495)
    at weblogic/application/internal/SingleModuleDeployment.unprepare(SingleModuleDeployment.java:39)
    at weblogic/application/internal/DeploymentStateChecker.unprepare(DeploymentStateChecker.java:205)
    at weblogic/deploy/internal/targetserver/AppContainerInvoker.unprepare(AppContainerInvoker.java:117)
    at weblogic/deploy/internal/targetserver/BasicDeployment.unprepare(BasicDeployment.java:287)
    at weblogic/management/deploy/internal/ConfiguredDeployments.undeployPreStandbyInternalApps(ConfiguredDeployments.java:108)
    at weblogic/management/deploy/internal/DeploymentServerService.undeployPreStandbyInternalApps(DeploymentServerService.java:167)
    at weblogic/management/deploy/internal/DeploymentPreStandbyServerService.halt(DeploymentPreStandbyServerService.java:36)
    at weblogic/t3/srvr/ServerServicesManager.haltInternal(ServerServicesManager.java:504)
    at weblogic/t3/srvr/ServerServicesManager.halt(ServerServicesManager.java:336)
    at weblogic/t3/srvr/T3Srvr.shutdown(T3Srvr.java:986)
    at weblogic/t3/srvr/T3Srvr.forceShutdown(T3Srvr.java:892)
    at weblogic/t3/srvr/T3Srvr$2.run(T3Srvr.java:905)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"Thread-24" id=90 idx=0x170 tid=9840 prio=5 alive, in native, daemon
    at bea/jmapi/DiagnosticCommandImpl.execute(Ljava/lang/String;Ljava/io/FileDescriptor;)V(Native Method)
    at bea/jmapi/DiagnosticCommandImpl.execute(DiagnosticCommandImpl.java:50)
    at com/bea/jvm/DiagnosticCommand.execute(DiagnosticCommand.java:242)
    at com/bea/jvm/DiagnosticCommand$Command.execute(DiagnosticCommand.java:394)
    at bea/jmapi/ThreadSystemImpl.getThreadStackDump(ThreadSystemImpl.java:98)
    at weblogic/platform/JRockitVM.threadDump(JRockitVM.java:61)
    at weblogic/t3/srvr/T3Srvr.logThreadDump(T3Srvr.java:280)
    at weblogic/t3/srvr/ServerLifeCycleTimerThread.run(ServerLifeCycleTimerThread.java:77)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=91 idx=0x174 tid=6896 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=92 idx=0x178 tid=11124 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=93 idx=0x17c tid=10316 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=94 idx=0x180 tid=10924 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=95 idx=0x184 tid=11208 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=96 idx=0x188 tid=9872 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=97 idx=0x18c tid=8372 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=98 idx=0x190 tid=7252 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=99 idx=0x194 tid=10636 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=100 idx=0x198 tid=8736 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=101 idx=0x19c tid=5012 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=102 idx=0x1a0 tid=7468 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=103 idx=0x1a4 tid=10484 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=104 idx=0x1a8 tid=8852 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=105 idx=0x1ac tid=10904 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=106 idx=0x1b0 tid=7472 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=107 idx=0x1b4 tid=6240 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=108 idx=0x1b8 tid=9548 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=109 idx=0x1bc tid=11216 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=110 idx=0x1c0 tid=10400 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=111 idx=0x1c4 tid=3328 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=112 idx=0x1c8 tid=11228 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=113 idx=0x1cc tid=7524 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=114 idx=0x1d0 tid=10260 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=115 idx=0x1d4 tid=10648 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

"SIGINT handler" id=116 idx=0x1d8 tid=8788 prio=5 alive, in native, blocked, daemon
    -- Blocked trying to get lock: java/lang/Class@0x02C28598[thin lock]
    at jrockit/vm/Threads.sleep(I)V(Native Method)
    at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)
    at java/lang/Shutdown.exit(Shutdown.java:164)
    at java/lang/Terminator$1.handle(Terminator.java:35)
    at sun/misc/Signal$1.run(Signal.java:195)
    at java/lang/Thread.run(Thread.java:619)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace

Blocked lock chains
===================
Chain 2:
"SIGINT handler" id=92 idx=0x178 tid=11124 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 3:
"SIGINT handler" id=93 idx=0x17c tid=10316 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 4:
"SIGINT handler" id=94 idx=0x180 tid=10924 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 5:
"SIGINT handler" id=95 idx=0x184 tid=11208 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 6:
"SIGINT handler" id=96 idx=0x188 tid=9872 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 7:
"SIGINT handler" id=97 idx=0x18c tid=8372 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 8:
"SIGINT handler" id=98 idx=0x190 tid=7252 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 9:
"SIGINT handler" id=99 idx=0x194 tid=10636 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 10:
"SIGINT handler" id=100 idx=0x198 tid=8736 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 11:
"SIGINT handler" id=101 idx=0x19c tid=5012 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 12:
"SIGINT handler" id=102 idx=0x1a0 tid=7468 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 13:
"SIGINT handler" id=103 idx=0x1a4 tid=10484 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 14:
"SIGINT handler" id=104 idx=0x1a8 tid=8852 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 15:
"SIGINT handler" id=105 idx=0x1ac tid=10904 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 16:
"SIGINT handler" id=106 idx=0x1b0 tid=7472 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 17:
"SIGINT handler" id=107 idx=0x1b4 tid=6240 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 18:
"SIGINT handler" id=108 idx=0x1b8 tid=9548 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 19:
"SIGINT handler" id=109 idx=0x1bc tid=11216 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 20:
"SIGINT handler" id=110 idx=0x1c0 tid=10400 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 21:
"SIGINT handler" id=111 idx=0x1c4 tid=3328 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 22:
"SIGINT handler" id=112 idx=0x1c8 tid=11228 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 23:
"SIGINT handler" id=113 idx=0x1cc tid=7524 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 24:
"SIGINT handler" id=114 idx=0x1d0 tid=10260 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 25:
"SIGINT handler" id=115 idx=0x1d4 tid=10648 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Chain 26:
"SIGINT handler" id=116 idx=0x1d8 tid=8788 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 in chain 1

Open lock chains
================
Chain 1:
"SIGINT handler" id=91 idx=0x174 tid=6896 waiting for java/lang/Class@0x02C28598 held by:
"SIGINT handler" id=89 idx=0x158 tid=10460 (waiting on notification)

===== END OF THREAD DUMP ===============

>

上一篇：自定义python异步非阻塞Web框架实例代码

下一篇： ps插件怎么安装

weblogic 11g StuckThreadMaxTime 问题解决 以及 线程池、数据库连接池参数调优