欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

weblogic调优以及线程阻塞引起的宕机

程序员文章站 2022-03-08 08:37:13
...

由weblogic相关日志分析后认为2020-1-19 下午03时04分43秒宕机原因为weblogic的线程阻塞,引起weblogic宕机;

weblogic调优以及线程阻塞引起的宕机

####<2020-1-19 下午02时53分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579416823035> <BEA-310002> <49% of the total memory in the server is free> 
####<2020-1-19 下午02时55分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579416943036> <BEA-310002> <32% of the total memory in the server is free> 
####<2020-1-19 下午02时56分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579417003036> <BEA-310002> <43% of the total memory in the server is free> 
####<2020-1-19 下午03时00分11秒 CST> <Error> <HTTP> <VM_35_2_redhat> <nc1> <[ACTIVE] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1579417211420> <BEA-101017> <[aaa@qq.com[app:nc65 module:/ path: spec-version:2.5]] Root cause of ServletException.
javax.servlet.ServletException: not supported
	at nc.bs.framework.comn.serv.CommonServletDispatcher.doGet(CommonServletDispatcher.java:39)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at nc.bs.framework.server.LoggerServletFilter.doFilter(LoggerServletFilter.java:35)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
> 
####<2020-1-19 下午03时04分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579417483038> <BEA-310002> <32% of the total memory in the server is free> 

 

经排查后进行以下优化:

1)优化JDK内存;

2)修改StuckThreadMaxTime参数;

3)   增大线程数,防止线程阻塞问题。

 

优化JDK内存

setDomainEnv.sh

修改前:

if [ "${JAVA_VENDOR}" = "Sun" ] ; then

       WLS_MEM_ARGS_64BIT="-Xms256m -Xmx512m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms256m -Xmx512m"

       export WLS_MEM_ARGS_32BIT

else

       WLS_MEM_ARGS_64BIT="-Xms512m -Xmx512m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms512m -Xmx512m"

       export WLS_MEM_ARGS_32BIT

fi

 

MEM_PERM_SIZE_64BIT="-XX:PermSize=128m"

MEM_PERM_SIZE_32BIT="-XX:PermSize=48m"

 

MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=256m"

MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=128m"

 

MEM_ARGS="-Xms512m -Xmx1536m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:CompileThreshold=8000 -Dorg.owasp.esapi.resources=/home/ap/nc65/ierp/bin/esapi  -Dfile.encoding=utf-8"

 

修改后:

if [ "${JAVA_VENDOR}" = "Sun" ] ; then

       WLS_MEM_ARGS_64BIT="-Xms4096m -Xmx4096m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms1024m -Xmx1024m"

       export WLS_MEM_ARGS_32BIT

else

       WLS_MEM_ARGS_64BIT="-Xms4096m -Xmx4096m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms1024m -Xmx1024m"

       export WLS_MEM_ARGS_32BIT

fi

 

MEM_PERM_SIZE_64BIT="-XX:PermSize=1024m"

MEM_PERM_SIZE_32BIT="-XX:PermSize=512m"

 

MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=2048m"

MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=1024m"

 

MEM_ARGS="-Xms4096m -Xmx4096m -XX:PermSize=1024m -XX:MaxPermSize=2048m -XX:CompileThreshold=8000 -Dorg.owasp.esapi.resources=/home/ap/nc65/ierp/bin/esapi  -Dfile.encoding=utf-8"

 

修改StuckThreadMaxTime参数

config.xml 配置文件

修改所有 server下stuck-thread-max-time 参数

修改前:

<stuck-thread-max-time>600</stuck-thread-max-time>

修改后:

<stuck-thread-max-time>60000</stuck-thread-max-time>

 

增大线程数(64位操作系统 3000 ~ 5000 左右)

在所有server中增加参数

<self-tuning-thread-pool-size-min>2000</self-tuning-thread-pool-size-min>

         <self-tuning-thread-pool-size-max>4000</self-tuning-thread-pool-size-max>