weblogic调优以及线程阻塞引起的宕机

由weblogic相关日志分析后认为2020-1-19 下午03时04分43秒宕机原因为weblogic的线程阻塞，引起weblogic宕机；

####<2020-1-19 下午02时53分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579416823035> <BEA-310002> <49% of the total memory in the server is free> 
####<2020-1-19 下午02时55分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579416943036> <BEA-310002> <32% of the total memory in the server is free> 
####<2020-1-19 下午02时56分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579417003036> <BEA-310002> <43% of the total memory in the server is free> 
####<2020-1-19 下午03时00分11秒 CST> <Error> <HTTP> <VM_35_2_redhat> <nc1> <[ACTIVE] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1579417211420> <BEA-101017> <[[e-mail][app:nc65 module:/ path: spec-version:2.5]] Root cause of ServletException.
javax.servlet.ServletException: not supported
	at nc.bs.framework.comn.serv.CommonServletDispatcher.doGet(CommonServletDispatcher.java:39)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at nc.bs.framework.server.LoggerServletFilter.doFilter(LoggerServletFilter.java:35)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
> 
####<2020-1-19 下午03时04分43秒 CST> <Info> <Health> <VM_35_2_redhat> <nc1> <weblogic.GCMonitor> <<anonymous>> <> <> <1579417483038> <BEA-310002> <32% of the total memory in the server is free> 

 

经排查后进行以下优化：

1）优化JDK内存；

2）修改StuckThreadMaxTime参数；

3)   增大线程数，防止线程阻塞问题。

 

优化JDK内存

setDomainEnv.sh

修改前：

if [ "${JAVA_VENDOR}" = "Sun" ] ; then

       WLS_MEM_ARGS_64BIT="-Xms256m -Xmx512m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms256m -Xmx512m"

       export WLS_MEM_ARGS_32BIT

else

       WLS_MEM_ARGS_64BIT="-Xms512m -Xmx512m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms512m -Xmx512m"

       export WLS_MEM_ARGS_32BIT

fi

 

MEM_PERM_SIZE_64BIT="-XX:PermSize=128m"

MEM_PERM_SIZE_32BIT="-XX:PermSize=48m"

 

MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=256m"

MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=128m"

 

MEM_ARGS="-Xms512m -Xmx1536m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:CompileThreshold=8000 -Dorg.owasp.esapi.resources=/home/ap/nc65/ierp/bin/esapi  -Dfile.encoding=utf-8"

 

修改后：

if [ "${JAVA_VENDOR}" = "Sun" ] ; then

       WLS_MEM_ARGS_64BIT="-Xms4096m -Xmx4096m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms1024m -Xmx1024m"

       export WLS_MEM_ARGS_32BIT

else

       WLS_MEM_ARGS_64BIT="-Xms4096m -Xmx4096m"

       export WLS_MEM_ARGS_64BIT

       WLS_MEM_ARGS_32BIT="-Xms1024m -Xmx1024m"

       export WLS_MEM_ARGS_32BIT

fi

 

MEM_PERM_SIZE_64BIT="-XX:PermSize=1024m"

MEM_PERM_SIZE_32BIT="-XX:PermSize=512m"

 

MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=2048m"

MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=1024m"

 

MEM_ARGS="-Xms4096m -Xmx4096m -XX:PermSize=1024m -XX:MaxPermSize=2048m -XX:CompileThreshold=8000 -Dorg.owasp.esapi.resources=/home/ap/nc65/ierp/bin/esapi  -Dfile.encoding=utf-8"

 

修改StuckThreadMaxTime参数

config.xml 配置文件

修改所有 server下stuck-thread-max-time 参数

修改前：

<stuck-thread-max-time>600</stuck-thread-max-time>

修改后：

<stuck-thread-max-time>60000</stuck-thread-max-time>

 

增大线程数（64位操作系统 3000 ~ 5000 左右）

在所有server中增加参数

<self-tuning-thread-pool-size-min>2000</self-tuning-thread-pool-size-min>

         <self-tuning-thread-pool-size-max>4000</self-tuning-thread-pool-size-max>