欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

解决Caused By:java.Lang.OutOfMemoryError: Unable To Create New Native Thread

程序员文章站 2022-03-04 13:36:39
...

业务背景

一台服务器上部署3个tomcat,并且每个tomcat的server.xml配置如下,只是端口不一致

<Executor name="tomcatThreadPool" namePrefix="wkApi-exec-"  maxThreads="1200" minSpareThreads="400"/>
<Connector  executor="tomcatThreadPool"  port="9000" 
               protocol="org.apache.coyote.http11.Http11NioProtocol"
               connectionTimeout="60000"
               redirectPort="8443"
               URIEncoding="UTF-8"
               maxPostSize="0"
               maxThreads="1200" 
               minSpareThreads="256"
               acceptCount="300" 
               compression="on" 
               compressionMinSize="2048" 
               noCompressionUserAgents="gozilla,traviata" 
               CompressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,text/json,application/x-javascript,application/javascript,application/json,image/png,image/jpeg"
               />

出现问题

1)发现日志出现Caused by:java.lang.OutOfMemoryError: unable to create new native thread,但是程序还在继续跑着,
2)当前起服务的用户无法操作任何命令,并且报:-bash: fork: retry: Resource temporarily unavailable

ok,不是服务器配置的错误。是当前用户起的服务创建的进程总和>当前用户的最大值了

通过:ulimit -a 命令

[appuser@123.57.86.172/10.173.41.156 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7415
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

max user processes (-u) 1024

可以看到当前用户最大只能起1024个进程

可以使用pstree -a|wc -l 统计当前进程总数

[appuser@123.57.86.172/10.173.41.156 ~]$ pstree -a|wc -l
75
 #将所有的进程树导出来
[appuser@123.57.86.172/10.173.41.156 ~]$ pstree -a > 1.log

可以查看所有的进程

解决办法:

1,限制所有tomcat的进程数总和小于1024,(ps: 最大值在9000左右)
2,通过 /etc/security/limits.d/90-nproc.conf 调整最大进程数,

可以看到其他用户限制在1024

[[email protected]123.57.86.172/10.173.41.156 ~]$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024
root       soft    nproc     unlimited