java.lang.OutOfMemoryError: unable to create new native thread while starting wildfly

Question

I want to start a wildfly on my virtual server. Every time I run ./wildfly/bin/standalone.sh I get the following error while bootstrapping:

MSC000001: Failed to start service jboss.ironjacamar.idle-remover: org.jboss.msc.service.StartException in service jboss.ironjacamar.idle-remover: java.lang.OutOfMemoryError: unable to create new native thread
    at org.jboss.as.connector.subsystems.jca.IdleRemoverService.start(IdleRemoverService.java:53)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_45]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_45]
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) [rt.jar:1.8.0_45]
    at java.lang.Thread.start(Thread.java:714) [rt.jar:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) [rt.jar:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357) [rt.jar:1.8.0_45]
    at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) [rt.jar:1.8.0_45]
    at org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover.start(IdleRemover.java:137)
    at org.jboss.as.connector.subsystems.jca.IdleRemoverService.start(IdleRemoverService.java:51)
    ... 5 more

A ulimit -a gets me the following:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 385262
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 385262
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

As you can see, virtual memory is unlimited and max user processes is pretty hugh.
I also increased/decreased the stack size with -Xss256kb when starting the jvm as described in this post but without any success.
Also increasing max-threads by 50 in standalone.xml for every subsystems did not work (described in this post).
Do you have any other idea how to solve this problem?

Java version is

java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

Try increasing `open files`, see http://stackoverflow.com/a/15736766/3080094 — vanOekel, Jun 21 '15 at 09:47
Here are the JAVA_OPTS from from wildfly/bin/standalone.conf: `-Xms64m -Xmx1500m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true` These are the standard values expect -Xmx1500m — Allipon, Jun 22 '15 at 12:24

score 1 · Answer 1 · answered Jun 21 '15 at 06:21

1

To fix OutOfMemoryError, launch your java process using -Xmx2048m switch.

Hope this helps.

answered Jun 21 '15 at 06:21

Nitin Arora

2,650
1
26
27

This does not work as the error message does not point to heap space but to "unable to create new native thread". I also tried this but the error still occurs. – Allipon Jun 21 '15 at 16:17
Did you try increasing the open files handles as suggested by @vanOkeal? That looks like the last option. You have tried pretty much everything. – Nitin Arora Jun 21 '15 at 19:16
I did. In theory I understand what's happening but practically I have no idea why. On my local dev machine everything works fine with the very same settings for the wildfly. – Allipon Jun 21 '15 at 19:33

user140547 · Answer 2 · 2015-06-22T14:48:35.333

0

How many CPUs does your box have? IIRC, some thread pools in wildfly are creating threads in relation to the CPU count, so if you have unusually many CPUs, that might be a problem. Although it says it cannot create new threads it might still be a problem with memory (and not with max user processes) because every thread need memory as well.

Otherwise you might try to create a heap dump to find out how many thread are created by wildfly.

Maybe a complete startup log could also provide more information

edited Jun 22 '15 at 14:48

answered Jun 22 '15 at 09:17

user140547

7,750
3
28
80

Thanks for your reply. The server has just one CPU.I will try to create a heap dump. – Allipon Jun 22 '15 at 12:53
added hint on startup log – user140547 Jun 22 '15 at 14:48

score 0 · Answer 3 · answered Jun 22 '15 at 16:25

0

You need to increase the number of open files allowed. The key to the error is unable to create new native thread. On Linux if you execute ulimit -n 8192 that should fix it.

answered Jun 22 '15 at 16:25

James R. Perkins

16,800
44
60

score 0 · Answer 4 · answered Mar 08 '21 at 18:49

On Linux systems, try increasing vm.max_map_count sysctl value if your system supports a higher value. Default OS limits on Memory Map counts are generally low - 65530.

On a CentOS or RHEL machine,

sysctl -w vm.max_map_count=131072

To make this change permanent, echo "vm.max_map_count = 131072" >> /etc/sysctl.conf

java.lang.OutOfMemoryError: unable to create new native thread while starting wildfly

4 Answers4