21

I cannot create more than 32k Java threads in Linux machine with 15G memory.

maczniak
  • 1,082
  • 1
  • 8
  • 13
  • 1
    @guillaumegirod-vitouchkina It happened on a load generating machine for load tests. Due to bad design of 3rd-party legacy library, multiple threads were necessary per network connection. Then thread count was increased rapidly. Most threads are in an idle state. – maczniak Dec 24 '15 at 12:52
  • 1
    OK. I understand an sympathize. – guillaume girod-vitouchkina Dec 24 '15 at 13:03

3 Answers3

58

You can use a sample program to find out the current threads limit.

If you encounter Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread, check these:

  1. In small memory machines

    Every Java thread consume its own stack memory. Default stack size is 1024k (= 1M). You can reduce the stack size like java -Xss512k .... JVM cannot be started if the stack size is too low.

    And beware heap memory configurations: (initial) -Xms and (maximum) -Xmx. The more memory is allocated to heap, the less available memory for stack.

  2. System limits

    Some values in ulimit -a can affect a thread limit.

    • max memory size - unlimited on most 64bit machines
    • max user processes - linux treats threads like processes
    • virtual memory - unlimited on most 64bit machines. virtual memory usage is increased by -Xss configuration (default 1024k)

    You can change these values by (temporal) running ulimit command or (permanent) editing /etc/security/limits.conf.

  3. sys.kernel.threads-max

    This value is the system-global (including non-JVM processes) maximum number of threads. Check cat /proc/sys/kernel/threads-max, and increase if necessary.

    echo 999999 > /proc/sys/kernel/threads-max
    or
    sys.kernel.threads-max = 999999 in /etc/sysctl.conf to change permanently.

  4. sys.kernel.pid_max

    If cat /proc/sys/kernel/pid_max is similar to current limit, increase this. Linux treats threads like processes.

    echo 999999 > /proc/sys/kernel/pid_max
    or
    sys.kernel.pid_max = 999999 in /etc/sysctl.conf to change permanently.

    And you may need to increase sys.vm.max_map_count, too.

  5. sys.vm.max_map_count

    cat /proc/sys/vm/max_map_count should be at least (2 x thread-count).

    Attempt to protect stack guard pages failed. and OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed. error messages are emitted by JavaThread::create_stack_guard_pages(), and it calls os::guard_memory(). In Linux, this function is mprotect().

    echo 1999999 > /proc/sys/vm/max_map_count
    or
    sys.vm.max_map_count = 1999999 in /etc/sysctl.conf to change permanently.

Community
  • 1
  • 1
maczniak
  • 1,082
  • 1
  • 8
  • 13
  • I went through a few solutions only this offer the answer that works! – javaPhobic Jul 01 '16 at 00:57
  • 1
    I apply config #sysctl -p and error error: "sys.kernel.threads-max" is an unknown key error: "sys.kernel.pid_max" is an unknown key error: "sys.vm.max_map_count" is an unknown key – Thoman Jun 19 '19 at 03:49
  • Regarding the `-Xss` stack size option, is there some way to look at how much stack space your threads are using? Knowing how to configure this value is nice, but I'm wondering if there's a JMX metric or something I could look at to tell me what my average and/or maximum thread stack utilization is, so that I can decide whether I actually need to change this value or not. – Dasmowenator Mar 19 '21 at 21:51
17

Additional information for modern (systemd) linux systems.

There are many resources about this of values that may need tweaking (the other answer is a good source for most of them); however a new limit is imposed by way of the systemd "TasksMax" limit which sets pids.max on the cgroup.

For login sessions the UserTasksMax default is 33% of the kernel limit pids_max (usually 12,288) and can be override in /etc/systemd/logind.conf.

For services the DefaultTasksMax default is 15% of the kernel limit pids_max (usually 4,915). You can override it for the service by setting TasksMax in "systemctl edit" or update DefaultTasksMax in /etc/systemd/system.conf

Trent Lloyd
  • 1,832
  • 1
  • 15
  • 13
5

I ran into a similar problem in a Python program and the following worked for me. This is based on maczniak's answer above and https://superuser.com/questions/1219960/cannot-edit-proc-sys-kernel-threads-max.

echo kernel.threads-max = 1073741823 >> /etc/sysctl.conf && echo 1073741823 > /proc/sys/kernel/threads-max
echo kernel.pid_max = 999999 >> /etc/sysctl.conf && echo 999999 > /proc/sys/kernel/pid_max
echo vm.max_map_count = 2147483646 >> /etc/sysctl.conf && echo 2147483646 > /proc/sys/vm/max_map_count
echo vm.overcommit_memory = 1 >> /etc/sysctl.conf && echo 1 > /proc/sys/vm/overcommit_memory
echo fs.inotify.max_user_instances = 256 >> /etc/sysctl.conf && echo 256 > /proc/sys/fs/inotify/max_user_instances
sysctl -p

I also had to set DefaultTasksMax in /etc/systemd/system.conf (or /etc/systemd/user.conf for user-run services) to DefaultTasksMax=unlimited.

Systemd also applies a limit for programs run from a login-shell. These default to 4096 per user (will be increased to 12288) and are configured as UserTasksMax in the [Login] section of /etc/systemd/logind.conf.

That is from this StackExchange question. Setting my UserTasksMax to UserTasksMax=999999 worked for me.

Jonathan
  • 10,792
  • 5
  • 65
  • 85