Where does the limitation of the thread pool come from?

Question

Using the following code, I found that the maximum size of the multiprocessing.pool.ThreadPool is 11689 (on my machine). If I make it bigger, I get

RuntimeError: can't start new thread

Can somebody explain where this comes from? 11689 seems like a weird system constant... so maybe I ran out of a resource?

Please note: This is not about how to choose the best number of threads for a Thread pool. This question is about where the 11689 comes from?

Code

Are you on Windows or Linux? There's probably an OS-level limit, and 11689 is esseintially that number minus all the threads already running in all the other processes on your machine. — Tom Dalton, Apr 23 '18 at 11:02
@TomDalton `cat /proc/sys/kernel/threads-max 125150` looks about right. Should I post a community wiki answer? — Martin Thoma, Apr 23 '18 at 11:06
(See https://stackoverflow.com/q/344203/562769 for the command) — Martin Thoma, Apr 23 '18 at 11:13

score 0 · Answer 1 · answered Apr 24 '18 at 13:53

A brief look at the CPython source code handling threads suggests the interpreter is not delving into details too much when a thread creation fails. It just raises the RuntimeError you see.

According to pthread_create manpage, a thread creation can fail with the following error codes and reasons.

EAGAIN Insufficient resources to create another thread.

EAGAIN A system-imposed limit on the number of threads was encountered. There are a number of limits that may trigger this error: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of pro‐ cesses and threads for a real user ID, was reached; the kernel's system-wide limit on the number of processes and threads, /proc/sys/kernel/threads-max, was reached (see proc(5)); or the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5)).

...

My blind guess is that you are incurring in the first problem rather than the second. You might be running out of memory to allocate new threads.

A deeper explanation on ways a thread creation can fail can be found at this link.

This is a blind guess though. A more verbose error handling from the interpreter would surely help.

Where does the limitation of the thread pool come from?

Code

1 Answers1