how many processes can be created with python os.fork()

Question

If I do this below:

for i in range(10000):
    os.fork()

What's going on? don't consider reaping.. I just wanna know about os.fork() can make how much processes in linux, if it 's just like windows, only can make about 2000 processes, how the next 8000 processes will do?
Thanks.

`os.fork` creates a new process. To be pedantic, that new process includes one new thread, so a new thread is indeed created, but this is probably not what you mean. — William Pursell, Dec 08 '13 at 10:09
The same number as POSIX fork. Your question is not about python, but rather POSIX/Linux. — Michael Kruglos, Dec 08 '13 at 10:36
@msw I have maybe ten thousands virtual machines, I want to ssh every machine and then do something , so I use os.fork to do it. — user3079567, Dec 08 '13 at 14:34

score 3 · Accepted Answer · edited May 23 '17 at 11:50

3

os.fork() spawns a new OS-level process. Any limits on the number of those aren't dependent on Python, but on the operating system. According to this previous question, on Linux you can find out about any software-imposed constraints on that by looking at:

cat /proc/sys/kernel/pid_max

but that might be further restrained by /etc/security/limits.conf. If those don't hit you first, you will eventually run into problems with available hardware resources - your code is a fork bomb, and type of trivial denial of service attack (any software limits on the number of processes are set to avoid this kind of attack).

edited May 23 '17 at 11:50

Community

1
1

answered Dec 08 '13 at 10:29

lvc

34,233
10
73
98

2

there's also `ulimit -u` – Karoly Horvath Dec 08 '13 at 10:30
@lvc I understand it now, for instance ,my cpu,mem or any other limits only can support 2000 threads,how does the rest do? what bad things will happen? – user3079567 Dec 08 '13 at 14:30
@lvc Looks like I need a better idea.I find a way, I will divided into many groups of my 10,000 virtual machines. – user3079567 Dec 08 '13 at 14:54

Karoly Horvath · Answer 2 · 2013-12-08T10:28:08.947

2

The code doesn't do what you think it does.

When you're forking, you're actually duplicating the current process.

This means that after forking, both processes will continue the iteration from the same point. And then both will fork, and so on... The number of created threads will be exponential.

Example:

import os
for i in range(3):
    os.fork();

print "T"

When executed, you should see 2^3 = 8 processes.

Just to return to your original question, since fork creates processes, the limit on the number of threads is irrelevant.

edited Dec 08 '13 at 10:28

answered Dec 08 '13 at 10:22

Karoly Horvath

94,607
11
117
176

.In my code, during the loop ,if os.fork ,it will continue, otherwise,it will do something. so it should work well. but the effect you point out, it's indeed interesting,i didn't notice it before – user3079567 Dec 08 '13 at 14:22

Roland Smith · Answer 3 · 2013-12-08T10:41:18.287

The os.fork() function creates a copy of the calling process. Threads are created in Python by using the threading module.

The amount of processes you can create on a UNIX-like system such as Linux is generally limited by the amount of memory the computer has and certain limits set in the operating system. The output of the command ulimit -a shows all the limits.

The amount of threads you can create within a process is usually limited by the amount of stack space that the process has and the maximal amount of (virtua)l memory that the process is allowed to have. These limits are also shown by ulimit.

Note that when you are using the standard python implementation (a.k.a CPython), only one thread at a time can be executing Python bytecode.

how many processes can be created with python os.fork()

3 Answers3

Linked