How many child processes can a parent spawn before becoming infeasible?

Question

I'm a C programmer learning about fork(), exec(), and wait() for the first time. I'm also whiteboarding a Standard C program which will run on Linux and potentially need a lot of child processes. What I can't gauge is... how many child processes are too many for one parent to spawn and then wait upon?

Suppose my code looked like this:

pid_t status[ LARGENUMBER ];
status[0] = fork();
if( status[0] == 0 )
{
    // I am the child
    exec("./newCode01.c");
}
status[1] = fork();
if( status[1] == 0 )
{
    // child
    exec("./newCode02.c");
}
...etc...
wait(status[0]);
wait(status[1]);
...and so on....

Obviously, the larger LARGENUMBER is, the greater the chance that the parent is still fork() ing while children are segfaulting or becoming zombies or whatever.

So this implementation seems problematic to me. As I understand it, the parent can only wait() for one child at a time? What if LARGENUMBER is huge, and the time gap between running status[0] = fork(); and wait(status[0]); is substantial? What if the child has run, becomes a zombie, and been terminated by the OS somehow in that time? Will the parent then wait(status[0]) forever?

In the above example, there must be some standard or guideline to how big LARGENUMBER can be. Or is my approach all wrong?

#define LARGENUMBER 1
#define LARGENUMBER 10
#define LARGENUMBER 100
#define LARGENUMBER 1000
#define LARGENUMBER ???

I want to play with this, but my instinct is to ask for advice before I invest the development time into a program which may or may not turn out to be infeasible. Any advice/experience is appreciated.

It really depends on countless factors - the OS limitation, the machine architecture, speed, memory, other processes running. The work these processes do and so on. BTW, you can't `exec` `c` files... — Eugene Sh., Feb 24 '17 at 18:05
This is actually not a C but an OS question. `fork()` and `wait()` call system functions with probably less API overhead. I remember that I once wrote a wrong shell script which called accidentally itself. I needed less than a second to fork so much processes that the OS (IRIX) denied any further creation of process. — Scheff's Cat, Feb 24 '17 at 18:05
@Scheff Hmm, yeah, that's a fair comment. Every program I've written has largely done its computation within the process, with little OS I/O. Processes are a new frontier for me. — Pete, Feb 24 '17 at 18:15
@Pete I guess a waiting process does not cost much OS load. The OS scheduler should suspend it until a dieing child process raises the check whether it may be awaked again. — Scheff's Cat, Feb 24 '17 at 18:24
Aside from the technical fixes regarding doing `wait` to prevent zombies, you can spawn a few hundred or a thousand, but the system will saturate and slow to a crawl with a smaller number in practical terms. So, how many processes to keep "in flight" should be a tuning option that you set, based on benchmarking. (e.g. 100 processes might perform more poorly than 4 or 8) — Craig Estey, Feb 24 '17 at 19:11
Executing `.c` files is a novelty — it generally won't work. — Jonathan Leffler, Feb 24 '17 at 19:48
@JonathanLeffler you can execv complied c files, which I think OP wanted to write ? — Tony Tannous, Feb 24 '17 at 22:47
@EugeneSh. True, but forking countless processes will exhaust the system resources by one process and that is an unwanted scenario I guess. I remember I read something good about this I will try to find and edit this comment, — Tony Tannous, Feb 24 '17 at 22:48
Also see [`getrlimit(RLIMIT_NPROC, ...)`](http://stackoverflow.com/q/32204824/608639) and Android's setuid/pid exhaustion bug, a.k.a [Rage Against The Cage](https://thesnkchrmr.wordpress.com/2011/03/24/rageagainstthecage/). — jww, Feb 25 '17 at 01:30

score 2 · Answer 1 · edited May 23 '17 at 12:17

If you read the documentation of wait, you would know that

If status information is available prior to the call to wait(), return will be immediate.

That means, if the child has already terminated, wait() will return immediately. The OS will not remove the information from the process table until you have called wait¹ for the child process or your program exits:

If a parent process terminates without waiting for all of its child processes to terminate, the remaining child processes will be assigned a new parent process ID corresponding to an implementation-dependent system process.

Of course you still can't spawn an unlimited amount of children, for more detail on that see Maximum number of children processes on Linux (as far as Linux is concerned, other OS will impose other limits).

¹: https://en.wikipedia.org/wiki/Zombie_process

Tony Tannous · Accepted Answer · 2017-02-28T23:20:23.413

I will try my best to explain.

First a bad example: where you fork() one child process, then wait for it to finish before forking another child process. This kills the multiprocessing degree, bad CPU utilization.

pid = fork();
if (pid == -1) { ... } // handle error
else if (pid == 0) {execv(...);} // child
else (pid > 0) {
    wait(NULL);  // parent
    pid = fork();
    if (pid == -1) { ... } // handle error
    else if (pid == 0) {execv(...);} // child
    else (pid > 0) {wait(NULL); } // parent
}

How should it be done ? In this approach, you first create the two child process, then wait. Increase CPU utilization and multiprocessing degree.

pid1 = fork();
if (pid1 == -1) { ... } // handle error
if (pid1 == 0) {execv(...);}
pid2 = fork();
if (pid2 == -1) { ... } // handle error
if (pid2 == 0) {execv(...);}
if (pid1 > 0) {wait(NULL); }
if (pid2 > 0) {wait(NULL); }

NOTE:
even though it seems as parent is waiting before the second wait is executed, the child is still running and is not waiting to execv or being spawned.

In your case, you are doing the second approach, first fork all processes and save return value of fork then wait.

the parent can only wait() for one child at a time?

The parent can wait for all its children one at a time!, whether they already finished and became zombie process or still running. For more explained details look here.

How many child processes can a parent spawn before becoming infeasible?

It might be OS dependent, but one acceptable approach is to split the time given to a process to run in 2, half for child process and half for parent process. So that processes don't exhaust the system and cheat by creating child processes which will run more than the OS wanted to give the parent process in first place.

How many child processes can a parent spawn before becoming infeasible?

2 Answers2