0

I have the following code in a Jupyter notebook:

import resource
resource.setrlimit(resource.RLIMIT_NPROC, (1024, 1024))
while True:
    try:
        pid = os.fork()
        if pid != 0:
            count += 1
        else: 
            break
    except Exception as e:
        print(e)
        print("COUNT: ", count)

From what I understand, this should keep forking until it hits 1024 and then an exception is caught and the number of created processes should be 1024.

When I ran this, the first time I got 896, but every time after, even the first fork wouldn't run.
I'm guessing the 896 is because of some Jupyter overhead, but why would the number of processes persist after an execution when I specifically reconnect so that popen is called anew?

What contributes to the nproc limit? Is it all child processes and forks? If so, is there any way I can get a list of these somehow?

I've tried reading children from /proc/self/task, but the number of those is constant.

Also, is setting RLIMIT_NPROC equivalent to running ulimit -u in the Jupyter popen command? Is this equivalent to setting RLIMIT_NPROC in the preexec_fn?

1419636215
  • 275
  • 2
  • 13
  • It is per user, so it counts all processes running as that user including shells, daemons and commands. parents or children. – eckes Dec 08 '17 at 17:30
  • Ah, so what if you wanted to set a limit on the number of subprocesses a single process can create? – 1419636215 Dec 08 '17 at 17:33
  • Not sure if that is possible, maybe by setting up its own pid namespace. – eckes Dec 08 '17 at 17:34
  • Systemd is doing that BTW it generates a cgroup and configures `pids.max` attribute based on `TasksMax= `setting for the unit. Might be a good idea to start the notebook via systemd. – eckes Dec 08 '17 at 17:39
  • I think that would work. https://stackoverflow.com/questions/1770209 Thanks – 1419636215 Dec 08 '17 at 17:39
  • Hmm. Hadn't considered systemd. Would probably break a lot of stuff to change that part of jupyter while I could just insert a cell at the top to change the user to a temp one that has a ulimit set for it – 1419636215 Dec 08 '17 at 17:41

0 Answers0