Why do I get subprocess resource warnings despite the process being dead?

Question

I've been trying to figure out how to spin up different subprocess instances and then killing them and then creating new ones. The parent python process never does, it just kills the subprocesses. I followed a lot of links on SO but I keep getting the following message once the parent python process ends:

F/Users/Lucifer/miniconda3/envs/rltp/lib/python3.6/subprocess.py:761: ResourceWarning: subprocess 40909 is still running ResourceWarning, source=self)

it seems interesting because I did ps but I get nothing:

  PID TTY           TIME CMD
 7070 ttys001    0:00.06 /Applications/iTerm.app/Contents/MacOS/iTerm2 --server login -fp Lucifer
 7072 ttys001    0:00.61 -bash
17723 ttys002    0:00.06 /Applications/iTerm.app/Contents/MacOS/iTerm2 --server login -fp Lucifer
17725 ttys002    0:00.06 -bash
38586 ttys002    0:00.16 sertop --no_init

I simply want to start a process:

self.serapi = subprocess.Popen(['sertop','--no_init'],
    stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,
    preexec_fn=os.setsid,shell=True
    ,)

and kill it:

    os.killpg(os.getpgid(self.serapi.pid), signal.SIGTERM)

the above code is essentially copied from the top answer:

How to terminate a python subprocess launched with shell=True

but I am unsure why I get this message. Am I killing the child process successfully? I plan to start and kill many of them.

Note I don't know or need shell=True. I just copied that cuz thats how the answer/question I posted has it. I'd prefer to not have that parameter.

according to the answer I tried:

def kill(self):
    self.serapi.wait()
    #self.serapi.kill()
    self.serapi.terminate()
    #os.killpg(os.getpgid(self.serapi.pid), signal.SIGTERM)
    #self.serapi.wait()

and different permutations of the above but nothing really seemed to work. Any advice?

Did you `wait()` on the subprocess? Even after a child process exits, it remains a zombie until its parent calls `wait()` and retrieves its exit status. — Daniel Pryden, Mar 07 '19 at 20:15
@DanielPryden I did not call `wait()` I did not know that call existed until you mentioned it. — Charlie Parker, Mar 07 '19 at 22:01
Great! I figured as much, which is why I wrote the answer below. If you found it useful, please feel free to upvote and/or accept it. Glad I could help! — Daniel Pryden, Mar 07 '19 at 22:01
@DanielPryden looking at the docks calling `wait()` seems to deadlock if both are `stdout,stderr = PIPE`, besides my process never ends until I tell it to end. — Charlie Parker, Mar 07 '19 at 22:02

Daniel Pryden · Answer 1 · 2019-03-07T20:29:06.487

4

The ResourceWarning: subprocess N is still running warning comes from the __del__ method of the subprocess.Popen class.

If you look at the source for that method, you'll see this comment:

        # Not reading subprocess exit status creates a zombie process which
        # is only destroyed at the parent python process exit
        _warn("subprocess %s is still running" % self.pid,
              ResourceWarning, source=self)

The solution is to ensure you call wait() on the child process.

See the NOTES section of the man page for wait(2) for more background information.

In Python, the easiest way to handle this situation is to keep track of all the Popen objects you have created, and ensure that something calls wait() on them, either directly or indirectly.

Alternatively, you can install a SIGCHLD handler that ignores SIGCHLD events; then your child processes will disappear immediately, but you will now be unable to call wait() on them. See also How can I prevent zombie child processes? and How can I handle SIGCHLD?

edited Mar 07 '19 at 20:29

answered Mar 07 '19 at 20:23

Daniel Pryden

59,486
16
97
135

I tried different variations of the following `self.serapi.wait();#self.serapi.kill();#self.serapi.terminate();` but it didn't work. So I guess calling `wait()` and then terminating/killing the process doesn't work? Or am I missing something? – Charlie Parker Mar 07 '19 at 22:07
but is the process still running? I don't understand what is going on because when I do a `ps` I don't see it... :/ – Charlie Parker Mar 07 '19 at 22:15
2

You should kill the process first, and then wait on it. At the kernel level, `wait` doesn't literally mean "wait", it means "receive postmortem information". The process may in fact not be a zombie, depending on how you fork it. But the `Popen` class *thinks* the subprocess must still be running (or must be a zombie), because you haven't received any postmortem information yet. Does that make sense? – Daniel Pryden Mar 08 '19 at 03:08
1

Ok so I call `wait` (really badly named name for what its suppose to be for) after `kill`. How do I know if everything was successful? I am sure I tried this but it wasn't clear if it worked. What I am trying to do is running a new subprocess for each of my unit tests. I'd like to create a new subprocess. – Charlie Parker Mar 11 '19 at 00:24

Why do I get subprocess resource warnings despite the process being dead?

1 Answers1

Linked