ResourceWarning: unclosed file <_io.BufferedReader name=4>

Question

Consider the following program:

import tempfile
import unittest
import subprocess

def my_fn(f):
    p = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f)
    yield p.stdout.readline()
    p.kill()
    p.wait()

def my_test():
    with tempfile.TemporaryFile() as f:
        l = list(my_fn(f))

class BuggyTestCase(unittest.TestCase):
    def test_my_fn(self):
        my_test()

my_test()
unittest.main()

Running it results in the following output:

a.py:13: ResourceWarning: unclosed file <_io.BufferedReader name=4>
  l = list(my_fn(f))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.
----------------------------------------------------------------------
Ran 1 test in 0.005s

OK

What is the actual cause of the warning and how to fix it? Note that if I comment out unittest.main() the problem disappears, which means that it's specific to subprocess+unittest+tempfile.

Answer 1

You should be closing the streams associated with the Popen() object you opened. For your example, that's the Popen.stdout stream, created because you instructed the Popen() object to create a pipe for the child process standard output. The easiest way to do this is by using the Popen() object as a context manager:

with subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f) as p:
    yield p.stdout.readline()
    p.kill()

I dropped the p.wait() call as Popen.__exit__() handles this for you, after having closed the handles.

If you want to further figure out exactly what cause the resource warning, then we can start by doing what the warning tells us, and enable the tracemalloc module by setting the PYTHONTRACEMALLOC environment variable:

$ PYTHONTRACEMALLOC=1 python a.py
a.py:13: ResourceWarning: unclosed file <_io.BufferedReader name=4>
  l = list(my_fn(f))
Object allocated at (most recent call last):
  File "/.../lib/python3.8/subprocess.py", lineno 844
    self.stdout = io.open(c2pread, 'rb', bufsize)
.
----------------------------------------------------------------------
Ran 1 test in 0.019s

OK

So the warning is thrown by a file opened by the subprocess module. I'm using Python 3.8.0 here, and so line 844 in the trace points to these lines in subprocess.py:

if c2pread != -1:
    self.stdout = io.open(c2pread, 'rb', bufsize)

c2pread is the file handle for one half of a os.pipe() pipe object created to handle communication from child process to Python parent process (created by the Popen._get_handles() utility method, because you set stdout=PIPE). io.open() is exactly the same thing as the built-in open() function. So this is where the BufferedIOReader instance is created, to act as a wrapper for a pipe to receive the output the child process produces.

You could also explicitly close p.stdout:

p = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f)
yield p.stdout.readline()
p.stdout.close()
p.kill()
p.wait()

or use p.stdout as a context manager:

p = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f)
with p.stdout:
    yield p.stdout.readline()
p.kill()
p.wait()

but it's easier to just always use subprocess.Popen() as a context manager, as it'll continue to work properly regardless of how many stdin, stdout or stderr pipes you created.

Note that most subprocess code examples don't do this, because they tend to use Popen.communicate(), which closes the file handles for you.

Answer 2

I also got some ResourceWarning: unclosed file <_io.FileIO name=7 mode='wb' closefd=True> messages when running unit tests. I couldn't use the Popen object as a context manager, so I did this in my classes __del__ method:

import subprocess

class MyClass:
    def __init__(self):
        self.child= subprocess.Popen(['dir'],
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE)

    def __del__(self):
        self.child.terminate()
        self.child.communicate()

Answer 3

You can just change my_fn to use with, then you won't need kill and wait. Python'll close it for you:

def my_fn(f):
    with subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f) as p:
        yield p.stdout.readline()