4

I have a function that yields from a context manager:

def producer(pathname):
    with open(pathname) as f:
        while True:
            chunk = f.read(4)
            if not chunk:
               break
            yield chunk

It is not a problem when the generator is entirely consumed since, during the last iteration, the generator resumes execution after the yield statement, and the loop breaks and we nicely exit the context manager.

However, if the generator is only partially consumed, and there are no more consumers to consume it entirely, will the generator remain suspended forever? In that case, we will never exit from the context manager. Would that mean the file will remain open for the rest of the program execution? Or at least until the generator is garbage collected? Is this a corner case I should take care of by myself, or can I rely on the Python runtime to close dangling context manager in time?

FWIW, I've seen Generator and context manager at the same time and How to use a python context manager inside a generator but I don't think they really answer the same question. Unless I missed something?
Sylvain Leroux
  • 50,096
  • 7
  • 103
  • 125
  • *Is this an issue?* - it's "for granted": in that case a context manager comes under generator control, thus obeying the generator's "suspend" nature. – RomanPerekhrest Nov 21 '19 at 12:29
  • Thanks for the comment @Roman. _Is this an issue?_ I think I was not as clear as I would have hoped it. I was not questioning the fact the context manager is suspended. But the fact it may be suspended _forever_ if there are no more consumers using it. – Sylvain Leroux Nov 21 '19 at 12:55

1 Answers1

5

If you fail to consume the whole generator, the context manager won't be cleaned until the generator is garbage collected, which may take quite a while if reference cycles are involved, or you're running on a non-CPython interpreter.

You can work around this by close-ing the generator-iterator; all generator functions provide a close method on the resulting generator-iterator that raises GeneratorExit inside it; the exception bubbles out of with statements and the like to make sure they're properly cleaned deterministically.

To make it occur at a guaranteed point in time, you can use contextlib.closing to get guaranteed closing of the generator itself:

 from contextlib import closing

 with closing(producer(mypath)) as produced_items:
     for item in produced_items:
         # Do stuff, maybe break loop early

Even if you break, return, or raise an exception, the with controlling produced_items will close it, which will in turn invoke cleanup for the with statements within it.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271