Is it still unsafe to process files without using with in python 3?

Question

For the sake of a small example. Let's say that I want to read each line from this file into a list:

First line
Second line
Third line
Fourth line

and it's called example.txt and in my cwd.

I've always been told that I should do it using a with statement like these:

# method 1

lines = []
with open("example.txt") as handle:
    for line in handle:
        lines.append(line.strip("\n"))

# method 2

with open("example.txt") as handle:
    lines = [line.strip("\n") for line in handle]

There's a lot of info out there on this.. I found some good stuff in What is the python keyword "with" used for?. So it seems this is/was recommended so that the file was properly closed, especially in the cases of exceptions being thrown during processing.

# method 3

handle = open("example.txt")
lines = [line.strip("\n") for line in handle]
handle.close()

# method 4

lines = [line.strip("\n") for line in open("example.txt")]

For that reason the above two methods would be frowned upon. But all of the information I could find on the subject is extremely old, most of it being aimed at python 2.x. sometimes using with instead of these methods within more complex code is just impractical.

I know that in python3 at some point variables within a list comprehension were limited to their own scope (so that the variables would not leak out of the comprehension).. could that take care of the issue in method 4? Would I still experience issues when using method 3 if an exception was thrown before I explicity called the .close() statement?

Is the insistence on always using with to open file objects just old habits dying hard?

In my humble opinion you don't need to use `with` statement as long as you make sure that you always eventualy `.close()` handle. — Toni Sredanović, Sep 18 '19 at 12:27
But how does that affect method 4, where there is no variable to call `.close()` on? — Zhenhir, Sep 18 '19 at 12:29
Failure to close a file that was open *for reading only* is not that big of a deal - you'd eventually hit the operating system's open file limit if you did this enough times, but that would require that you were somehow keeping references to *all* of the file objects. It's more of a concern when writing - failure to close means that an arbitrary amount of data may still be in buffers rather than actually written to disk. The sooner you close it (and `with` is by far the most convenient way of ensuring that), the sooner the file is complete and usable for its intended purpose. — jasonharper, Sep 18 '19 at 12:37
@Zhenhir: in practice, using CPython, the file will be closed immediately. That's because CPython uses reference counting to track object lifetimes and so the file is closed. But this doesn't translate to other Python implementations. — Martijn Pieters, Sep 18 '19 at 14:37
That list comprehensions are executed in a new function object has nothing to do with this. Leaving files open for a bit longer is not 'dangerous' either, it can at best lead to your program running out of file handles if you do this a lot during the lifetime of your program. — Martijn Pieters, Sep 18 '19 at 14:39

Šimon Kocúrek · Answer 1 · 2019-09-18T12:50:40.300

There is also a 5th way:

handle = None

try:
    handle = open("example.txt")
except:
    pass
finally:
    if handle is not None:
        handle.close()

Note, this would not work, in case open would throw an exception, handle.close() would crash the program.:

try:
    handle = open("example.txt")
except:
    pass
finally:
    handle.close()

Yes. Closing files is still needed. This concept is present in all programming languages and is caused by buffering, which improves performance of working with IO. To avoid the need of closing file handles you would need to find a way of reading files without buffering.

score 2 · Answer 2 · answered Sep 18 '19 at 12:43

The with keyword is a context manager. It is not required to work with a file. However, it is good practice.

It is good practice to use the with keyword when dealing with file objects. The advantage is that the file is properly closed after its suite finishes, even if an exception is raised at some point. Using with is also much shorter than writing equivalent try-finally blocks... Python Tutorial Section 7.2

Using it is syntactic sugar for a try/except block similar to:

f = open('example.txt', 'w')
try:
    f.write('hello, world')
finally:
    f.close()

You could explicitly write that block out whenever you want to work with a file. In my opinion it is more pythonic, and easier to read, to use the context manager syntax.

with open('example.txt', 'w') as f:
    f.write('hello, world')

If you would like to learn a little more about context managers check out this blog post and of course the Python Documentation on contextlib.

Is it still unsafe to process files without using with in python 3?

2 Answers2