Understanding the Python 'with' statement

Question

I'm trying to understand if there is there a difference between these, and what that difference might be.

Option One:

file_obj = open('test.txt', 'r')

with file_obj as in_file:
    print in_file.readlines()

Option Two:

with open('test.txt', 'r') as in_file:
    print in_file.readlines()

I understand that with Option One, the file_obj is in a closed state after the with block.

Answer 1

I don't know why no one has mentioned this yet, because it's fundamental to the way with works. As with many language features in Python, with behind the scenes calls special methods, which are already defined for built-in Python objects and can be overridden by user-defined classes. In with's particular case (and context managers more generally), the methods are __enter__ and __exit__.

Remember that in Python everything is an object -- even literals. This is why you can do things like 'hello'[0]. Thus, it does not matter whether you use the file object directly as returned by open:

with open('filename.txt') as infile:
    for line in infile:
        print(line)

or store it first with a different name (for example to break up a long line):

the_file = open('filename' + some_var + '.txt')
with the_file as infile:
    for line in infile:
        print(line)

Because the end result is that the_file, infile, and the return value of open all point to the same object, and that's what with is calling the __enter__ and __exit__ methods on. The built-in file object's __exit__ method is what closes the file.

Answer 2

These behave identically. As a general rule, the meaning of Python code is not changed by assigning an expression to a variable in the same scope.

This is the same reason that these are identical:

f = open("myfile.txt")

vs

filename = "myfile.txt"
f = open(filename)

Regardless of whether you add an alias, the meaning of the code stays the same. The context manager has a deeper meaning than passing an argument to a function, but the principle is the same: the context manager magic is applied to the same object, and the file gets closed in both cases.

The only reason to choose one over the other is if you feel it helps code clarity or style.

Answer 3

There is no difference between the two - either way the file is closed when you exit the with block.

The second example you give is the typical way the files are used in Python 2.6 and newer (when the with syntax was added).

You can verify that the first example also works in a REPL session like this:

>>> file_obj = open('test.txt', 'r')
>>> file_obj.closed
False
>>> with file_obj as in_file:
...     print in_file.readlines()
<Output>
>>> file_obj.closed
True

So after the with blocks exits, the file is closed.

Normally the second example is how you would do this sort of thing, though.

There's no reason to create that extra variable file_obj... anything that you might want to do with it after the end of the with block you could just use in_file for, because it's still in scope.

>>> in_file
<closed file 'test.txt', mode 'r' at 0x03DC5020>

Answer 4

If you just fire up Python and use either of those options, the net effect is the same if the base instance of Python's file object is not changed. (In Option One, the file is only closed when file_obj goes out of scope vs at the end of the block in Option Two as you have already observed.)

There can be differences with use cases with a context manager however. Since file is an object, you can modify it or subclass it.

You can also open a file by just calling file(file_name) showing that file acts like other objects (but no one opens files that way in Python unless it is with with):

>>> f=open('a.txt')
>>> f
<open file 'a.txt', mode 'r' at 0x1064b5ae0>
>>> f.close()

>>> f=file('a.txt')
>>> f
<open file 'a.txt', mode 'r' at 0x1064b5b70>
>>> f.close()

---

More generally, the opening and closing of some resource called the_thing (commonly a file, but can be anything) you follow these steps:

set up the_thing                       # resource specific, open, or call the obj
try                                    # generically __enter__
    yield pieces from the_thing
except
    react if the_thing is broken 
finally, put the_thing away            # generically __exit__

You can more easily change the flow of those subelements using the context manager vs procedural code woven between open and the other elements of the code.

Since Python 2.5, file objects have __enter__ and __exit__ methods:

>>> f=open('a.txt')
>>> f.__enter__
<built-in method __enter__ of file object at 0x10f836780>
>>> f.__exit__
<built-in method __exit__ of file object at 0x10f836780>

The default Python file object uses those methods in this fashion:

__init__(...)            # performs initialization desired

__enter__() -> self      # in the case of `file()` return an open file handle

__exit__(*excinfo) -> None.  # in the case of `file()` closes the file.

These methods can be changed for your own use to modify how a resource is treated when it is opened or closed. A context manager makes it really easy to modify what happens when you open or close a file.

Trivial example:

class Myopen(object):
    def __init__(self, fn, opening='', closing='', mode='r', buffering=-1):
        # set up the_thing

        if opening:
            print(opening)
        self.closing=closing    
        self.f=open(fn, mode, buffering)

    def __enter__(self):
        # set up the_thing
        # could lock the resource here
        return self.f

    def __exit__(self, exc_type, exc_value, traceback):
        # put the_thing away
        # unlock, or whatever context applicable put away the_thing requires
        self.f.close()
        if self.closing:
            print(self.closing)  

Now try that:

>>> with Myopen('a.txt', opening='Hello', closing='Good Night') as f:
...     print f.read()
...
Hello
[contents of the file 'a.txt']
Good Night

Once you have control of entry and exit to a resource, there are many use cases:

1. Lock a resource to access it and use it; unlock when you are done
2. Make a quirky resource (like a memory file, database or web page) act more like a straight file resource
3. Open a database and rollback if there is an exception but commit all writes if there are no errors
4. Temporarily change the context of a floating point calculation
5. Time a piece of code
6. Change the exceptions that you raise by returning True or False from the __exit__ method.

You can read more examples in PEP 343.

Answer 5

Is remarkable that with works even if return or sys.exit() is called inside (that means __exit__ is called anyway):

#!/usr/bin/env python
import sys

class MyClass:
  def __enter__(self):
    print("Enter") 
    return self

  def __exit__(self, type, value, trace):
    print("type: {} | value: {} | trace: {}".format(type,value,trace))

# main code:
def myfunc(msg):
  with MyClass() as sample:
    print(msg)
    # also works if uncomment this:
    # sys.exit(0) 
    return

myfunc("Hello")

return version will show:

Enter
Hello
type: None | value: None | trace: None

exit(0) version will show:

Enter
Hello
type: <class 'SystemExit'> | value: 0 | trace: <traceback object at 0x7faca83a7e00>