8

Let's say I want to open a text file for reading using the following syntax:

with open(fname,'r') as f:
    # do something
    pass

But if I detect that it ends with .gz, I would call gzip.open(). 

if fname.endswith('.gz'):
    with gzip.open(fname,'rt') as f:
            # do something
            pass
else:
    with open(fname,'r') as f:
            # do something
            pass

If "do something" part is long and not convenient to write in a function (e.g. it would create a nested function, which cannot be serialized), what is the shortest way to call with either gzip.open or open based on the return of fname.endswith('.gz')?

cellepo
  • 4,001
  • 2
  • 38
  • 57
Cindy Almighty
  • 903
  • 8
  • 20
  • related question: https://stackoverflow.com/questions/49224375/open-statement-according-to-a-file-extension . but the answers here are slightly different therefore i think it should remain open and not be closed as a duplicate. – hiro protagonist Oct 23 '18 at 15:13

3 Answers3

11

The context manager helps close the object.

You don't have to create the object used as a context manager, at the same time you use with to enter the context, though. The open() and gzip.open() calls return a new object that happens to be a context manager, and you can create them before you enter the context:

if fname.endswith('.gz'):
    f = gzip.open(fname,'rt')
else:
    f = open(fname, 'r')

with f:
    # do something

In both cases, the object returns self on entering the context, so there is no need to use as f here.

Also, functions are first-class citizens, so you can also use a variable to store the function and then call that in the with statement to create the context manager and file object:

if fname.endswith('.gz'):
    opener = gzip.open
else:
    opener = open

with opener(fname, 'rt') as f:  # yes, both open and gzip.open support mode='rt'
    # do something

This doesn't really buy you anything over the other method here, but you could use a dictionary to map extensions to callables if you so desire.

The bottom line is that with calls context-manager hook methods, nothing less, nothing more. The expression after with is supposed to supply such a manager, but creating that object is not subject to the context management protocol.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Thanks! Can you just explain why "opener, mode = open"? How is it possible that "opener = open" wouldn't work? – Cindy Almighty Oct 23 '18 at 15:47
  • @CindyAlmighty: that was an editing error, sorry. I originally assigned both an opener and a mode (so `opener, mode = gzip.open, 'rt'`), but then realised that `open()` accepts `rt` to mean the same thing as `r` for the mode. – Martijn Pieters Oct 23 '18 at 15:48
9

You can bind either context manager to the same name and choose early:

if fname.endswith('.gz'):
    context = gzip.open(fname,'rt')
else:
    context = open(fname,'r')

with context as f:
    # do the same thing in either case

This allows for some nice patterns, for instance if the input is possibly an opened file handle then you could use contextlib.nullcontext to get a no-op in the with block for the given case.

Andras Deak -- Слава Україні
  • 33,737
  • 11
  • 83
  • 111
  • 2
    It's one of the beautiful things about Python because everything is an object you can get away with lots of neat tricks like this. Another favorite of mine is using `dict`s as a switch. – r.ook Oct 23 '18 at 15:17
  • 1. can you explain the last sentence please? :) 2. @Idlehands What is this use of dict as a switch? – Cindy Almighty Oct 23 '18 at 15:43
  • Here's an answer that utilizes the `dict` object as a switch: https://stackoverflow.com/questions/60208/replacements-for-switch-statement-in-python#60211 – r.ook Oct 23 '18 at 15:46
  • 1
    @CindyAlmighty it isn't directly applicable to your use case. But imagine the scenario when you have "_My `fname` is either a file path that needs to be opened or it has already been opened elsewhere by `open(the_actual_file_name)`_". In this case you could do, say `context = gzip.open(fname, 'rt') if isinstance(fname, str) else contextlib.nullcontext(fname)` and use a single `with context as f:` block which will work in both cases just fine. This is exactly the second example given [in the docs](https://docs.python.org/3/library/contextlib.html#contextlib.nullcontext), actually. – Andras Deak -- Слава Україні Oct 23 '18 at 15:55
  • 1
    @Idlehands. very true, and Python is usually pretty clear in its intent. but context managers _are_ a bit unusual in their implicit enter/exit triggering. It’s not all that obvious that the context creation can syntactically be separated from **as**, so this post is extra informative - despite using them for a while this is a new trick for me. – JL Peyret Oct 23 '18 at 20:33
2
with gzip.open(fname, 'rt') if fname.endswith('.gz') else open(fname, 'r') as f:
    # do something
    pass
Hkoof
  • 756
  • 5
  • 14