2

Question:

I am interested in doing a list comprehension inside a Python with statement, so that I can open multiple context managers at the same time with minimal syntax. I am looking for answers that work with Python 2.7.

Consider the following code example. I want to use the with statement on variables in an arbitrarily-long list at the same time, preferably in a syntactically-clean fashion.

def do_something(*args):
    contexts = {}
    with [open(arg) as contexts[str(i)] for i, arg in enumerate(args)]:
        do_another_thing(contexts)

do_something("file1.txt", "file2.txt")

Does anybody know if there is a way to involve a list comprehension inside of a with statement in Python 2.7?

Answers to similar questions:

Here are some things I've already looked at, with an explanation of why they do not suit my purposes:

For Python 2.6-, I could use contextlib.nested to accomplish this a bit like:

def do_something(*args):
    contexts = {}
    with nested(*[open(arg) for arg in args]) as [contexts[str(i)] for i in range(len(args))]:
        do_another_thing(contexts)

However, this is deprecated in Python 2.7+, so I am assuming it is bad practice to use.

Instead, the new syntax was given on this SO answer, as well as this SO answer:

with A() as a, B() as b, C() as c:
    doSomething(a,b,c)

However, I need to be able to deal with an arbitrary input list, as in the example I gave above. This is why I favour list comprehension.

For Python 3.3+, this SO answer described how this could be accomplished by using ExitStack. However, I am working in Python 2.7.

There is also this solution, but I would prefer to not write my own class to accomplish this.

Is there any hope of combining a list comprehension and a with statement in Python 2.7?

Update 1-3: Updated example to better emphasize the functionality I am looking for

Update 4: Found another similar question. This one has an answer which also suggests ExitStack, a function that is not available in 2.7.

Community
  • 1
  • 1
Miles
  • 780
  • 7
  • 19
  • 2
    This has all the feel of a "*I have `ship1 = 1; ship2 = 2` code and now I want an unlimited number of ships, how do I create ship variables with new numbers in code??? Is this even possable?*" nonsensical question. Loop over the with statement? `for item in list: with open(item) as f: f.read()` ... – TessellatingHeckler Aug 08 '16 at 17:17
  • I need to be able to have all of the context managers open at the same time though, i.e., `with open(item1) as f1: with open(item2) as f2: #do something with both`. I also have an arbitrary number of items. – Miles Aug 08 '16 at 17:24
  • Updated my example to show how the context managers need be open at the same time, not just in sequence. – Miles Aug 08 '16 at 17:30

2 Answers2

2

The major tasks of with statement is invoking the __exit__ attribute of its context manager. Specially for dealing with files. So In this case due to this point and the fact that open() returns an file_object you can dimple use a list comprehension in order to create the list of your file objects and just call the exit()/close() manually. But be aware that in this case you have to handle the exceptions manually. 

def print_files(*args):
    f_objs = [open(arg) for arg in args]
    # do what you want with f_objs
    # ...
    # at the end
    for obj in f_objs:
        f.close()

Note that if you only want to run some parallel operations on your file objects, I recommend this approach, other wise the best way is using a with statement inside a for loop, and opennning the files in each iteration (on names) like following:

for arg in args:
    with open(arg) as f:
         # Do something with f

For more safety you can use a custom open function in order to handle the exceptions too:

def my_open(*args, **kwargs):
    try:
        file_obj = open(*args, **kwargs)
    except Exception as exp:
        # do something with exp and return a proper object
    else:
        return file_obj

def print_files(*args):
    f_objs = [my_open(arg) for arg in args]
    # do what you want with f_objs
    # ...
    # at the end
    for obj in f_objs:
        try:
            f.close()
        except Exception as exp:
            # handle the exception
Mazdak
  • 105,000
  • 18
  • 159
  • 188
  • This doesn't have the exception safety benefits of `with` or `ExitStack`, especially if opening one of the files fails. – user2357112 Aug 08 '16 at 17:32
  • @user2357112 Indeed, also exceptions during `exit()`. – Mazdak Aug 08 '16 at 17:36
  • Thanks for your answer @Kasramvd. Unfortunately, as @user2357112 pointed out, your example does not have the safety benefits of `with`, which is really what I am looking for. And yes, I am looking to do something in parallel with the objects. – Miles Aug 08 '16 at 17:41
  • @Miles Yes that was a good point. In that case I think you must do them manually like the exit. Maybe using a decorator, or another function, check out the edit. – Mazdak Aug 08 '16 at 17:47
  • You're not closing the files if an exception occurs between opening and closing; you need a try-finally for that. Also, one of the major advantages of an `ExitStack` over manual handling is that it lets you cleanly propagate exceptions that occur in setup or teardown. With manual handling, it's quite awkward to propagate them instead of silencing them. – user2357112 Aug 08 '16 at 19:22
  • @user2357112 I've let the OP to takes care of that, because it can be a lot of scenarios based on main code which I'm not aware of them. And regarding the `ExitStack` I totally agree with that. It's more comprehensive and powerful for propagating the exceptions. – Mazdak Aug 08 '16 at 21:36
2

Doing this yourself is really tricky, especially handling exceptions that occur while opening or closing the files. I'd recommend just getting a library like contextlib2 that implements the contextlib.ExitStack functionality. Then you can do

with contextlib2.ExitStack() as stack:
    files = [stack.enter_context(open(arg)) for arg in args]
    ...

just like you were using contextlib.ExitStack from Python 3, and everything is handled correctly for you.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • Thanks @user2357112 - this is definitely a good solution to my problem. However, I would like to use something inside Python 2.7 itself, without having to import external libraries. If such an answer does not exist, however, I will take yours! – Miles Aug 08 '16 at 17:47