with keyword works when openning file but not when calling a function

Question

I'm experimenting with multiprocessing module and copying example codes from this page. Here is one example:

#!/usr/bin/python
from multiprocessing import Pool
from time import sleep

def f(x):
    return x*x

if __name__ == '__main__':
    # start 4 worker processes
    with Pool(processes=4) as pool:

        # print "[0, 1, 4,..., 81]"
        print(pool.map(f, range(10)))

        # print same numbers in arbitrary order
        for i in pool.imap_unordered(f, range(10)):
            print(i)

        # evaluate "f(10)" asynchronously
        res = pool.apply_async(f, [10])
        print(res.get(timeout=1))             # prints "100"

        # make worker sleep for 10 secs
        res = pool.apply_async(sleep, [10])
        print(res.get(timeout=1))             # raises multiprocessing.TimeoutError

    # exiting the 'with'-block has stopped the pool

After running this code I get:

Traceback (most recent call last):
  File "example01.py", line 11, in <module>
    with Pool(processes=4) as pool:
AttributeError: __exit__

Somehow I've find that this is due to with keyword. However this code is also using with and it is running:

#!/usr/bin/python
with open("input.csv", "wb") as filePath:
    pass
filePath.close()

When I want to run mentioned example I have to modify it following way:

#!/usr/bin/python

from multiprocessing import Pool
from time import sleep
import traceback

def f(x):
    return x*x

if __name__ == '__main__':
  # start 4 worker processes
  # with Pool(processes=4) as pool:
  try:
    pool = Pool(processes = 4)

    # print "[0, 1, 4,..., 81]"
    print(pool.map(f, range(10)))

    # print same numbers in arbitrary order
    for i in pool.imap_unordered(f, range(10)):
      print(i)

    # evaluate "f(10)" asynchronously
    res = pool.apply_async(f, [10])
    print(res.get(timeout=1))             # prints "100"

    # make worker sleep for 10 secs
    res = pool.apply_async(sleep, [10])
    print(res.get(timeout=1))             # raises multiprocessing.TimeoutError

    # exiting the 'with'-block has stopped the pool

  # http://stackoverflow.com/questions/4990718/python-about-catching-any-exception
  # http://stackoverflow.com/questions/1483429/how-to-print-an-error-in-python
  # http://stackoverflow.com/questions/1369526/what-is-the-python-keyword-with-used-for
  except Exception as e:
    print "Exception happened:"
    print type(e)
    print str(e)
    print traceback.print_exc()

The output then looks like this:

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
0
1
4
9
16
25
36
49
64
81
100
Exception happened:
<class 'multiprocessing.TimeoutError'>

Traceback (most recent call last):
  File "example01_mod.py", line 29, in <module>
    print(res.get(timeout=1))             # raises multiprocessing.TimeoutError
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 563, in get
    raise TimeoutError
TimeoutError
None

Why I get an error when using with keyword, are those codes (with and try-catch) equivalents? I'm using python 2.7.10.

Answer 1

This PEP describes the context manager interface which is needed for using the 'with' keyword. Python 2.7's version of the Pool class does not support this interface so you cannot use the 'with' keyword in the way you described.

You can just rewrite the code to not use with and join/terminate the Pool directly as in the example here or you can upgrade to Python 3 which does support 'with' for Pools.

Answer 2

You have wrongly assumed that the multiprocessor Pool class of Python 2.7 returns a context manager. Instead it returns a List of worker processes. The Pool was converted to a context manager object from Python 3.4.3 and if your version predates that, you cannot use it with a context manager.

To use with the with statement, the expression should return an object with enter and exit methods so you get the familiar error.

AttributeError: __exit_