Multiprocessing with Python 2.7 throwing attribute error

Question

from itertools import product
from multiprocessing import Pool

with Pool(4) as pool:
            pool.map(lambda x: run_test_function(x, arg2, arg3, arg4), arg1)
    

I am getting below error after executing above code. There some other code as well which I can't write here. But actual problem is coming from this piece of code only.

Traceback (most recent call last):
  File "modProfileChange_test.py", line 347, in <module>
    main(sys.argv[1:])
  File "modProfileChange_test.py", line 336, in main
    test_run_code(arg1, arg2, arg3, arg4, arg5, arg6)
  File "modProfileChange_test.py", line 23, in test_run_code
    with Pool(4) as pool:
AttributeError: __exit__

Answer 1

1. In Python 2.7, multiprocessing.Pool is not a context manager and thus it can't be used in a with statement
   1. Solution - create a pool using regular assignment to a variable:

      my_pool = Pool(4)
      my_pool.map(...)
      

2. lambda functions don't work with multiprocessing.Pool, even in Python 3.
   1. Solution - emulate a closure using a solution in the link above:

      from functors import partial
      
      def run_test_function(x, fun_arg2, fun_arg3, fun_arg4):
          # your code here
          ...
      
      process_func = partial(run_test_function, fun_arg2=arg2, fun_arg3=arg3, fun_arg4=arg4)
      

Putting this together:

from multiprocessing import Pool
from functools import partial

def run_test_function(x, fun_arg2, fun_arg3, fun_arg4):
    # this is an example
    print x, fun_arg2, fun_arg3, fun_arg4

if __name__ == "__main__":
    arg1 = 1,2,3,4
    arg2 = "hello"
    arg3 = "world"
    arg4 = "!"

    process_func = partial(run_test_function, fun_arg2=arg2, fun_arg3=arg3, fun_arg4=arg4)

    my_pool = Pool(4)
    my_pool.map(process_func, arg1)

Output:

~/test $ python2.7 so10.py
1 hello world !
2 hello world !
3 hello world !
4 hello world !