Multiprocessing in python compatible from 3 to 2.7

Question

I have the following code runnning smoothly in Python 3, and I can't convert it to Python 2.7.

from multiprocessing import *

def func(path, filename, path_2):
    #Things to do

for i in range(0,len(chr_names)): #len(chr_names) = 24
    tuple_var.append((path, chr_names[i][0], chrom_sizes[i][0]))

cores = 4
with Pool(cores) as p:
    p.starmap(func, tuple_var)

I get the following error.

python AttributeError: __exit__

I know starmap is not supported in Python 2.7.

What code should I use in Python 2.7?

So, are you asking how to find a suitable replacement for Pool.starmap? Can you just refactor `func`? — juanpa.arrivillaga, Nov 06 '18 at 17:54
Yes and if possible an example on how to pass multiple variables in func. — D Grigor, Nov 06 '18 at 17:56
Possible duplicate of [python 2.7 multiprocessing parallelization for and arguments](https://stackoverflow.com/questions/52651506/python-2-7-multiprocessing-parallelization-for-and-arguments) — Darkonaut, Nov 06 '18 at 23:13

score 0 · Answer 1 · answered Nov 06 '18 at 18:00

0

One simple approach, use a wrapper function:

def star_wrapper(args):
    return func(*args)

....

with Pool(cores) as p:
    p.map(star_wrapper, tuple_var)

answered Nov 06 '18 at 18:00

juanpa.arrivillaga

88,713
10
131
172

Thanks worked by using wrapper and with @blhsing answer replacing 'with Pool(cores) as p:' with 'p = Pool(cores)' – D Grigor Nov 06 '18 at 18:10

score 0 · Answer 2 · answered Nov 06 '18 at 18:00

Unless I'm misunderstanding you, it seems like you can use Pool's map function in Python 2.6+. All you need is a function that can apply tuple arguments to the original function. Something like:

def pool_starmap(pool, fn, items):
    def map_fn(args):
        fn(*args)
    return pool.map(map_fn, items)

cores = 4
with Pool(cores) as p:
    pool_starmap(p, func, tuple_var)

score 0 · Answer 3 · answered Nov 06 '18 at 18:01

The other answers have already covered how you can port starmap, but as for the AttributeError: __exit__ error, it comes from the fact that multiprocessing.Pool cannot be used as a context manager in Python 2.7, so you would simply have to do the following instead:

p = Pool(cores)

score 0 · Answer 4 · answered Nov 06 '18 at 18:03

First:

In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers

Have a look at this post for more info: Python Multiprocessing Lib Error (AttributeError: __exit__)

Second:

Use a helper function

Or choose one of the other options presented here: Multiprocessing with multiple arguments to function in Python 2.7

Sample code:

from contextlib import contextmanager
from multiprocessing import *

@contextmanager
def terminating(thing):
    try:
        yield thing
    finally:
        thing.terminate()

def func(path, filename, path_2):
    # Things to do
    print(path)
    print(filename)
    print(path_2)
    print('-----------------------------\n')

def helper(args):
    return func(args[0], args[1], args[2])

def task():
    tuple_var = []
    for i in range(0, 10):
        tuple_var.append(('hi_' + str(i), i, i))

    with terminating(Pool(processes=2)) as p:
        p.map(helper, tuple_var)

if __name__ == '__main__':
    task()

Multiprocessing in python compatible from 3 to 2.7

4 Answers4