Run many functions in parallel, and gather all results in a list

Question

I have a very CPU-intensive function:

def entity_intersections(ent, collidable):
    intersections = []

    for line1, line2 in product(ent.shape, collidable.shape):

        pair_intersections = find_intersections(line1 + ent.position, ent.velocity, ent.acceleration, line2 + collidable.position, collidable.velocity, collidable.acceleration, ent, collidable)
        intersections.extend(pair_intersections)

    return intersections

I want to make all of the calls to find_intersections run in parallel, so that they execute faster, while still gathering all of the results together (once all executions finish). What library would allow me to do this, given that find_intersections is a pure function?

An example of how to generate these parallel executions, as well as gathering together the results would be greatly appreciated.

What does your research suggest is an appropriate python library? — Marcin, Mar 22 '12 at 15:51
@Marcin For performance enhancement, there's a very large selection: one of several parallel processing libraries (including 2 python built-in libraries), restructuring into coroutines, Cython, etc. Parallel processing is not my strong point... — Casey Kuball, Mar 22 '12 at 21:33
Yes, there are a large number - why not read the documentation of some, and then ask a question when you can't decide between several good candidates. — Marcin, Mar 22 '12 at 21:35

Sven Marnach · Accepted Answer · 2012-03-22T21:42:56.247

8

The easiest way is to use the multiprocessing module:

class FindIntersectionsWrapper(object):
    def __init__(self, ent, collidable):
        self.ent = ent
        self.collidable = collidable
    def __call__(self, dims):
        line1, line2 = dims
        return find_intersections(
            line1 + self.ent.position, self.ent.velocity,
            self.ent.acceleration, line2 + self.collidable.position,
            self.collidable.velocity, self.collidable.acceleration, 
            self.ent, self.collidable)

def entity_intersections(ent, collidable):
    find_inter = FindIntersectionsWrapper(ent, collidable)
    pool = multiprocessing.Pool()
    return pool.map(find_inter, product(ent.shape, collidable.shape))

The helper function find_intersections_wrapper() is necessary since Pool.map() expects a function with a single argument.

You might want to move the creation of pool out of entity_intersections() to have the overhead of generating the process pool only once.

Edit: Used a class instead of a closure since the callable passed to Pool.map() must be picklable on Windows.

edited Mar 22 '12 at 21:42

answered Mar 22 '12 at 16:04

Sven Marnach

574,206
118
941
841

`pool.map` wasn't happy with `wrapper` being defined inside find_intersections, so I generated the list of arguments for `find_intersections` inside `entity_intersections`, and mapped them that way. Thanks! :) – Casey Kuball Mar 22 '12 at 21:19
@Darthfett: Obviously I couldn't test this code easily. I'd like to understand what exactly the problem was. Are you on Windows? – Sven Marnach Mar 22 '12 at 21:22
Yes, I am on Windows. The issue is nearly the same as [here](http://stackoverflow.com/q/3288595/936083). – Casey Kuball Mar 22 '12 at 21:26
1

@Darthfett: Thanks. On Linux, this can be resolved by instantiating all necessary closures before creating the process pool, but this won't help on Windows. (Windows is lacking `fork()`.) – Sven Marnach Mar 22 '12 at 21:35
1

On Windows, you could use an object instead of a closure -- I'll edit my post later because I don't have time right now. – Sven Marnach Mar 22 '12 at 21:38

score 0 · Answer 2 · answered Mar 22 '12 at 15:51

0

You should use the threading class

http://docs.python.org/library/threading.html#module-threading

This page have some good examples: http://www.tutorialspoint.com/python/python_multithreading.htm

answered Mar 22 '12 at 15:51

StefanE

7,578
10
48
75

1

Threading doesn't really allow increased CPU utilization with CPython because of the GIL. – agf Mar 22 '12 at 15:56
2

@agf: This is true as long as you are using CPython and don't spend most of the time inside functions releasing the GIL, like NumPy array operations. I think the more relevant objection here is that you concepttually don't want to use threads here since `find_intersections()` is a pure function. – Sven Marnach Mar 22 '12 at 16:00

Run many functions in parallel, and gather all results in a list

2 Answers2