Multiprocessing Pool PicklingError

Question

I'm working on a class that can take in a list of Request type objects and execute the requests (using the object's .execute() API) and return the result of the request. I'm running into an issue when trying to make this multi-processed. I have used Pool in the past, and I am having a tough time getting it to work here. When attempting to run the staticmethod in the Executor class from a Pool, I receive a PicklingError. But when running directly from a new Process object, it seems to work as expected. What is the cause of this behavior? Is there something obvious that I am doing incorrectly here ? Running this on Python 2.6 btw

Thanks!

Executor class:

class Executor(object):
    def __init__(self, max_threads):
        self.max_threads = max_threads

    @staticmethod
    def execute_request_partition(requests):
        return [request.execute() for request in requests]

    def execute_requests(self, list_of_requests):
        partitions = split_list(list_of_requests, self.max_threads)
        # Seems to execute as expected
        process = Process(target=self.execute_request_partition, args=(partitions[0],))
        process.run()

        # PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
        p = Pool(1)
        p.apply_async(self.execute_request_partition, args=(partitions[0],))
        p.close()
        p.join()

Answer 1

This should explain the issue your having: Can't pickle static method - Multiprocessing - Python

On a side note: I wouldn't use multiprocessing if I'm going to be spending most of my time waiting for IO. Try threading or even coroutines (http://www.gevent.org/). You'll get the same performance and there's less fuss with pickling