5

I am getting this error inside a unittest while using the multiprocessing package.
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Running the following code works inside of a regular python file but when put into a unittest, it raises the pickling error as I posted above. This is the simplest example I could spin up. 

    # The following, works. 
    import multiprocessing

    def hello(number):
        print "hello"

    number_processes = 2
    pool = multiprocessing.Pool(number_processes)
    total_tasks = 2
    tasks = range(total_tasks)
    results = pool.map(hello, tasks)
    pool.close()
    pool.join()

Next code block:

     # This does not work, i'm running it via unittest runner
    import multiprocessing
    import unittest
    class Testing123(unittest.TestCase):
        test_1(self):

            def hello(number):
                print "hello"

            number_processes = 2
            pool = multiprocessing.Pool(number_processes)
            total_tasks = 2
            tasks = range(total_tasks)
            results = pool.map(hello, tasks)
            pool.close()
            pool.join()
Will H
  • 51
  • 3
  • 1
    See the answer [here](https://stackoverflow.com/a/8805244/4799172). Basically (frustratingly), multiprocessing can only pickle functions that are at the top level of a module. I have read an answer from here from Tim Peters saying that it would be extremely difficult for them to implement some kind of fix to do what you want to do. – roganjosh Jul 20 '17 at 19:17
  • 1
    Appreciate it, I am working around this now by not using a python closure - just having the function as the first one in the py file. – Will H Jul 20 '17 at 19:28

0 Answers0