Pathos Multiprocessing Pickling Issue

Question

I am developing an automation framework for some testing we are doing on my team. We have come across this Pickling error and after doing some research, I came across a number of of solutions and reviewed the accompanying github page:

Python multiprocessing pickling error
Multiprocessing: How to use Pool.map on a function defined in a class?
Pickling issue with python pathos <---From this link, I confirmed the code is referencing the correct library.
https://github.com/uqfoundation/pathos

I installed pathos and believe I have implemented it correctly but I am still getting the Pickling error. Here is the sample code and traceback:

import multiprocess

collect_host = 'test_host'
collect_name = 'test_collect'
username = 'user'

start_func = test_start_callback(perf_collect=collect_name,
                                 perf_user=username,
                                 perf_host=collect_host,
                                 password= password)

dpool = multiprocess.Pool(1)
args = ('', '')
worker_results = []
worker = dpool.apply_async(start_func, args)
worker_results.append(worker)
dpool.close()
dpool.join()

for result in worker_results:
        result.wait()
        r = result.get()
        assert r, 'No results returned'

The test_start_callback is a factory function to generate a function that takes no arguments. I edited out the code segment that isn’t used.

def test_start_callback(dim_collect_host=None, dim_collect_user=None, 
                        perf_host=None, perf_user=None, perf_collect=None, 
                        password=None, tpcc_filename=None):

    if perf_host and perf_user and perf_collect:
        def perf_test_start():
            from <internal> import PerfmonManager
            perf = PerfmonManager(host=perf_host,
                                  username=perf_user,
                                  password=password,
                                  collect_name=perf_collect)
            perf.start_collect()
    return perf_test_start

Traceback

What am I missing that wasn't mentioned in the other SO links or is the way I'm passing this function not supported by pathos either.

Mike McKerns · Answer 1 · 2017-10-11T15:31:07.053

I'm the pathos and multiprocess author. This is a recent known issue that I'm not sure of the origin of yet... it's apparently surfaced in the past month without a change in the code. I'm working actively to track down what the source of what the problem is. It seems to only affect python 2.7.

See: https://github.com/uqfoundation/pathos/issues/121

UPDATE: If this is the same issue as the above-referenced one, then all you need to do is clean out old builds (eggs, etc) of multiprocess and rebuild. That seems to do it. For more information see the above URL.

Pathos Multiprocessing Pickling Issue

1 Answers1