4

I have a class that is implemented in cython containing c-pointers which I'm trying to use together with python's multiprocessing module. The class takes a DLL-file to return an instance of the class.

The problem I have is that while the instances preserve their data type, they seem to be empty, i.e. I can access all their class functions but they've lost all their instance values I set before they entered. The code containing special_class is very big so I'm not able to include it.

import time
import multiprocessing as mp
from special_module import special_class

def run_task(tasks,nr):
    obj = tasks[nr]['data']
    print obj.get_name()



if __name__ == "__main__":

    m1 = special_class("a.dll")
    m2 = special_class("b.dll")


    tasks = dict()

    tasks[1] = {'data': m1}
    tasks[2] = {'data': m2}


    process1 = mp.Process(target = run_task, name = 'process1', args = (tasks, 1))
    process2 = mp.Process(target = run_task, name = 'process2', args = (tasks, 2))

    process1.start()

    time.sleep(0.2)

    process2.start()

    process1.join()
    process2.join()

Above script gives me the output

None
None

The correct output should be in the style of

name.a
name.b

If I create the instances inside the function run_task it will work fine, but I'm looking for a way to make it work by creating the instances in the main process. Is this possible?

Chicony
  • 393
  • 3
  • 17
  • Try [using a custom pickler](https://docs.python.org/3/library/copyreg.html). – Phillip Dec 01 '16 at 15:52
  • Every windows(OS) `class access` required a registered point on Python. You access a DLL but how to set all `data access` procedure. Need using `pythoncom.CoInitialize()`. [check this](http://stackoverflow.com/questions/26764978/using-win32com-with-multithreading) – dsgdfg Dec 02 '16 at 08:12
  • 1
    Umm... `import special_class` followed by `m1 = special_class("a.dll")` isn't legal; you can't call a module. I assume it's slightly different in the real code, but either way, the Cython class definition is important here; you can't omit it. – ShadowRanger Dec 05 '16 at 20:46

3 Answers3

2

I believe multiprocessing.Process pickles all its arguments. So you need to tell Python how to pickle special_class. You just need to implement method special_class.__reduce__ so that the data can be pickled properly.

christian.simms
  • 866
  • 8
  • 6
2

The multiprocessing library works by pickling objects and then piping the data to other spawned processes. The issue is that your special_class is unpicklable.

If I create the instances inside the function run_task it will work fine

This works because then the object does not need to be pickled, which works around the issue.

You need to make your special_class picklable. This can be done in various ways. They are all documented here: https://docs.python.org/3/library/pickle.html#pickle-inst

Basically, there are 3 mechanisms:

  • Use a custom pickler
  • Implement a __reduce__ method on special_class
  • Implement __getstate__ and __setstate__ methods on special_class (if your class instances have states)

I have a feeling that you have a reference to an external object in your special_class. In that case, refer to: https://docs.python.org/3/library/pickle.html#persistence-of-external-objects

pradyunsg
  • 18,287
  • 11
  • 43
  • 96
0

It seems as if you are making m1 and m2 both the full special_class module. If you are trying to make them a certain class, either do:

from special_class import *

(which I recommend) or 

m1 = special_class.special_class("a.dll")
m2 = special_class.special_class("b.dll")

The None is probably appearing because the methods you input m1 and m2 into also accept the module, for some reason. I would suggest trying from special_class import * and work it out.

GreenHawk1220
  • 133
  • 1
  • 10
  • No, the `None` appears because it's not able to pickle the object. If I dont use the multiprocessing module I dont get this problem. – Chicony Dec 07 '16 at 08:27