Pickling errors when using multiprocessing

Question

I want to control and run two pieces of lab equipment at the same time (at least starts them simultaneously), so I use multiprocessing module.

from myLabModule import Newport1936R
from myLabModule import DCx_camera
from multiprocessing import Process, Queue

def multiprocess_camera(camera, shots_per_img, queue, delay):
    time.sleep(delay)
    img, camera_time = camera.capture(shots_per_img, np.float64, True) #img is a 2x2 ndarray
    results = [camera_time[0], camera_time[1], img, 'img']
    queue.put(results)

def multiprocess_power(newport, interval, N, queue, delay):
    time.sleep(delay)
    power_reading, newport_time = newport.get_power(interval, N, True)
    results = [newport_time[0], newport_time[1], power_reading, 'power']
    queue.put(results)

if __name__ == "__main__":
    thorcam = DCx_camera()
    newport = Newport1936R()
    queue = Queue()
    p1 = Process(target=multiprocess_camera, args=(thorcam,10,queue,0))
    p2 = Process(target=multiprocess_power, args=(newport,1,10,queue,0))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    results = []
    results.append(queue.get())
    results.append(queue.get())

When I run the code, I get the following error: AttributeError: Can't pickle local object 'CDLL.__init__.<locals>._FuncPtr'. I read this post, it seems that it has to do with the scope of the variables (I am passing two classess defined in another module to the functions, and the functions basically run the classes' methods). What I'm not sure exactly which part is causing the problem. How should I modify the code to make it work?

why do you have to create instances and then forward them just to call their methods with forwarded arguments? Looks like that is the issue, and if that is not a strict requirement, why don't you make instances in functions? — Ilija, Feb 16 '18 at 18:13
@ikac Sorry I don't quite understand what you mean. As I said what I wanna achieve at the end is to execute `thorcam.capture()` and `newport.get_power()` at the same time (using multiprocessing) and also get their return values. I just checked a few relevant posts and it seems that making a separate functions is the way to do it, of course I'm open to other better methods. — Physicist, Feb 17 '18 at 09:00
Well why don't you create instances in target methods? If real use case is similar to this example, you do not have code referencing instances `thorcam` and `newport` bellow processes are started. — Ilija, Feb 17 '18 at 11:57
Do mean defining `thorcam` and `newport` in the current script? I can do so, the only problem is the module I have written for `thorcam` and `newport` are a few hundred lines long, so I think it looks cleaner if I keep them as separate files. Is it possible to just redefine the classes in a few lines and referring the actual code to the separate files? — Physicist, Feb 18 '18 at 09:57
I mean to not have to worry about pickling at all. Here, pickling is performed when you pass instance as an argument for target, e.g. `args=(thorcam,10,queue,0)`. `thorcam` has to be pickled and I'm not sure how `DCx_camera` is implemented. — Ilija, Feb 18 '18 at 12:47
Oh you mean defining the `thorcam` and `newport` at the top of the script (make them global variables?) and not to pass them as arguments in the two functions? — Physicist, Feb 18 '18 at 13:39

Ilija · Answer 1 · 2018-02-18T16:24:21.500

In this case pickling is performed when passing arguments to target method that will be executed as new (child) process.

To be precise, I'm not sure if pickling instances of DCx_camera and Newport1936R is possible at all. Depends on how those classes are defined, but looking at the error message I suppose you can't. That is why you can try to not pass those references at all. It seems, by the example, that you do not need to.

Here is what I expect to work:

from myLabModule import Newport1936R
from myLabModule import DCx_camera
from multiprocessing import Process, Queue

def multiprocess_camera(shots_per_img, queue, delay):
    camera = DCx_camera()
    time.sleep(delay)
    img, camera_time = camera.capture(shots_per_img, np.float64, True) #img is a 2x2 ndarray
    results = [camera_time[0], camera_time[1], img, 'img']
    queue.put(results)

def multiprocess_power(interval, N, queue, delay):
    newport = Newport1936R()
    time.sleep(delay)
    power_reading, newport_time = newport.get_power(interval, N, True)
    results = [newport_time[0], newport_time[1], power_reading, 'power']
    queue.put(results)

if __name__ == "__main__":
    queue = Queue()
    p1 = Process(target=multiprocess_camera, args=(10,queue,0))
    p2 = Process(target=multiprocess_power, args=(1,10,queue,0))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    results = []
    results.append(queue.get())
    results.append(queue.get())

Is this something that can get you desired results?

There is an alternative, according to this link.

Better to inherit than pickle/unpickle

However, one should generally avoid sending shared objects to other processes using pipes or queues. Instead you should arrange the program so that a process which needs access to a shared resource created elsewhere can inherit it from an ancestor process.

That is why I think you can even have global variables that reference instances of DCx_camera and Newport1936R, and then use those in both target methods.

Pickling errors when using multiprocessing

1 Answers1