How to pass object from main process to child process using multiprocessing

Question

I try to use Multiprocessing in a class.I use Multiprocessing.pipe() to pass the instance o from parent process to child process.

        self.listofwritetags = self.collectwritetaglist()
        self.progressbar['value'] = 20
        self.frame.update_idletasks()
        self.alldevice = alldevices_V3.AllDevices(self.comm_object)
        self.progressbar['value'] = 40
        self.frame.update_idletasks()

        con1,con2 = multiprocessing.Pipe()
        con1.send(self.alldevice)
        con2.send(self.comm_object)        


        # Multithreading section
        # self.callmotor1dprocess = thread_with_trace(target=self.callallmotor1d,args=(self.comm_object,self.alldevice))
        self.callmotor1dprocess = multiprocessing.Process(target = self.callallmotor1d,args= (con1,con2))
        self.listofthread.append(self.callmotor1dprocess)



        self.button2.config(text="Initialized")
        initial = "True"
        self.progressbar.stop()

Now I call all Multiprocessing Initiate

def startprocess(self):
        for item in self.listofthread:
            item.start()
        self.button3.config(text="started")

   def stopprocess(self):
        for item in self.listofthread:
            item.kill()

This Code I call inside the class.Now Executed method I should call outside of the class.

def callallmotor1d(con1,con2):
comobject = con1.recv()
devices = con2.recv()
while True:
    Allmotorprocessing.process(comobject, devices) 

But I got the error which is very common:- Error message:-

Traceback (most recent call last): File "C:\Users\misu01\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 236, in _feed obj = _ForkingPickler.dumps(obj) File "C:\Users\misu01\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) TypeError: can't pickle _thread.lock objects Traceback (most recent call last): File "C:\Users\misu01\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 236, in _feed obj = _ForkingPickler.dumps(obj) File "C:\Users\misu01\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) TypeError: can't pickle _thread.lock objects

I don't know why thread.lock object is created. To avoid this error I try to modify my Alldevices class and comm_object class like this:-

Here is my modification:-

class AllDevices:

def __init__(self,comobject):
    self.mylock = threading.Lock()
    self.comobject = comobject
    self.dfM1D = pd.read_excel(r'C:\OPCUA\Working_VF1_5.xls', sheet_name='Motor1D')
    self.allmotor1dobjects = callallmotor1D_V3.Cal_AllMotor1D(self.dfM1D, self.comobject)


def __getstate__(self):
    state = vars(self).copy()
    # Remove the unpicklable entries.
    del state['mylock']
    return state

def __setstate__(self, state):
    # Restore instance attributes.
    vars(self).update(state)

Here is the comobject class.

class General():
def __init__(self):
    self.client = Communication()
    self.mylock = threading.Lock()
    self.sta_con_plc = self.client.opc_client_connect()
    self.readgeneral = ReadGeneral(self.client.PLC)
    self.writegeneral = WriteGeneral(self.client.PLC)

def __getstate__(self):
    state = vars(self).copy()
    # Remove the unpicklable entries.
    del state['mylock']
    return state

def __setstate__(self, state):
    # Restore instance attributes.
    vars(self).update(state)

But still I got an error.

Is this my implementation is correct?

self.allmotor1dobjects = callallmotor1D_V2.Cal_AllMotor1D(self.dfM1D, self.comobject,self.logger)

Here self.allmotor1dobjects is also class instances.

like:-

    self.client = Communication()        
    self.readgeneral = ReadGeneral(self.client.PLC)
    self.writegeneral = WriteGeneral(self.client.PLC)

These are also class instances.

I never used thread.lock any of these two classes. I don't know how it is created.

As per the docs suggested https://docs.python.org/3/library/pickle.html#pickling-class-instances If I use getstate and setstate, it should remove this error.

In my case it doesn't work.

How can I remove this error.

any help in this regard will be highly appreciated

Answer 1

You can pass most classes through a pipe, but not if the instances have attributes that are unpicklable types. In this case, the instances of alldevice (the instances of which are stored in some collection you stored as self.devices) have, directly or indirectly, a threading.Lock attribute (the error message says _thread.lock because under all the abstraction, that's the actually class that threading.Lock returns an instance of).

threading.Lock isn't picklable (because thread locks only make sense within a given process; even if you recreated them in another process, they wouldn't actually provide any sort of synchronization between the processes).

If the alldevice class is under your control, you have a few options:

1. Remove the per-instance threading.Lock if possible (easiest solution, but assumes synchronization isn't required, or that synchronization could be done by a shared global lock)
2. If synchronization is required, and must operate across processes, but needn't be per-instance, and you're on a UNIX-like system (Linux, BSD) you could use a shared global multiprocessing.Lock() instead; it will be inherited when the Process call triggers a fork

3. If synchronization is required, and you must have a per-instance Lock (and it's okay for the lock to operate across processes), you can replace threading.Lock with a pickle-friendly multiprocessing.Manager's Lock. You'd need to make a common multiprocessing.Manager() instance somewhere (e.g. globally, or as a class attribute of alldevice), then use Lock from that manager instead of from threading. Simple example:

   import multiprocessing
   
   class alldevice:
       MANAGER = multiprocessing.Manager()  # Shared manager for all alldevice instances
   
       def __init__(self, ... other args here ...):
           self.mylock = self.MANAGER.Lock()  # Make picklable lock
           # ... rest of initialization ...
   

   When multiprocessing isn't involved, this will be slower than a threading.Lock, as it will require IPC to lock and unlock the lock (the Manager's Lock instances are actually a proxy object that communicates with the Manager on whatever process it's actually running in), and it will lock across processes (probably for the best if you're locking access to actual hardware that can't be used concurrently from multiple processes), but it's relatively simple.

4. If synchronization is required, but only within a process, not across processes, you can take control of the pickling process to avoid trying to pickle the threading.Lock, and instead recreate it when it's unpickled on the other side. Just explicitly implement the pickle support methods to avoid pickling the Lock, and force it to be recreated on the other side. Example:

   import copy
   
   class alldevice:
       def __init__(self, ... other args here ...):
           self.mylock = threading.Lock()  # Use regular lock
           # ... rest of initialization ...
   
       def __getstate__(self):
           state = vars(self).copy()  # Make copy of instance dict
           del state['mylock']        # Remove lock attribute
           return state
   
       def __setstate__(self, state):
           vars(self).update(state)
           self.mylock = threading.Lock()  # Make new lock on other side
   
       # If you ever make copies within the same thread, you may want to define
       # __deepcopy__ so the local copy process doesn't make a new lock:
       def __deepcopy__(self, memo):
           # Make new empty instance without invoking __init__
           newself = self.__class__.__new__(self.__class__)
           # Individually deepcopy attributes *except* mylock, which we alias
           for name, value in vars(self).items():
               # Cascading deepcopy for all other attributes
               if name != 'mylock':
                   value = copy.deepcopy(value, memo)
               setattr(newself, name, value)
           return newself
   

   The __deepcopy__ override is only needed if you want the copy to continue sharing the lock; otherwise, if a deep copy should behave as an entirely independent instance, you can omit it, and you'll end up with an unrelated lock in the copy.

If you don't have control of the alldevice class, but can identify the problematic attribute, your only option is to register a copyreg handler for alldevice to do the same basic thing as option #4, which would look something like this:

import copyreg

def unpickle_alldevice(state):
    self = alldevice.__new__(alldevice)  # Make empty alldevice
    vars(self).update(state)             # Update with provided state
    self.mylock = threading.Lock()       # Make fresh lock
    return self        

def pickle_alldevice(ad):
    state = vars(ad).copy()  # Make shallow copy of instance dict
    del state['mylock']      # Remove lock attribute
    return unpickle_alldevice, (state,)  # Return __reduce__ style info for reconstruction

# Register alternate pickler for alldevice
copyreg.pickle(alldevice, pickle_alldevice)