How to pass unpickled object as argument in multiprocessing.Process?

Question

I am unpickling an object and passing as one of the args for process.Getting no output.

I want to know if this method of unpickling and passing an object as argument is causing any error for multiprocessing. What will be work around for this?

from multiprocessing import Process, Queue
def func(arg1, arg2,q):
    df_temp = arg1[arg1['col'].isin(arg2)]
    q.put(df_temp)

if __name__ == '__main__':
    import pickle
    import pandas as pd
    arg1= pickle.load(open('paths.p','rb'))
    arg2= pd.Series(pd.date_range(end = 'some_Date', periods=12,freq = 'MS')).dt.to_pydatetime()
    arg2=[i.date() for i in arg2]
    q = Queue()
    p =Process(target=func, args=(arg1,arg2,q))
    p.start()
    p.join()    
    while not q.empty():
        w=q.get()

score 2 · Accepted Answer · answered Aug 04 '19 at 15:44

Your are deadlocking for other reasons.

By default if a process is not the creator of the queue then on exit it will attempt to join the queue’s background thread. The process can call cancel_join_thread() to make join_thread() do nothing. docs

Your Process won't exit because it is joining a feeder thread from multiprocessing.Queue, which gets started as soon as you queue.put() the first time. You need to queue.get() in your parent before you join the process.

Warning: As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe. This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children. docs

Also don't use while not q.empty(), it's an anti-pattern which will lead to deadlocks as soon as you have multiple consumers. Use sentinel-values instead to notify the consumer that no further items are coming. More about this, here.

How to pass unpickled object as argument in multiprocessing.Process?

1 Answers1