1

I am running two models in parallel using Multiprocessing in python with the below code :

def pro(process):      
    #print(process)                                                       
    os.system('python {}'.format(process)) 

def run_model_multiprocessing(ml_model1,ml_model2):

    processes = (ml_model1,ml_model2)   
    pool = Pool(processes=7)
    start = datetime.datetime.now()
    print('Start:',start)   
    pool.map(defs.pro, processes)
    end = datetime.datetime.now()
    print('End :',end)   
    total = end-start
    print('Total :', total)

However both my model return one output file,but I am unable to get that output file in the above multiprocessing process. I have used the below code to get the return output from both the model , but it did not worked for me.

def run_model_multiprocessing(ml_model1,ml_model2):     
    processes = (ml_model1,ml_model2)   
    pool = Pool(processes=7)
    start = datetime.datetime.now()
    print('Start:',start)   
    outdf1,outdf2 = pool.map(defs.pro, processes)
    end = datetime.datetime.now()
    print('End :',end)   
    total = end-start
    print('Total :', total)
    return outdf1, outdf2

Although program run successfully , but there is nothing in outdf1 and outdf2

Aditya sharma
  • 143
  • 2
  • 11

1 Answers1

0

If I understood your query correctly, your objective is running two machine learning models (ml_model1,ml_model2) in parallel for prediction .

The process pool, provides only data parallelism (Distribution of data through the different processes that operate on data in parallel.)

If your objective is running two machine learning models in parallell for prediction, then you should use Process based parallelism or Thread based parallelism.

Refer following link.

Multiprocessing vs Threading Python

Roji Augustine
  • 1