Python concurrent.futures multiple calls to imported modules

Question

Before diving in the details I want to mention that've read this post, which problematic seems very similar : Python current.futures import libraries multiple times (execute code in top scope multiple times)

Spyder 4.0.1 / Python 3.7.1 64-bit / Windows 10

Yet, having taken the careful steps of protecting my top levels code does'nt seems to be working in my case. I'm using concurrent.futures with ProcessPollExecutor to execute the code below. Outside pandas, numpy... the code calls two custom modules : HAL_PROC and HAL_FUNC, containing the classes and functions necessary to execute the loop below.

When executed, HAL_PROC.py calls an other module that generate a GUI where she or he can specify inputs path, file names... this values or then stored in variables.

My problem is : when I run the code, wether in Spyder or or when compiled into an .exe file, and it executes the main() function the GUI spawns 8 times on the screen. Eventhought there're no call to the module or any related module in the main section since everything has been stored in variabled even the functions.

I'm sure i'm missing important nuance on how to properly set up my code for multiprocessing, any help would be very appreciated.

import pandas as pd
import numpy as np
from HAL_PROC import N_tab, N_tab_FCT, TYPE, TR, CA
from HAL_FUNC import *
import multiprocessing
from concurrent.futures import ProcessPoolExecutor


alpha = 0.05
alg_type = TYPE

if alg_type == "O":
    F = FFR_ME

elif alg_type == "NO":
    F = FFR_ME_NO        

else :
    raise TypeError("Algortithm type selected does not exist. Must be 'O' or 'NO'")


def main():

    """ Set the variables for the loop """
    futures = []
    e = ProcessPoolExecutor(8)      

    """ Loop over the different task summarized in the tab 'N_tab' during the MPO_PROC step. """
    for task in N_tab["TASK_NUMBER"]:

         """ Declare variables N, n , f based on the previous calculations """ 

         N = int(N_tab.loc[N_tab["TASK_NUMBER"] == task, "N_Task"])
         n = int(N_tab.loc[N_tab["TASK_NUMBER"] == task, "n_i"])
         f = int(N_tab.loc[N_tab["TASK_NUMBER"] == task, "F"])

         """" Implement the function using the concurrent.future module for multiprocessing. """ 

         future = e.submit(F, N, n, f, alpha)     
         futures.append(future)

    results = [ff.result() for ff in futures] 

    for i in range(len(results)):
        f = int(N_tab.loc[i, "F"])                                                
        N_tab.loc[i,"LBound"] = results[i][0][f]
        N_tab.loc[i,"UBound"] = results[i][1][f]
        N_tab.loc[i,"ME"] = (N_tab.loc[i,"UBound"] - N_tab.loc[i,"LBound"])/\
                                               (2*N_tab.loc[i,"N_Task"])
        N_tab.loc[i,"FFR"] = (N_tab.loc[i,"LBound"] + (N_tab.loc[i,"UBound"] - N_tab.loc[i,"LBound"])/2)/\
                                                N_tab.loc[i,"N_Task"]                                                

if __name__ == "__main__":
    multiprocessing.freeze_support()
    main()

score 0 · Accepted Answer · answered May 19 '20 at 18:23

After some digging I found a solution to my problem, In case it might be of help. Contrary to a lot of posts I've found, guarding my code with the if "__name__" == __main__ was not enough.

1) I change to multiprocessing library instead of concurrent.future which is a simplified version of the former. (see code below).

2) Turns out multiprocesing on Windows in an Ipython console is not very reliable (Python multiprocessing pool stuck). To solve the problem of multiple calls to the import I added this piece of code at the begining :

import sys
sys.modules['__main__'].__file__ = 'ipython'

In this end the code look something like this and works perfectly, (no broken process pool nor multiple GUI poping up anymore).

import pandas as pd
import numpy as np
import json

from HAL_PROC import N_tab, N_tab_FCT, TYPE, TR, CA
from HAL_FUNC import *
from HAL_GRAPH import Dossier_graph, Forecast_graph

from multiprocessing import Pool
import multiprocessing as mp
import matplotlib.backends.backend_pdf  

import sys
sys.modules['__main__'].__file__ = 'ipython'


""" ========================================================== """
"""                   FFR +/- ME CALCULATION                   """
""" ========================================================== """


def main():    
    with Pool() as p:
         results = p.starmap(F,zip(N,n,f,alpha))
         results_fct = p.starmap(F,zip(N_fct,n_fct,f_fct,alpha_fct))
         p.close()
         p.join()

if __name__ == "__main__":
    N_tab, N_tab_FCT = main()

Python concurrent.futures multiple calls to imported modules

1 Answers1