Python multiprocessing calling function from module

Question

I have a my_module module with a my_func() function. When I use this function in the parent process, then everything works well (I get the correct result from this function). But when I call this function from child processes then I get this error:

AttributeError: module 'my_module' has no attribute 'my_func'

Does anybody know what I did wrong? Here is my template code:

import pandas as pd, gc, sys, multiprocessing as mp, traceback as tb
sys.path.append('c:/my_libs/')
import my_module

def mp_func(df, i):
    df['res'] = df.apply(lambda x: my_module.my_func(x, i))
    return df

def other_function(df):
    # do something
    my_list = [1,2,3] # it can change dynamically based on previous code

    df_res = pd.DataFrame()

    mp_pool = mp.Pool(processes=min(len(my_list), mp.cpu_count()))
    try:
        for i_df in [y.get() for y in [mp_pool.apply_async(mp_func\
            , [df[['my_attr']], i]) for i in my_list]]:

            df_res = df_res.append(i_df)

    except Exception as e:
        print('Something went wrong during multiprocessing:\n', e)
        print('\nFull traceback:\n', tb.format_exc())

    finally:
        mp_pool.close()
        gc.collect()

    df_res.to_csv(<...>)    

if __name__ == '__main__':
    df = pd.read_csv(<...>)
    other_function(df)

And my_module module looks like this:

def my_func(x, i):
    return x * i

Again, if I run the NOT multiprocessed version of the same code then everything works well. (I use python 3)

Thank you!

Hmm, this template code works perfectly... My real code is too large, complex and "private" to publish, but I try to find out what the mistake is... — ragesz, May 25 '16 at 13:19
Strange. What happen if right after your import statement, immediately call `my_func`, just throw in some wild parameter, e.g. `my_func(1, 2)` -- See if you get the same error. Also, delete my_module.pyc to see if it helps. — Hai Vu, May 26 '16 at 00:57
Thank you for your tip. It did not help directly, but now I had some time to find the problem. — ragesz, May 26 '16 at 17:21

score 0 · Accepted Answer · edited May 23 '17 at 10:28

0

Finally I found the problem. I use Spyder, and in it the default working directory was set to an other folder where I had a previous version of my module. This topic helped me to print out the path of my imported modules in the child processes.

edited May 23 '17 at 10:28

Community

1
1

answered May 26 '16 at 17:19

ragesz

9,009
20
71
88

Python multiprocessing calling function from module

1 Answers1