Python multiprocessing from a imported file

Question

Go ahead to the las edit please

I'm facing some problems to make this code run correctly. It should be easy but there is something I do not manage correctly with "if name = 'main'"

I have 2 files, test1 and test2 and it works as simple as:

TEST1

import test2

test2.call(0,7)

TEST2

import multiprocessing as mp

in1 = 0
in2 = 0

def cube(x):
    return x ** 3

def call(data1, data2):
    global in1
    global in2
    in1 = data1
    in2 = data2


if __name__ == '__main__':
    pool = mp.Pool(processes=7)
    results = [pool.apply_async(cube, args=(x,)) for x in range(in1, in2)]
    output = [p.get() for p in results]
    print(output)

I cannot make run the multiprocess because of the "__ main __" condition but if I change the condition to "test1", nothing happens and with "test2" as condition I receive a infinite loop with an error:

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

How should I proceed?

More:

Hello,

I just read In the documentation: "Calling freeze_support() has no effect when invoked on any operating system other than Windows. In addition, if the module is being run normally by the Python interpreter on Windows (the program has not been frozen), then freeze_support() has no effect."

Is is happening to my script? I always get the infinite loop in Pycharm using Python interpreter. How could I test it?

New Try

If I execute this script directly, it works and I get my calcs.

def cube(x):
    calc = x ** 3
    print(calc)
    return calc

if __name__ == '__main__':

print('hola')
x = 2
y = 4
p1 = mp.Process(target=cube, args=(x,))
p2 = mp.Process(target=cube, args=(y,))

p1.start()
p2.start()

p1.join()
p2.join()

But when I execute it from another file and I change the condition __ main __ for "test2" (name of the own execution) I get the error showed above. What is so wrong trying to call a multiprocess from another file?

Any specific reason why you're using global variables? You could just get rid of them and put the multiprocessing code into the function `call`. If you need `in1` and `in2` for other functions as well, consider using a class ;) — swenzel, May 03 '18 at 15:31
Possible duplicate of [python multiprocessing on windows, if \_\_name\_\_ == "\_\_main\_\_"](https://stackoverflow.com/questions/20222534/python-multiprocessing-on-windows-if-name-main) — William Perron, May 03 '18 at 15:57
If I use multiprocessing inside the function call the full function will be repeated right ? I don't want to rerun this function again — mhkore, May 03 '18 at 19:35

score 2 · Accepted Answer · answered May 04 '18 at 12:19

I realised that is not possible to initiate a multitask run from another file. The way to avoid this situation was importing the function needed and creating the multitask run in the same file.

TEST 1

from test2 import cube
import multiprocessing as mp

in1 = 0
in2 = 4
pool = mp.Pool(processes=7)
    results = [pool.apply_async(cube, args=(x,)) for x in range(in1, in2)]
    output = [p.get() for p in results]
    print(output)

TEST2

def cube(x):
    calc = x ** 3
    print(calc)
    return calc

It is a solution, but I would like to have found a more elegant one.

Thanks

score 1 · Answer 2 · answered May 03 '18 at 14:46

1

You can debug to see what name corresponds to. Like add this to test2.py:

def print_name():
    print(__name__)

Now you should see that since it's in module test2.py, your name when importing this module there will be test2

Also, notice that you're importing test2, but calling from non-existing test. So you might try to call from the lib you imported:

test2.call(0, 7)

answered May 03 '18 at 14:46

Andrew_Lvov

4,621
2
25
31

Thank, I edited that test2.call mistake (I was using other right code) Anyway, using "test2" for the if __ name __ condition I get a infinite loop and I don't know why __ERROR__: This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() – mhkore May 03 '18 at 16:05

score 1 · Answer 3 · answered Oct 30 '20 at 14:58

This works for me:

TEST 1

import test2
if __name__ == '__main__':
    test2.multiprocess_data(0,7)

TEST 2

import multiprocessing as mp

def cube(x):
    return x ** 3

def multiprocess_data(data1, data2):
    if __name__ == 'test2':
        pool = mp.Pool(processes=7)
        results = [pool.apply_async(cube, args=(x,)) for x in range(data1, data2)]
        output = [p.get() for p in results]
        print(output)

Python multiprocessing from a imported file

3 Answers3