Multiprocessing a loop inside a loop inside a function

Question

I wrote some code to break up a for loop into multiple processes to speed up calculations.

import numpy as np
import formfactors
from subdivide_loop import subdivide_loop
import multiprocessing


def worker(start, end, triangleI, areaI, scene, kdtree, samples, output):
    form_factors = np.zeros(end-start)
    for j in range(start, end):
        triangleJ = np.array(scene[j][0:4])
        form_factors[start] = formfactors.uniform(triangleJ, triangleI, areaI, kdtree, samples)
    result = output.get(block=True)
    for j in range(start, end):
        result[j] = form_factors[j]
    output.put(result)


def calculate_formfactors(start, end, triangleI, areaI, scene, kdtree, samples, output, nb_processes,
                          max_interval_length):
    intervals = subdivide_loop(start, end, max_interval_length, nb_processes)
    print("start")
    jobs = []
    for k in range(nb_processes):
        p = multiprocessing.Process(target=worker,
                                    args=(intervals[k][0], intervals[k][1], triangleI, areaI, scene, kdtree,
                                          samples, output))
        jobs.append(p)
    for p in jobs:
        p.start()
    for p in jobs:
        p.join()
    results = output.get()
    return results

I would like to be able to call calculate_formfactors() inside a function inside a loop, like this:

def outer_function():
    for i in range(1000):
        for j in range(i + 1, 1000, max_interval_length):
            form_factors = calculate_formfactors(args)

But running this gives an error:

An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:

    if __name__ == '__main__':
        freeze_support()
        ...

Because of how the outer function works, breaking up outer_function() instead of calculate_formfactors() is not possible.

So, any advice on how to do this?

score 0 · Accepted Answer · answered Dec 07 '17 at 23:14

As the error suggests, make sure your outer_function() (or whatever initiates it) is called from within an __main__ guard, e.g.

if __name__ == "__main__":
    outer_function()

It doesn't have to be the outer_function() but you need to trace it all back to the first step that initializes the chain that ultimately leads to the call to multiprocessing.Process() and put it within the above block.

This is because on non-forking systems multiple processes are essentially run as subprocesses so creating new processes from the main script would end up with an infinite recursion/processes spawning. You can read more about it in this answer. Because of that, you have to make sure your multiprocessing initialization code executes only once which is where the __main__ guard comes in.

Multiprocessing a loop inside a loop inside a function

1 Answers1

Linked