Python Multiprocessing Module

Question

I have this script but It returns an error. The problem is that I omit to write

if _name_ == "__main__":

This is the script I took it from a tutorial but Im adapting it to my needs

import concurrent.futures
import time

start = time.perf_counter()


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleeping...{seconds}'


with concurrent.futures.ProcessPoolExecutor() as executor:
    secs = [5, 4, 3, 2, 1]
    results = executor.map(do_something, secs)

    # for result in results:
    #     print(result)

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

the problem I have is that I'm using Windows and I don't know where I have to write: if __name__ == "__main__":

You have to put any function right after `if _name_ == "__main__":` — KingDarBoja, Aug 26 '20 at 20:32
You have to put that so that it encompasses the use of the `executor`, so right after your definition of `def do_something` — juanpa.arrivillaga, Aug 26 '20 at 20:34
Take a look at this example: https://stackoverflow.com/questions/53924018/python-multiprocessing-on-windows Cheers! — KingDarBoja, Aug 26 '20 at 20:35
Its too bad that the tutorial ignores something so basic to cross platform coding. If there is a way to provide feedback to the author, it would be good to mention it. — tdelaney, Aug 26 '20 at 20:44
@tdelaney The `multiprocessing` [documentation](https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods) states: "Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process)." — Roland Smith, Aug 26 '20 at 20:50

bitranox · Answer 1 · 2020-08-26T20:44:05.097

here the working, with inline comments - You need to prevent that main() is called again when forking-like behaviour under windows.

This is only necessary under windows, because windows does not support fork - therefore python simulates some "forking-like" behaviour and tries to create the same environment in the new process :

import concurrent.futures
import time

def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleeping...{seconds}'

def main():
    start = time.perf_counter()
    with concurrent.futures.ProcessPoolExecutor() as executor:
        secs = [5, 4, 3, 2, 1]
        results = executor.map(do_something, secs)

    # for result in results:
    #     print(result)

    finish = time.perf_counter()
    print(f'Finished in {round(finish-start, 2)} second(s)')

if __name__ == '__main__':
    # this guards main() when forking
    main()

score 1 · Answer 2 · answered Aug 26 '20 at 20:40

Your code needs to be import safe because multiprocessing will re-import it in the subprocess. That means that you shouldn't do anything at module level that you don't want to run on mere import. For instance, creating the executor at module level means you'd have an infinite number of subprocesses created because each new one would import the module and create yet another subprocess.

Here's how you would solve the problem

import concurrent.futures
import time

def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleeping...{seconds}'

if __name__ == "__main__":
    start = time.perf_counter()
    with concurrent.futures.ProcessPoolExecutor() as executor:
        secs = [5, 4, 3, 2, 1]
        results = executor.map(do_something, secs)

    # for result in results:
    #     print(result)

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

Python Multiprocessing Module

2 Answers2