Dask Multiprocessing

Question

I am trying to run the following code. It works if executed in a shell but crashes if executed as macro (py macro.py). Please could you tell me what's wrong. Thank you

import os
import sys
import dask
from dask.distributed import Client

def inc(x):
    return x + 1

def add(x, y):
    return x + y

client = Client(n_workers=2, threads_per_worker=2, memory_limit='1GB')

a = client.submit(inc, 10)
b = client.submit(inc, 20)
print(a.result())
print(b.result())

Can you post an error message? Also it's not clear to me at least what you mean by run as a macro (unless py is an alias for python or python3)? — David Waterworth, Nov 06 '19 at 04:35
`if executed as macro` Python doesn’t have macros, do you mean that the _file_ is named “macros”? — AMC, Nov 06 '19 at 04:36
I meant storing the code in macro.py and executing the code (py macro.py) — jpkoc, Nov 06 '19 at 04:44
..... RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable. — jpkoc, Nov 06 '19 at 04:47

score 1 · Answer 1 · answered Nov 07 '19 at 04:05

1

This is a problem with running scripts that create processes. You need to create your Client object within the if __name__ == "__main__": block

See the answer in where to put freeze_support() in a Python script? for more information.

answered Nov 07 '19 at 04:05

MRocklin

55,641
23
163
235

thank you MRocklin . One last question. Is this recommandation Windows specific? I use a Mac. – jpkoc Nov 07 '19 at 05:31
The `__main__` thing is general good practice, but there are additional constraints on windows which require freeze. – mdurant Nov 07 '19 at 14:55

Dask Multiprocessing

1 Answers1