2

Environment:

Python 3 and Windows 10

I have some cpu intensive code that currently runs through all rows in a data-set and computes something for each row. It currently takes about 1 minute to do around 10k rows and I think this can can dramatically reduced with a process pool (it doesn't matter what order the results are computed in).

I am relatively new to python programming and after some research, futures seems ideal. Doing something like:

with futures.ProcessPoolExecutor() as executor:
   my_futures = [executor.submit(my_function,row) for row in my_data]
   ..do stuff with results

The problem is that in windows it seems that I need to run this code in the main module?

if __name__ == '__main__':

Due to recursive calls (and yes I managed to crash my computer like others have :) )

My code is in a class called by another class so it is not under the main module. Without rewriting everything (and making the code somewhat clunky) I don't see a way to sort this. Is there a way around this or another library for concurrent processes which can be used?

akalikin
  • 1,071
  • 12
  • 35
charlie6411
  • 96
  • 6
  • How does it crash? Does it display an error message? – Lambda Fairy Sep 29 '15 at 08:54
  • RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() – charlie6411 Sep 29 '15 at 08:59
  • http://stackoverflow.com/questions/18204782/runtimeerror-on-windows-trying-python-multiprocessing is the basic error – charlie6411 Sep 29 '15 at 09:01

0 Answers0