All example concurrent.futures code is failing with "BrokenProcessPool"

Question

I am trying to get a basic understanding of this before I create the actual application I need. I recently moved over from 2.7 to 3.3.

A direct copy-paste of this code from the python docs fails, as does a slightly simpler example from here.

This is my code, derived from the second example:

import concurrent.futures

nums = [1,2,3,4,5,6,7,8,9,10]

def f(x):
    return x * x

# Make sure the map and function are working
print([val for val in map(f, nums)])

# Test to make sure concurrent map is working
with concurrent.futures.ProcessPoolExecutor() as executor:
    for item in executor.map(f, nums):
        print(item)

And this is the output:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
Traceback (most recent call last):
  File "<string>", line 420, in run_nodebug
  File "<module1>", line 13, in <module>
  File "C:\Python33\lib\concurrent\futures\_base.py", line 546, in result_iterator
    yield future.result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 399, in result
    return self.__get_result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 351, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

How can I get this code to work as expected? I was hoping the examples would just work out of the box.

Maybe an helpful explanantion: https://stackoverflow.com/a/59320867/1315472 — Nairolf, May 20 '20 at 19:25

score 76 · Accepted Answer · answered Apr 10 '13 at 06:42

76

This was my fault, for two reasons:

The code was un-guarded, i.e no if __name__
The strange looking Traceback was because the file was not saved. Never caused me an issue before, but did in this case.

Correcting both of those fixed the error.

Final test code:

import concurrent.futures

nums = [1,2,3,4,5,6,7,8,9,10]

def f(x):
    return x * x
def main():
    # Make sure the map and function are working
    print([val for val in map(f, nums)])

    # Test to make sure concurrent map is working
    with concurrent.futures.ProcessPoolExecutor() as executor:
        print([val for val in executor.map(f, nums)])

if __name__ == '__main__':
    main()

Output, as expected:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

answered Apr 10 '13 at 06:42

Pep_8_Guardiola

5,002
1
24
35

9

That's so strange. It only works for me in a saved .py file. It fails when I tried it in either the iPython console or in Jupyter notebook. Good find! – David Oct 22 '18 at 20:38
25

@David It is now mentioned in the [docs for ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html) that it will not work in interactive consoles. It says *The __main__ module must be importable by worker subprocesses. This means that ProcessPoolExecutor will not work in the interactive interpreter.* – spirit Mar 14 '19 at 15:20
3

Another finding of mine is that, that message appears when you try to execute Process pools inside Windows. When executing under Linux, it ran fine. – Mauricio Maroto Dec 07 '20 at 05:45
1

I'm running this exactly as shown in jupyter lab (windows) and I only see the first line printed followed by "A process in the process pool was terminated abruptly while the future was running or pending." Edit: I seem to recall in windows one has to use ThreadProcessExecutor rather than ProcessThreadExecutor. When swapping the calls, it ran as expected. – thistleknot Dec 26 '21 at 13:35
Your latest code still gives the same error in my case: ended abruptly. – MsTais May 09 '22 at 19:31
I encountered this error when I multi-processed my algorithm using more processes than my computer could carry anjd the ram 'exploded'. I just decreased the number of workers and the rest is history – Karantai May 20 '22 at 08:11

score 39 · Answer 2 · answered Jul 25 '17 at 11:57

Under Windows, it is important to protect the main loop of code to avoid recursive spawning of subprocesses when using processpoolexecutor or any other parallel code which spawns new processes.

Basically, all your code which creates new processes must be under if __name__ == '__main__': , for the same reason you cannot execute it in interpreter.

All example concurrent.futures code is failing with "BrokenProcessPool"

2 Answers2

Linked