5

I’m trying to understand how to use the concurrent.futures module in Python 3.2.2, and have been playing with examples from the documentation. When I try to apply my understanding, my own examples fail. I hope somebody can help me get on track!

I want to be able to set a number of processes running concurrently but asynchronously. My processes don't return anything. To simulate this I have written a trivial example:

import concurrent.futures

fred = [1,2,3,4,5,6,7,8,9,10]

def f(x):
    print(x * x)

def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for num in fred:
            executor.submit(f, num)

if __name__ == "__main__":
    main()

This code runs (on a 4 core Windows XP box) and returns:

1 4 9 16 25

...but then hangs.

Clearly I’m doing something wrong. So, what is the correct way to have Python run processes in a process pool? I don’t want to use the executor.map approach, because I don’t have any return from my process.

Or…do I have to fake it, by having the process return True or False (or something)?

Thanks!

Zearin
  • 1,474
  • 2
  • 17
  • 36
MappaGnosis
  • 1,179
  • 1
  • 11
  • 25

1 Answers1

2

I don't quite understand why you don't want to use executor.map...I tried it with a function that didn't return anything (with your f function, actually) and it worked fine...

now, when you run it with map, it won't actually print the values, but here is a version of your f function that does print the values:

import concurrent.futures

fred = [1,2,3,4,5,6,7,8,9,10]

def f(x):
    return x * x

with concurrent.futures.ProcessPoolExecutor() as executor:
    for num in executor.map(f, fred):
        print(num)
  • 1
    Thanks - The map function in 3.2 doesn't just 'go'. You have to ask it to return something first. I was trying to simulate a much more complex situation where my function does a lot of work, possibly even spawning a subprocess and the result is another file on disk. So I _could_ have the function return 'True' or 'False' I suppose, but I'd then have to have a line of 'fake' code just to make the map function do its job. – MappaGnosis Nov 10 '11 at 15:07
  • I don't think you have to have the function return anything at all...I used `executor.map` with your original `f` function, which doesn't return anything, and it worked just fine (though the data wasn't printed...) –  Nov 10 '11 at 15:08
  • I don't think it does work like that because I have tried an alternative where I put in a time.sleep(x) call in the function and then timed the whole operation. The code takes the same amount of time to execute with the wait as without it. The documentation says that 'map' in 3.2 works differently - which is why I have to do something to make 'map' actually perform the functions. – MappaGnosis Nov 10 '11 at 15:34