4

I'm having trouble understanding why I would use the concurrent.futures module. The ProcessPoolExecutor seems comparable to the multiprocessing pool for processes, and likewise for the ThreadPoolExecutor vs the multiprocessing.dummy module for threads (or even the undocumented multiprocessing.pool.ThreadPool).

For example, if I want to concurrently map some function call and have an iterator returned, I can use multiprocessing like this:

import multiprocessing

bar = [some list]
pool = multiprocessing.Pool()
results = pool.imap(foo, bar)

Or I can use concurrent.futures with a ProcessPoolExecutor like this:

from concurrent.futures import ProcessPoolExecutor

bar = [some list]
with ProcessPoolExecutor() as pool:
    results = pool.map(foo, bar)

It seems like concurrent.futures provides the ability to cancel asynchronous calls before they complete, but the multiprocessing pools provide a lot more methods that can be selected and tailored specifically for your needs, which makes it much more flexible without (as far as I can tell) adding much complexity.

I have read over a couple of resources on the concurrent.futures module:
http://www.developer.com/lang/other/using-the-new-python-32-concurrent-programming-features.html
http://eli.thegreenplace.net/2013/01/16/python-paralellizing-cpu-bound-tasks-with-concurrent-futures https://gist.github.com/mangecoeur/9540178

But I still don't see what I can gain from it that I didn't already have from process or thread pools. What advantages are there to this new module and when should I use it over the existing pools?

Community
  • 1
  • 1
skrrgwasme
  • 9,358
  • 11
  • 54
  • 84
  • I seem to recall seeing somewhere that concurrent.futures is a newer implementation (based on futures) with a general approach more compatible with current developments in python. – songololo Dec 02 '15 at 08:58

0 Answers0