44

Please explain to me what is the difference between these two classes?

I noticed multiprocessing module existed in Python 2. But functionally?

ArekBulski
  • 4,520
  • 4
  • 39
  • 61

1 Answers1

76

As stated in the documentation, concurrent.futures.ProcessPoolExecutor is a wrapper around a multiprocessing.Pool. As such, the same limitations of multiprocessing apply (e.g. objects need to be pickleable).

However, concurrent.futures aims to provide an abstract interface that can be used to manage different types of asynchronous tasks in a convenient way. e.g. changing your async strategy from using process pools to using threads is frequently as simple as changing one or two lines of code (rather than needing to code it all up yourself). Another (related) benefit in the abstraction is that concurrent.futures provides a single API to remember -- And you can pick the Executor that is most suited for the task. Is using your process IO bound? Awesome, use a ThreadPoolExecutor. Are you going to have trouble speeding things up because of the Global Interpreter Lock (GIL)? No problem, use a ProcessPoolExecutor.

mgilson
  • 300,191
  • 65
  • 633
  • 696
  • 15
    @ospider -- Care to elaborate? As far as I can tell, the question is about what the difference is between `ProcessPoolExecutor` and `multiprocessing.Pool`. The answer (as I see it) is that they have basically the same functionality (the latter is a wrapper around the former that provides a different interface for the purpose of making an interface that can also be utilized with threads by "simply" switching out the Executor class). – mgilson Aug 07 '18 at 12:30
  • While concurrent.futures.ProcessPoolExecutor does use the multiprocessing module. It's not a wrapper around Pool. For one thing, it raises an exception if a process of the pool gets killed rather than hang forever as multiprocessing.Pool – Rafael Almeida Apr 09 '21 at 20:16