14

I am trying to use the multiprocessing package for Python. In looking at tutorials, the clearest and most straightforward technique seems to be using pool.map, which allows the user to easily name the number of processes and pass pool.map a function and a list of values for that function to distribute across the CPUs. The other technique that I have come across is using queues to manage a pool of workers. This answer does an excellent job explaining the difference between pool.map, pool.apply, and pool.apply_async, but what are the advantages and disadvantages of using pool.map versus using queues like in this example?

Community
  • 1
  • 1
Michael
  • 13,244
  • 23
  • 67
  • 115

2 Answers2

10

The pool.map technique is a "subset" of the technique with queues. That is, without having pool.map you can easily implement it using Pool and Queue. That said, using queues gives you much more flexibility in controlling your pool processes, i.e. you can make it so that particular types of messages are read only once per processes' lifetime, control the pool processes' shutdown behaviour, etc.

Maciej Gol
  • 15,394
  • 4
  • 33
  • 51
  • Does `pool.map` automatically shut down the processes when it completes? – Michael Sep 26 '13 at 17:47
  • 1
    From the docs: ``When the pool object is garbage collected terminate() will be called immediately.``. But i wouldn't rely on it - it'd better if you had ensured the pool has closed at the end of your app. Otherwise, you might end up with some zombies left in your system. As for `pool.map` - no, it doesn't shut down the processes. The processes live as long as their pool unless closed or orphaned. – Maciej Gol Sep 26 '13 at 18:30
4

If you're really looking for the "clearest and most straightforward technique", using concurrent.futures.ProcessPoolExecutor is probably the easiest way. It has a map method as well as some other primitives that make it very usable. It is also compatible with Queues.

Veedrac
  • 58,273
  • 15
  • 112
  • 169