59

I'm interested in running a Python program using a computer cluster. I have in the past been using Python MPI interfaces, but due to difficulties in compiling/installing these, I would prefer solutions which use built-in modules, such as Python's multiprocessing module.

What I would really like to do is just set up a multiprocessing.Pool instance that would span across the whole computer cluster, and run a Pool.map(...). Is this something that is possible/easy to do?

If this is impossible, I'd like to at least be able to start Process instances on any of the nodes from a central script with different parameters for each node.

astrofrog
  • 32,883
  • 32
  • 90
  • 131

4 Answers4

51

If by cluster computing you mean distributed memory systems (multiple nodes rather that SMP) then Python's multiprocessing may not be a suitable choice. It can spawn multiple processes but they will still be bound within a single node.

What you will need is a framework that handles spawing of processes across multiple nodes and provides a mechanism for communication between the processors. (pretty much what MPI does).

See the page on Parallel Processing on the Python wiki for a list of frameworks which will help with cluster computing.

From the list, pp, jug, pyro and celery look like sensible options although I can't personally vouch for any since I have no experience with any of them (I use mainly MPI).

If ease of installation/use is important, I would start by exploring jug. It's easy to install, supports common batch cluster systems, and looks well documented.

notpeter
  • 1,046
  • 11
  • 16
Shawn Chin
  • 84,080
  • 19
  • 162
  • 191
  • 19
    I wrote jug (thank for you the kind words). I'll add that the module is also well supported. If you find any bugs at all, they'll get fixed very fast (<24h, generally). – luispedro Mar 03 '11 at 19:14
  • 4
    Python multiprocessing managers can talk between multiple systems, not just within a system. See: http://eli.thegreenplace.net/2012/01/24/distributed-computing-in-python-with-multiprocessing/ – MattK May 06 '14 at 22:03
14

In the past I've used Pyro to do this quite successfully. If you turn on mobile code it will automatically send over the wire required modules the nodes don't have already. Pretty nifty.

Tomer Cohen
  • 374
  • 6
  • 13
chmullig
  • 13,006
  • 5
  • 35
  • 52
2

I have luck using SCOOP as an alternative to multiprocessing for single or multi computer use and gain the benefit of job submission for clusters as well as many other features such as nested maps and minimal code changes to get working with map().

The source is available on Github. A quick example shows just how simple implementation can be!

DMTishler
  • 501
  • 1
  • 6
  • 12
  • I now have to suggest PM's Ray Answer above. Ray is very powerful and offers a great deal of features and extensions such as shared memory objects. – DMTishler Apr 11 '20 at 03:28
2

If you are willing to pip install an open source package, you should consider Ray, which out of the Python cluster frameworks is probably the option that comes closest to the single threaded Python experience. It allows you to parallelize both functions (as tasks) and also stateful classes (as actors) and does all of the data shipping and serialization as well as exception message propagation automatically. It also allows similar flexibility to normal Python (actors can be passed around, tasks can call other tasks, there can be arbitrary data dependencies, etc.). More about that in the documentation.

As an example, this is how you would do your multiprocessing map example in Ray:

import ray
ray.init()

@ray.remote
def mapping_function(input):
    return input + 1

results = ray.get([mapping_function.remote(i) for i in range(100)])

The API is a little bit different than Python's multiprocessing API, but should be easier to use. There is a walk-through tutorial that describes how to handle data-dependencies and actors, etc.

You can install Ray with "pip install ray" and then execute the above code on a single node, or it's also easy to set up a cluster, see Cloud support and Cluster support

Disclaimer: I'm one of the Ray developers.

Philipp Moritz
  • 241
  • 3
  • 3
  • I have to second Ray. I switched from scoop finally and not only was it easy to do so with a complex program, but the features ray offer are pretty amazing(shared memory objects, passing workers 'to' workers, etc). – DMTishler Apr 11 '20 at 03:31
  • A [medium blog](https://medium.com/distributed-computing-with-ray/how-to-scale-python-multiprocessing-to-a-cluster-with-one-line-of-code-d19f242f60ff) shows a 1 line version of ray from client side (slightly simpler than your code): – D A Apr 24 '21 at 18:17