5

I have recently discovered how awesome Pool.map() is for simple multiprocessing tasks, but my local machine only has two processors and I was hoping to take advantage of some remote computing resources.

Can a processing pool be constructed to use remote machines as well as the local machines in a transparent way?

What python code would I need to run on the remote machines, How would I set up the pool on the local (master) machine?

On the documentation page for multiprocessing, It says Managers are to be used for shared objects, but it's unclear to me how they should be used for a remote Pool. Nothing other than Managers seems make any reference to ports and addresses, so I'm guessing all remote activity happens through this class.

Nathan
  • 6,095
  • 10
  • 45
  • 54
  • 2
    Perhaps this is helpful: http://stackoverflow.com/questions/5181949/using-the-multiprocessing-module-for-cluster-computing – Janne Karila Apr 11 '11 at 13:04
  • Thanks's that was very helpful. I think jug (a link from that page) is just what I needed. – Nathan Apr 12 '11 at 19:49

1 Answers1

2

multiprocessing.Pool seems not designed for distributed processing -- if that's what you want to do, I suggest you check out the execnet package instead. That works over SSH pipes and sends code along to the remote Python interpreter if necessary.

Fred Foo
  • 355,277
  • 75
  • 744
  • 836