JVM (embarrasingly) parallel processing libraries/tools

Question

I am looking for something that will make it easy to run (correctly coded) embarrassingly parallel JVM code on a cluster (so that I can use Clojure + Incanter).

I have used Parallel Python in the past to do this. We have a new PBS cluster and our admin will soon set up IPython nodes that use PBS as the backend. Both of these systems make it almost a no-brainer to run certain types of code in a cluster.

I made the mistake of using Hadoop in the past (Hadoop is just not suited to the kind of data that I use) - the latency made even small runs execute for 1-2 minutes.

Is JPPF or Gridgain better for what I need? Does anyone here have any experience with either? Is there anything else you can recommend?

score 3 · Answer 1 · answered Jul 19 '10 at 19:07

3

Check out cascalog - http://github.com/nathanmarz/cascalog

answered Jul 19 '10 at 19:07

simon-says

31
1

score 2 · Accepted Answer · answered Mar 27 '10 at 19:30

2

Clojure is reported to work on Terracotta, subject to some patching.

answered Mar 27 '10 at 19:30

Stuart Sierra

10,837
2
29
35

score 1 · Answer 3 · edited Dec 16 '16 at 07:02

1

Look at Skandium

Edit :

Above link is no more live, so adding github link

https://github.com/mleyton/Skandium

edited Dec 16 '16 at 07:02

Dhananjay

3,903
2
29
44

answered Mar 24 '10 at 16:25

Aravind Yarram

78,777
46
231
327

Looks promising, but GPL is a pain in the ass. – Robert Harvey Mar 27 '10 at 19:44

score 0 · Answer 4 · answered Dec 31 '10 at 16:57

0

I suggest you look at Skandium, alternative licenses to GPL can be negotiated with the developers upon request.

answered Dec 31 '10 at 16:57

user559370

28
4

JVM (embarrasingly) parallel processing libraries/tools

4 Answers4