Using Spring batch, I want my steps to be distributed across nodes and get them executed for a given job. I have a usecase where a job has multiple steps and each step can run in multiple nodes where the app is hosted. Has anybody tried this ? Any ideas on the same would be highly appreciated!
Asked
Active
Viewed 2,876 times
8
-
Take a look at my answer here: http://stackoverflow.com/questions/20323229/difference-between-spring-batch-remote-chunking-and-remote-partitioning/20342308#20342308 – Michael Minella Dec 15 '15 at 15:47
1 Answers
6
There are two approaches:
Remote chunking - you read data on master node and process/write them on slaves
Remote partitioning - you slice your data set into partitions and read/process/write your partitions in remote nodes. So master is just coordinating and deciding how to slice partitions.
I wrote a book about Enterprise Spring, and I created examples of both approaches. These are hosted on Github. Look into examples 0939 and 0940. Unfortunately all the comments how to run them manually are in the book only. Hopefully you will be able to figure that out from integration tests.
Pre-requirement is to have messaging middleware (e.g. ActiveMQ or HornetQ) available for master-slave communication and it is also using Spring Integration to facilitate this communication.
luboskrnac
- 23,973
- 10
- 81
- 92
-
can we use REST calls for communication between master and slave instead of JMS? – Rams Apr 07 '18 at 08:38
-
not sure about this. I recommend searching for/opening separate SO question. – luboskrnac Apr 09 '18 at 12:14