Parallel For-Each vs Scatter Gather in mule

Question

I have multiple records :

{
 "item_id":1",
 key1: "data1"
}
{
  item_id:2
 key1: "data1"
}
{
  item_id:2
 key1: "data1"
}
{
  item_id:1
 key1: "data1"
}

I do not want to process them sequentially.There can be more than 200 records. Should I process them using for-each parallel or scatter-gather. Which approach would be best as per my requirement. I do not need the accumulated response, but if there is some exception while processing (hit an api for each record based on an if condition) any one of the record,processing of other records must remain unaffected.

I would opt to use what George have provided to push it out to VM queue. However, if your purpose for multithreading is not to increase performance but instead to avoid throwing an error when one of the records fail, I would suggest to do this on a normal for-each with try/catch scope. This is much simpler and easy to troubleshoot than over architecting your needs. — oim, Jan 16 '20 at 16:21
@oim I want to increase the performance, but I haven't used the VM connector before. Can you share any example ? — HMT, Jan 16 '20 at 16:29

score 2 · Answer 1 · answered Jan 16 '20 at 14:26

2

Why not then use the VM module, break the collection into its individual records and push them to a VM queue? Then have another flow with a VM listener picking up the individual records (in parallel) and processing them. Here's more details: https://docs.mulesoft.com/mule-runtime/4.2/reliability-patterns

answered Jan 16 '20 at 14:26

George

2,758
12
16

Hi , Thanks for answering my question. I just need to check the fields of a record and hit an api for each record and I have to do this for all the records. Do you still think using a VM connector will be a better option ? If possible can you share a sample flow to illustrate the working of the VM connector – HMT Jan 16 '20 at 16:10
Yes I do the VM will be better, especially since you don't care about responses. I will share something later tonight. – George Jan 16 '20 at 16:29
Thanks, will wait for your response. – HMT Jan 16 '20 at 16:30
1

Here's a link to my google drive that illustrates a solution to your issue where you can parallel process the records with the VM module: https://drive.google.com/open?id=1uUqNqjwXkJYEh1SO6poD7KODGDcKTORZ – George Jan 17 '20 at 01:23
Hi , Can you please give me access ? – HMT Jan 17 '20 at 04:04

score 2 · Answer 2 · answered Jan 16 '20 at 17:29

2

Scatter-gather is meant for cases where you have a static number of routes. Imagine one route to send to the HR system and another to the Accounting system.

For processing a variable number of records you should use parallel for-each.

answered Jan 16 '20 at 17:29

aled

21,330
3
27
34

score 1 · Accepted Answer · answered Sep 19 '20 at 18:37

1

Use the foreach async or parallel or jms pattern. Scater-gather receives one payload for all thread and you won't be able to cycle

answered Sep 19 '20 at 18:37

EvgenJin

206
2
7

Parallel For-Each vs Scatter Gather in mule

3 Answers3