2

I am working on Hazelcast jet application and I trying to join two Sources using Left, Right or Inner Join but I am stuck at below:

Here is my code:

BatchStage<Map<String,Object>> batch1= pipeline.readFrom(companyListBatchSource);
BatchStage<Map<String,Object>> batch2= pipeline.readFrom(employeeListBatchSource);

//Getting group by key
 BatchStageWithKey<Map<String,Object>, Object> jdbcGroupByKey = batch1.groupingKey(a -> a.getSource1().get(col1));
 BatchStageWithKey<Map<String,Object>, Object> fileGroupByKey = batch2.groupingKey(b -> b.getSource1().get(col2));

//trying to join but not sure what exactly is happening.
BatchStage<Entry<Object, Tuple2<List<Map<String,Object>>, List<Map<String,Object>>>>> d = jdbcGroupByKey.aggregate2(AggregateOperations.toList(),fileGroupByKey,AggregateOperations.toList());

From above code how can achieve data in BatchStage<Map<String,Object>> format? How we can apply different kind of joins here?

user3458271
  • 638
  • 12
  • 31

1 Answers1

3

Your Map<String, Object> represents a single item, let's call that entire type E. So your input is two streams of type E: companies and employees.

The result of a JOIN isn't of the type E, but, in full generality, Tuple2<List<E>, List<E>>.

From here on the details depend on the cardinalities of each side of the join. If you group by company on both sides, you should have a one-to-many association and the result type Tuple2<E, List<E>>. In this case, use AggregateOperations.pickAny() for the company stream, this will result in a single item and the result of the join will come out as Entry<Object, Tuple2<List<E>, List<E>>>. This is the equivalent of a LEFT OUTER JOIN. You get an inner join from that if you filter out all results where the list of employees is empty: joined.filter(e -> !e.getValue().f1().isEmpty())

Marko Topolnik
  • 195,646
  • 29
  • 319
  • 436
  • Thanks for the reply but this is little new to me, so you are saying if I will use it : BatchStage>, List>>>> d = jdbcGroupByKey.aggregate2(AggregateOperations.pickAny(),fileGroupByKey,AggregateOperations.toList()); then this will become left join? – user3458271 Jun 29 '21 at 09:07
  • Also how we can convert final output to BatchStage> format, as after this I have to join it with other 4 Sources also. – user3458271 Jun 29 '21 at 10:14
  • if you use `pickAny()`, then you don't get `List` in the output but just `E`. As for converting to `Map`, which is just `E` in your example, how you do that is up to your custom logic that combines the left and right objects. – Marko Topolnik Jun 29 '21 at 11:10