18

I have two data streams that I want to combine. The problem is that one data stream has a much higher frequency than the other and there are times where one stream is not receiving events at all. Is it possible to use the last event from the one stream and join it with the other stream on every event that is coming?

The only solution I found is using the join function, but you have to specify a common window, where you can apply the join function. This is window is not reached, when one stream is not receiving any events.

Is there a possibility to apply the join function on every event that is coming from either one stream or the other and maintain state of the last consumed event and use this event for the join function?

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
FLoppix
  • 215
  • 1
  • 2
  • 6

1 Answers1

20

There are many different approaches to combining or joining two streams in Flink, depending on requirements of each specific use case. When doing this "by hand", you want to be using Flink's ConnectedStreams with a RichCoFlatMapFunction or CoProcessFunction. Either of these will allow you to keep managed state (i.e. the last element from the infrequently updating stream), and join it with the faster stream. CoProcessFunction adds the ability to work with timers, which you should use to clear state for expired keys, if that's relevant.

There's an exercise on the Flink training site about different approaches for implementing such joins: Enrichment Joins. For a simpler example, see also the exercise about Expiring State.

Each recent release of Flink has included additional built-in join functions, so at this point it is less often necessary to roll your own. See the pages on joining with the DataStream API, joins with the Table API, and joins in SQL for more details.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • Thank you ! This is exactly what I was looking for ! – FLoppix Sep 07 '17 at 09:12
  • @DavidAnderson Oppositely, Would the example be different when joining two fast (frequent) streams? What would you change? – Alex Feb 24 '18 at 17:48
  • @Alex I think the idea is that the two streams have different frequencies rather than being synchronous. – devinbost Oct 07 '18 at 05:57
  • Hi, how this relates to Broadcast state that was added to Flink some time ago? https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html When to choose which solution? Do you know any pros and cons of each? What – Kristoff Nov 28 '19 at 20:04