Flink: Append an event to the end of finite DataStream

Question

Assuming there is a finite DataStream (from a database source, for example) with events

a1, a2, ..., an.

How to append one more event b to this stream to get

a1, a2, ..., an, b

(i.e. output the added event after all original events, preserving the original ordering)?

I know that all finite streams emit the MAX_WATERMARK after all events. So, is there a way to "catch" this watermark and output the additional event after it?

(Unfortunately, .union()ing the original DataStream with another DataStream consisting of a single event (with timestamp set to Long.MaxValue) and then sorting the united stream using this answer did not work.)

Do you know the count ahead of time? Also, if it is a finite set, why can't you use the DataSet API instead of the DataStream? — austin_ce, Feb 14 '19 at 15:18

score 2 · Accepted Answer · answered Feb 14 '19 at 18:55

2

Maybe I'm missing something, but it seems like you could simply have a ProcessFunction with an event time timer set for somewhere in the distant future, so that it only fires when the MAX_WATERMARK arrives. And then in the onTimer method, emit that special event if the currentWatermark is MAX_WATERMARK.

answered Feb 14 '19 at 18:55

David Anderson

39,434
4
33
60

Thanks! It worked for me. I didn't know that onTimer fires only when watermark arrives. – trolley813 Feb 15 '19 at 06:24

score 0 · Answer 2 · answered Feb 14 '19 at 22:29

0

Another approach might be to 'wrap' the original data source in another data source, which emits a final element when the delegate object's run() method returns. You'd need to be careful to call through to all of the delegate methods, of course.

answered Feb 14 '19 at 22:29

kkrugler

8,145
6
24
18

Flink: Append an event to the end of finite DataStream

2 Answers2