4

I'm working on implementing a use case wherein different physical devices are sending events, and due to network/power issues, there can be a delay in receiving events at flink source. One of the operators within the flink job is the Pattern operator, and there are certain patterns which are time sensitive, so I'm using Event time characteristic. But the problem comes when there are unpredictable delays in events from a particular device(s), which causes those events to be dropped (as I cannot really define a static bound to allow for lateness).

Since I'm using a KeyedStream, keyed on the source device ID, is there a way to allow each CEP operator instance (one per key) to progress its time based on the event time in the corresponding stream partition. Or in other words, is there a way to generate watermarks per partition in a KeyedStream?

shailesh
  • 73
  • 4

1 Answers1

3

Flink currently does not support per key watermarks. The watermark is global.

Till Rohrmann
  • 13,148
  • 1
  • 25
  • 51
  • How do you feel about the RichFlatMapFunction approach to achieve this functionality as proposed here? http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Maintaining-watermarks-per-key-instead-of-per-operator-instance-td7288.html – austin_ce Nov 06 '18 at 16:29
  • You would have to implement it in user code, meaning that the user code generates the "watermarks" and processes them. This would, however, entail that you cannot use it with Flink's build in windowing mechanism, for example. If you need windowing with a per key watermark, then you would also have to implement the windowing mechanism yourself. – Till Rohrmann Jun 09 '20 at 06:45