1

I'm creating an AWS Flink application in Java that stream from Iceberg and wondering if Flink has mechanism that providing possibility of restarting stream from last snapshot-id that was successfully processed, if the whole application is down. Should I sink snapshot-id to database or there is a better solution for that ?

Expected scenario:

  • Flink stream processing messages from Iceberg table (application is scalable and can run in many processes) and store results in Kinesis and another IcebergTable
  • application has died unexpectedly
  • I'm starting application again
  • it continues to processing messages without any data loss and without any manual interference

I don't use dedicated IcebergSource. Perhaps this implementation could solve it somehow. Right now I'm using FlinkSource. Both of source implementations has method for set snapshot-id, but it have to be set manually and store somewhere during processing. Is there any way to avoid it and use internal Flink mechanism ?

Netrunner
  • 31
  • 5

1 Answers1

0

Flink has a snapshotting mechanism (checkpointing and savepointing). It's used to write Flink's state to durable storage, for recovery purposes in case of an error or in situations where you want to perform an upgrade (of either your Flink cluster, or the business logic in your application). See https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/state/checkpoints/ for more details

Martijn Visser
  • 1,468
  • 1
  • 3
  • 9
  • Yes, but how to use it in Iceberg streaming ? It works well with, e. g., Kinesis. Additionally, I need to restore processing from last snapshot even if the stream was stopped for few days. – Netrunner Jun 29 '23 at 09:30
  • If Iceberg has implemented its connector in the right way (e.g. it's implemented using one of Flink's Sink interfaces), it works out of the box. – Martijn Visser Jun 30 '23 at 08:42