Need advice on migrating from Flink DataStream Job to Flink Stateful Functions 3.1

Question

I have a working Flink job built on Flink Data Stream. I want to REWRITE the entire job based on the Flink stateful functions 3.1.

The functions of my current Flink Job are:

1. Read message from Kafka
2. Each message is in format a slice of data packets, e.g.(s for slice):
   • s-0, s-1 are for packet 0
   • s-4, s-5, s-6 are for packet 1
3. The job merges slices into several data packets and then sink packets to HBase
4. Window functions are applied to deal with disorder of slice arrival

My Objectives

• Currently I already have Flink Stateful Functions demo running on my k8s. I want to do rewrite my entire job upon on stateful functions.
• Save data into MinIO instead of HBase

My current plan

I have read the doc and got some ideas. My plans are:

• There's no need to deal with Kafka anymore, Kafka Ingress(https://nightlies.apache.org/flink/flink-statefun-docs-release-3.0/docs/io-module/apache-kafka/) handles it
• Rewrite my job based on java SDK. Merging are straightforward. But How about window functions?
• Maybe I should use persistent state with TTL to mimic window function behaviors
• Egress for MinIO is not in the list of default Flink I/O Connectors, therefore I need to write my custom Flink I/O Connector for MinIO myself, according to https://nightlies.apache.org/flink/flink-statefun-docs-release-3.0/docs/io-module/flink-connectors/
• I want to avoid Embedded module because it prevents scaling. Auto scaling is the key reason why I want to migrate to Flink stateful functions

My Questions

I don't feel confident with my plan. Is there anything wrong with my understandings/plan?

Are there any best practice I should refer to?

Update:

windows were used to assemble results

1. get a slice, inspect its metadata and know it is the last one of the packet
2. also knows the packet should contains 10 slices
3. if there are already 10 slices, merge them
4. if there are not enough slices yet, wait for sometime (e.g. 10 minutes) and then either merge or record packet errors.

I want to get rid of windows during the rewrite, but I don't know how

Answer 1

Background: Use KeyedProcessFunctions Rather than Windows to Assemble Related Events

With the DataStream API, windows are not a good building block for assembling together related events. The problem is that windows begin and end at times that are aligned to the clock, rather than being aligned to the events. So even if two related events are only a few milliseconds apart they might be assigned to different windows.

In general, it's more straightforward to implement this sort of use case with keyed process functions, and use timers as needed to deal with missing or late events.

Doing this with the Statefun API

You can use the same pattern mentioned above. The function id will play the same role as the key, and you can use a delayed message instead of a timer:

• as each slice arrives, add it to the packet that's being assembled
• if it is the first slice, send a delayed message that will act as a timeout
• when all the slices have arrived, merge them and send the packet
• if the delayed message arrives before the packet is complete, do whatever is appropriate (e.g., go ahead and send the partial packet)