Stack Overflow
Stack Overflow

Questions tagged [flink-state]

11 questions
0
votes
0 answers

k8s persistent volume (PV) data gets deleted after pod recreation with all ReclaimPolicy : Retain

What happened? I created one PV(persistent volume) using .yaml file 10G on aws efs /data/flink/state created PVC of 10G and here we are using dynamic provisioning (storage-class) to mount PV. created pod deployment using deploy.yaml file. here we…
Abhishek G Joshi
  • 11
  • 2
0
votes
0 answers

Apache Flink + Openstack Swift

I came across this https://mvnrepository.com/artifact/org.apache.flink/flink-swift-fs-hadoop repo. Does Apache Flink still support this OS Swift plugin File System or not? It's not listed on the current site but has been on the previous…
tosi
  • 1,873
  • 2
  • 18
  • 38
0
votes
1 answer

Why does Flink ValueState.value() sometimes erroneously return null?

I am encountering an error in my Flink app where calling myValueState.value(), inside a KeyedProcessFunction, sometimes returns null despite the fact that the logic in the code should guarantee that the object returned by .value() is not null. These…
r_g_s_
  • 224
  • 1
  • 8
0
votes
0 answers

Flink-statefun dynamic function discovery and fan-out execution

What would be a scalable way to dynamically register and call remote-statefun? I know I can register statefun while submitting a flink job but it's not ideal to submit a new build per new function. I wonder why would flink need to know about remote…
Shunya One
  • 1
  • 1
0
votes
1 answer

In Flink is it possible to use state with a non keyed stream?

Lets assume that I have an input DataStream and want to implement some functionality that requires "memory" so I need ProcessFunction that gives me access to state. Is it possible to do it straight to the DataStream or the only way is to keyBy the…
kmylonas
  • 11
  • 3
0
votes
1 answer

Access to Subtask Metrics in a KeyedStream

I want to load a large amount of data into Flinks state backend (RocksDB) and process events using this data (for example in a CoProcessFunction). The data is much larger than the memory of a TaskManager. The data blocks for a key can be very large,…
fbc
  • 3
  • 2
0
votes
1 answer

Apache Flink CoFlatmap function member value is reset on failure

I have a RichCoFlatMapFunction in flink which connects two streams. This class has a constructor which initializes a member of Type1 It's dynamic configuration which we use in this flat map operator's flatmap2 function, to send out final data with…
Nikhil Pandit
  • 184
  • 5
0
votes
1 answer

With a CheckPointed function in Flink, does the user call initializeState and snapshotState or is it handled behind the scenes

I am following an example here: https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/StatefulSequenceSource.java I am trying to build a source using a jdbc connection which…
Daniel Allen
  • 3
  • 2
0
votes
1 answer

Is Flink KeyedStream sum() function stateful?

Let's write a simple wordcount job DataStream> counts = text.flatMap(new Tokenizer()) .keyBy(value -> value.f0) .sum(1); (source and other details are irrilevant) Suppose into the…
Vin
  • 701
  • 1
  • 9
  • 30
0
votes
1 answer

Flink schema evolution not working for broadcast state

I am using broadcast state pattern in flink where I am trying to connect the two streams, one stream being the control stream of Rules and other stream being stream of Integers(for dummy play purpose). I have a following Rule class public class Rule…
voidMainReturn
  • 3,339
  • 6
  • 38
  • 66
-2
votes
2 answers

Flink for abandoned workflow

We want to remind users to complete their workflow. These workflow events look like 'Workflow started', 'progressed stage 1', 'progressed stage 2',... 'Workflow ended' and they flow through Kafka. Each event has a unique identifier to identify a…
siliconsenthil
  • 1,380
  • 1
  • 14
  • 25