Stack Overflow
Stack Overflow

Questions tagged [flink-batch]

158 questions
6
votes
0 answers

RuntimeError: java.lang.UnsupportedOperationException: A serializer has already been registered for the state; re-registration is not allowed

I am using pyflink 1.17.1 and i am getting this error "RuntimeError: java.lang.UnsupportedOperationException: A serializer has already been registered for the state; re-registration is not allowed". Need your help with this. when i try to sink data…
Imran Munir
  • 91
  • 5
3
votes
1 answer

Apache Flink JobListener does not work expected

I wrote a flink batch job in flink 1.11.1. After job finishes successfull y, I want to do something like calling a http service. I added a simple job listener to hook job status. The problem is when when kafka sink operator throws a error, job…
mstzn
  • 2,881
  • 3
  • 25
  • 37
3
votes
2 answers

Periodically refreshing static data in Apache Flink?

I have an application that receives much of its input from a stream, but some of its data comes from both a RDBMS and also a series of static files. The stream will continuously emit events so the flink job will never end, but how do you…
mransley
  • 631
  • 7
  • 16
2
votes
2 answers

Flink evenly distributes the required slots across all Task Managers

I have two different tasks in my flink job Task1 & Task2. T1 needs 10 Parrelisms and T2 needs 2 parrallism, my cluster has two Taskmanager(TM) and each of my task managers have 8 slots. Task1 runs with default task groups and for Task 2 I m setting…
scoder
  • 2,451
  • 4
  • 30
  • 70
2
votes
1 answer

Flink batch mode does not sort by event timestamp

I have a custom flink Source, and I have a SerializableTimestampAssigner that assigns event timestamps to records emitted by the source. The source may emit records out of order because of the nature of the underlying data storage, however with…
sqd
  • 1,485
  • 13
  • 23
2
votes
1 answer

Apache Flink - Matching multiple patterns

We have a requirement where multiple events will be matched through the given pattern. Pattern 1 followed by within 10 mins Pattern 2 followed by within 10 mins Pattern 3. Pattern 1: 10 failed logins from the same username and same device within 10…
JDForLife
  • 91
  • 2
  • 10
2
votes
1 answer

Flink Elasticsearch sink success handler

I use Flink Elasticsearch sink to bulk insert the records to ES. I want to do an operation after the record is successfully synced to Elasticsearch. There is a failureHandler by which we can retry failures. Is there a successHandler in flink…
Vicky
  • 2,999
  • 2
  • 21
  • 36
2
votes
1 answer

I want to write ORC file using Flink's Streaming File Sink but it doesn’t write files correctly

I am reading data from Kafka and trying to write it to the HDFS file system in ORC format. I have used the below link reference from their official website. But I can see that Flink write exact same content for all data and make so many files and…
patel akash
  • 75
  • 8
2
votes
0 answers

Apache Flink Resource Manager (App Master) fails allocating new Task Managers after the first one is killed after idle timeout

I am using Apache Flink 1.9.1 on Apache Mesos 1.8.1 using more or less standard settings for both on a 3 nodes cluster. The first node (192.168.10.11) runs the Mesos master and the Flink App Master. Both other nodes (192.168.10.13, 192.168.10.14)…
Jerome Kehrli
  • 21
  • 4
1
vote
0 answers

Flink Gauge - Gauge failed to update metrics to mutliple topics

Using Fling Gauge metrics Flink 1.17.x Added multiple Gauge metrics and updating the metrics on condition bases as shown in the code. Publishing messages to 2 different topics, expecting each topic should have an metrics Query Flink metric REST API…
Hareesh
  • 41
  • 4
1
vote
0 answers

The close() cancle() function of Flink RichParallelSourceFunction does not work

The Flink Job is running in the workflow on the GCP DataProc cluster. A timeout is set for the workflow task, and the Dag is forcibly terminated after the set time. At this time, the close and cancel functions of Flink RichParallelSourceFunction do…
염경훈
  • 73
  • 1
  • 6
1
vote
1 answer

In Flink, is there a way to restrict processing in strict event-time ordering?

I'm working with IOT and my devices are sending data to several Pulsar topics at a rate of around 5 per sec. I have days worth of messages on those topics and I need to process them and save the output in a database. In my Flink code, after some…
André Casimiro
  • 133
  • 10
1
vote
0 answers

How to test flink jobs end to end for timer behavior

It is somewhat similar to this question: testing flink jobs with MiniCluster to trigger the timer using processing time I have written a Flink jobs graph and I defined a timer in KeyedProcessStream. Now I am trying to write an end to end test and…
Xuhui Ding
  • 11
  • 3
1
vote
0 answers

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline expired before operation could complete

In the dataproc cluster I am submitting the Flink job which reads the data from Pubsub. When I am executing the flink job I am getting the below error. We are using the flink version 1.12.5 and dataproc image version is 2.0.45-rocky8 The program…
Nagesh B Viswanadham
  • 27
  • 3
1
vote
1 answer

Flink KafkaSource read all messages from the topic

My goal is reading all messages from Kafka topic using Flink KafkaSource. I tried to execute with batch and streaming modes. The problem is the following : I have to use a Sink which contains bug when I set env.setParallelism higher than 2. Thus, I…
Sergey Gazaryan
  • 1,013
  • 1
  • 9
  • 25
1
2 3
…
10 11