Stack Overflow
Stack Overflow

Questions tagged [exactly-once]

39 questions
9
votes
2 answers

Flink Kafka EXACTLY_ONCE causing KafkaException ByteArraySerializer is not an instance of Serializer

So, I'm trying to enable EXACTLY_ONCE semantic in my Flink Kafka streaming job along with checkpointing. However I am not getting it to work, so I tried downloading the test sample code from…
Olindholm
  • 340
  • 4
  • 16
3
votes
1 answer

End-to-end Exactly-once processing in Apache Flink

Apache Flink guarantee exactly once processing upon failure and recovery by resuming the job from a checkpoint, with the checkpoint being a consistent snapshot of the distributed data stream and operator state (Chandy-Lamport algorithm for…
Mazen Ezzeddine
  • 662
  • 1
  • 8
  • 28
2
votes
0 answers

Kafka Exactly-Once and compression

I think I understood that idempotence means "each message is written once" in the log, but when talking about EOS (Exactly-Once Delivery Semantic) also the consumer plays a role and enters in the end-to-end guarantees. So, as stated here (for…
Antonio Macrì
  • 41
  • 1
  • 3
2
votes
0 answers

When is it safe to retry Redis command on error when using Lettuce?

I'm using Lettuce Redis client for JVM to build a queue backed by a Redis List. Ideally it would behave like an in-memory queue but since the network interaction is involved this is not possible. There is a section on error handling in Lettuce docs…
epsylon
  • 357
  • 3
  • 13
2
votes
1 answer

How to achieve exactly-once write guaranty with foreachBatch sink in Spark Structured Streaming

From the docs: By default, foreachBatch provides only at-least-once write guarantees. However, you can use the batchId provided to the function as way to deduplicate the output and get an exactly-once guarantee. Does this mean that unless we go…
Learnis
  • 526
  • 5
  • 25
2
votes
0 answers

Flink two phase commit for map function to implement exactly-once semantics

Background: We have a Flink pipeline which consists of multiple sources, multiple sinks and multiple operators along the pipeline which also update databases. For the sake of the question and to make it simpler let's assume we have a pipeline which…
Slava Shpitalny
  • 3,965
  • 2
  • 15
  • 22
1
vote
0 answers

How to build flink kafka producer with exactly once behavior?

I have a simple flink application which consumes alerts from one topic(kafka) and publishes to a new topic(kafka). I have set the exactly once guarantee for data sink. But, my consumer does not consume that data when I set such guarantees over the…
Chaitanya Kulkarni
  • 23
  • 6
1
vote
1 answer

Kafka distributed connect produce duplicated message

Operation Environment Three servers Three Kafka broker, connect, schema-registry (confluent-7.1.0) One ftp connector for test (3 tasks) Problem Connect produce duplicated message. However, I hope that the ftp connector to issue one message per…
Seongmin Heo
  • 13
  • 4
1
vote
1 answer

How can I process a newly uploaded object in Google Cloud Storage exactly once?

I would like to receive files into a Google Cloud Storage bucket and have a Python job run exactly once for each file. I would like many such Python jobs to be running concurrently, in order to process many files in parallel, but each file should…
jl6
  • 6,110
  • 7
  • 35
  • 65
1
vote
0 answers

Exactly once in kafka streams- not working

I am testing exactly once in kafka streams by shutting down multiple brokers. But when i restart the brokers same message is getting produced multiple times on outbound topic. I am using confluent version 6.1.0 Setting processing guarantee to…
samik
  • 93
  • 7
1
vote
0 answers

In kafka stream topology with 'exactly_once' processing guarantee, message lost on exception

I have a requirement where I need to process messages from Kafka without losing any message and also need to maintain the message order. Therefore, I used transactions and enabled 'exactly_once' processing guarantee in my Kafka streams topology. …
Gul
  • 21
  • 1
1
vote
1 answer

How to achieve exactly-once in Cassandra when stream processing?

I have a Cassandra table which looks like this CREATE TABLE tmp.inventory ( t_id text, is_available boolean, modified_at bigint, price double, available_units bigint, PRIMARY KEY(t_id, modified_at) ) WITH CLUSTERING ORDER BY…
Sivaprasanna Sethuraman
  • 4,014
  • 5
  • 31
  • 60
1
vote
1 answer

How Apache Kafka Exactly Once transaction id impact on the new fetch request producer fencing approach

In earlier versions of Kafka exactly-once semantics static mapping should have between transaction id and topic partitions during consumer group mismatch there are chances that transaction id gets different topic partition. To avoid such a scenario…
sujin
  • 2,813
  • 2
  • 21
  • 33
1
vote
0 answers

Kafka Streams exactly-once re-balance aggregation state data loss

Running 3 Kafka Streams instances with exactly-once, but experiencing loss of data when restarting one of the streams instances (the other 2 doing re-balance). If I restart the instance quickly (within session.timeout.ms), without the other 2 doing…
erikdjerf
  • 11
  • 1
1
vote
1 answer

Can kafka idempotent producer ensure exactly once with multiple partitions

I'm just a newbiee on Kafka and learnt a little about idempotence of kafka producer. As my understanding, when a producer sends a message to a broker and the broker need to send back an ACK to the producer to tell it that the message has been…
Yves
  • 11,597
  • 17
  • 83
  • 180
1
2 3