1

In earlier versions of Kafka exactly-once semantics static mapping should have between transaction id and topic partitions during consumer group mismatch there are chances that transaction id gets different topic partition.

To avoid such a scenario KIP-447: Producer scalability for exactly once semantics was implemented, what I was understood from the KIP-477 that the old producer fenced using the fetch offset call with the help of a new API(sendOffsetToProdcuer) so transactio.id not used for fencing.

But my doubt here is,

  1. still transactional producer expect transaction.id how should I choose this value for the latest Kafka version?

  2. transaction.id should have a static mapping with partitions, fetch offset fencing take effect only during consumer group rebalancing?

  3. Is this value invalid for the latest version?

Please help me with this I am trying to understand Kafka EoS and implement it in the production system.

sujin
  • 2,813
  • 2
  • 21
  • 33

1 Answers1

2

Since you tagged this with , I assume you are using it; the transactional.id can now be different for each instance (as it was previously required for producer-only transactions). There is no longer a need to tie the id to the group/topic/partition, and a much smaller number of producers is needed.

See https://docs.spring.io/spring-kafka/docs/current/reference/html/#exactly-once

The broker needs to be 2.5 or later.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • I am using spring-kafka but trying to understand the EOS internals. Before producer's ***beginTransaction*** step old producer fenced but in fetchOffset fencing mechanism consumer metadata send using ***sendOffsetsToTransaction*** so Where exactly producer fencing occurs in the new approach. – sujin Apr 23 '21 at 05:39
  • 1
    The KIP explains it well; if you have more questions, I suggest you ask on one of the Apache Kafka mailing lists. – Gary Russell Apr 23 '21 at 14:45