0

We're trying to push data to Kafka in a sync mode, but we're struggling on how to achieve this.

Afaik, there are two main Ruby Kafka libs ruby-kafka and rdkafka-ruby.

ruby-kafka is a ruby implementation that works great but lacks support for newer Kafka versions,

rdkafka-ruby is a C implementation but according to the maintainers it only supports async producing,

Our current plan is to push data into a Redis and then pop it to Kafka using a Python lib or a Kafka connector - but I feel there must be a better way.

What am I missing? Can someone provide source code on how to push to Kafka synchronously?

bbozo
  • 7,075
  • 3
  • 30
  • 56

1 Answers1

1

I'm one of the maintainers of rdkafka-ruby.

All Kafka clients do buffering when producing messages. Messages get produced in batches. This is the nature of how Kafka works and one of the methods it uses to be as performant as it is.

If you produce you can't know for sure the message is actually sent to the broker (with whatever ack settings you use) until you call wait on it, which waits until the message is fully flushed.

user3034074
  • 11
  • 1