Stack Overflow
Stack Overflow

Questions tagged [ruby-kafka]

13 questions
8
votes
1 answer

Difference between Kafka async and Kafka sync acks=0?

Based on my readings, Kafka with acks=0 basically just pushes the message into the producer buffer. It does not wait for any sort of acks. With that, I'm wondering how it is different from async producer? How does acks affect the async producer?
La Reina
  • 81
  • 1
  • 3
5
votes
1 answer

could not connect to any of the seed brokers

I am trying to use ruby-kafka gem to send messages but i am getting the error Could not connect to any of the seed brokers kafka::ConnectionError: Could not connect to any of the seed brokers: - kafka:broker1: Connection error EOFError: EOFError -…
john simons
  • 81
  • 1
  • 7
4
votes
1 answer

Ruby Kafka Uncaught exception: Failed to find group coordinator

I use Apache Kafka as Docker container https://hub.docker.com/r/wurstmeister/kafka/ I'm able to successfully connect to Kafka from my Java application with Spring Kafka. But when I try to connect to Kafka from the Ruby application via Ruby Kafka I…
alexanoid
  • 24,051
  • 54
  • 210
  • 410
3
votes
0 answers

How to efficiently use Kafka in place of Sidekiq for background Job in Rails?

Currently, I am to using sidekiq for background jobs in rails app. Now I want to use Kafka for the same using consumer-producer mechanism. For that I am using ruby-kafka and producing messages asynchronously. My approach: In the perform method of…
Shiv
  • 31
  • 5
2
votes
2 answers

Kafka producer says "unknown_topic_or_partition"

I've been trying to make kafka-docker work for a few days now and I don't know what I'm doing wrong. Right now, I can't access any topic with my ruby-kafka client because the node "doesn't exist". This is my docker-compose.yml file: version:…
J2RGEZ
  • 47
  • 1
  • 1
  • 5
1
vote
1 answer

Wait until value appears in the hash

Recently, I was given a task to build a REST API request that is responsible for sending messages to Kafka's inbound channel and then waiting for output from an outbound one. All went well until I encountered the issue related to waiting for this…
ThreadedStream
  • 23
  • 6
1
vote
0 answers

Ruby-kafka Read all messages topic and exit

I need to read all messages form Kafka topic then process and exit (no need to run like a daemon forever) . I have written a code like below , it serves the purpose if messages available in topic , if the topic is empty ( or no new message for…
Shan Sunny
  • 92
  • 10
0
votes
1 answer

How to produce data to Kafka from Ruby using sync mode with ack=1 or ack=all

We're trying to push data to Kafka in a sync mode, but we're struggling on how to achieve this. Afaik, there are two main Ruby Kafka libs ruby-kafka and rdkafka-ruby. ruby-kafka is a ruby implementation that works great but lacks support for newer…
bbozo
  • 7,075
  • 3
  • 30
  • 56
0
votes
0 answers

what is connection_builder in ruby-kafka gem?

I was trying to implement the functionalities of the gem ruby-kafka. When trying out one of them, I came across the connection_builder. I am finding it hard to understand more about the connection_builder.I dont know what needs to be passed as a…
Spoorthi kg
  • 9
  • 2
0
votes
1 answer

Long delays between processing of two consecutive kafka batches (using ruby/karafka consumer)

I am using karafka to read from a topic, and call an external service. Each call to external service takes roughly 300ms. And with 3 consumers (3 pods in the k8s) running in the consumer group, I expect to achieve 10 events per second. I see these…
Asad Iqbal
  • 3,241
  • 4
  • 32
  • 52
0
votes
1 answer

Racecar not consuming messages written by DeliveryBoy (kafka-ruby)

I have a question geared towards using zendesk's deliveryboy/racecar wrappers in tandem for processing Kafka events. I consume events using Racecar but send events back for reprocessing, at a delayed interval, into the same topic via DeliveryBoy…
mrlevitas
  • 80
  • 7
0
votes
1 answer

How to check to see if Ruby-Kafka retries works?

In the documentation it is mentioned that producer retries to send the message to the queue based on max_retries. So I shutdown the Kafka and then tried my producer. I get this error Fetching cluster metadata from…
Mr H
  • 5,254
  • 3
  • 38
  • 43
0
votes
1 answer

ruby-kafka: is it possible to publish to two kafka instances at the same time

Current flow of the project that I'm working on involves pushing to a local kafka using ruby-kafka gem. Now the need arose to add producer for the remote kafka, and duplicate also messages there. And I'm looking for a better way, than calling…
Elmor
  • 4,775
  • 6
  • 38
  • 70