Stack Overflow
Stack Overflow

Questions tagged [karafka]

9 questions
3
votes
0 answers

Kafka/Ruby on Rails - I don't understand the logs generated after running `bundle exec karafka server`

I am a newbie in the microservices world. I am migrating a monolith web application to a micro-services one. I got a problem with connecting the Ruby on Rails web application in charge of authentication with KAFKA. The Ruby on Rails itself don't…
Mael Fosso
  • 360
  • 6
  • 14
1
vote
1 answer

Why Karafka Server Throw error query watermark offset?

I'm using this gem for my Ruby on Rails Apps to produce and consume message from kafka server. https://karafka.io/docs/ and this is my karafka.rb file # frozen_string_literal: true class KarafkaApp < Karafka::App setup do |config| …
ramayeah
  • 39
  • 12
1
vote
0 answers

How do you process a sliding window using karafka?

I have a stream of messages in a Kafka topic. For each message, I need to examine previous messages that occur within 10 seconds of the current message. I assume that I can keep a buffer with the current window in my consumer. How do I reconstruct…
Vinay
  • 469
  • 3
  • 14
0
votes
1 answer

How to produce data to Kafka from Ruby using sync mode with ack=1 or ack=all

We're trying to push data to Kafka in a sync mode, but we're struggling on how to achieve this. Afaik, there are two main Ruby Kafka libs ruby-kafka and rdkafka-ruby. ruby-kafka is a ruby implementation that works great but lacks support for newer…
bbozo
  • 7,075
  • 3
  • 30
  • 56
0
votes
1 answer

Is Karafka compatible with applications other than Rails? i.e Sinatra

I have a service that publishes a Kafka event whenever a user attribute is updated, now to consume and process this event I have a gem that uses Karafa. Every application that boots the gem will be able to process that event (if possible). Does…
Ushay
  • 3
  • 1
0
votes
0 answers

Dockerized rails/kafka apllication problem

I have a dockerized application that uses Kafka/Rails and I'm having problems setting it up. My main problem at this moment is that the application seems to be connecting to the wrong port when I try to send a message on my local terminal. When I…
Rubens Cordeiro
  • 1
0
votes
0 answers

kafka is designed to have many producers

We have been testing kafka (cloud-based) to implement a data transmission system of many producers (approximately 27000 linux machines), a consumer (spring kafka listener) and a topic with 10 partitions, the problem is that when 9500 producers…
GreenLemonade
  • 13
  • 3
0
votes
1 answer

Long delays between processing of two consecutive kafka batches (using ruby/karafka consumer)

I am using karafka to read from a topic, and call an external service. Each call to external service takes roughly 300ms. And with 3 consumers (3 pods in the k8s) running in the consumer group, I expect to achieve 10 events per second. I see these…
Asad Iqbal
  • 3,241
  • 4
  • 32
  • 52
0
votes
1 answer

Karafka start_from_beginning not working as expected

I've created a project to help with my understanding of Kafka. It's set up as three identical Rails apps all inside Docker with Karafka configured to consume the messages - if you create a record in one, it's replicated across to the other two. I…
John Cleary
  • 957
  • 8
  • 9