Stack Overflow
Stack Overflow

Questions tagged [kafkajs]

KafkaJS is an open-source Node.js client library for Apache Kafka

KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.

199 questions
8
votes
1 answer

Waiting for leadership elections in KafkaJS

The Situation I am using kafkajs to write to some dynamically generated kafka topics. I am finding writing to those topics immediately after registering my producer will regularly cause an error: There is no leader for this topic-partition as we are…
slifty
  • 13,062
  • 13
  • 71
  • 109
7
votes
0 answers

Kafkajs consume and produce batches problems

I want to use kafkajs in my Node.js project. Let me show my code. Producer: const producer = kafka.producer(); await producer.connect(); const items = await getItems(); // getting somehow 5k items to produce await producer.send({ topic: "items", …
Владимир Говорухин
  • 393
  • 1
  • 2
  • 11
7
votes
1 answer

Failed to connect to seed broker with kafkajs

I am trying to use kafkajs in order to create a kafka consumer. However, I get already an error when connecting to kafka: "[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Closed connection" This is the code I am…
Rene B.
  • 6,557
  • 7
  • 46
  • 72
6
votes
1 answer

Kafka There is no leader for this topic-partition as we are in the middle of a leadership election

just starting learning Kafka. Im trying to setup a small kafka cluster including 2 brokers. I was successfull in sending messages to my topic when both brokers are up. I want to test the behavior of my cluster when one of 2 brokers goes done. I…
roamsmart
  • 85
  • 1
  • 2
  • 8
6
votes
1 answer

Kafka consumer manual offset commit

In one of my use-case consist of consuming the data, do some operations and produce it to new topic. I'm using https://www.npmjs.com/package/kafkajs npm library. I would like to commit the offset manually after successful operations to avoid any…
Parveen Kumar
  • 397
  • 1
  • 5
  • 12
5
votes
1 answer

Kafkajs - `The group is rebalancing, so a rejoin is needed` error causes message to be consumed more than once

I have an edge in Kafkajs consumer, where at times I get a rebalancing error: The group is rebalancing, so a rejoin is needed [Connection] Response Heartbeat(key: 12, version: 3) The group is rebalancing, so a rejoin is needed [Runner] The group is…
David Faizulaev
  • 4,651
  • 21
  • 74
  • 124
5
votes
0 answers

LibrdKafkaError: Broker: Unknown member after running around about 2 hours randomly

Right now, i want to implement node-rdkafka into our service, but i faced this error many times Broker: Unknown member. The same issue on github was https://github.com/confluentinc/confluent-kafka-dotnet/issues/1464. they say our consumer using same…
Ibid Athoillah
  • 95
  • 2
  • 10
4
votes
0 answers

How to pause KafkaJS consumer and retry after sometime

I'm using KafkaJS ( https://kafka.js.org ) to connect to a Kafka cluster. My consumers are responsible for processing tasks which can fail at some time and succeed after waiting for a duration (for example, 1 hour). Before this duration, any attempt…
Kha Nguyễn
  • 81
  • 5
3
votes
2 answers

Node convert Decimal to Bytes - Avro Schema Kafka / Kafkajs

I am using http://kafka.js.org/ to produce messages to Kafka, one of the schemas I am working on is a decimal byte type: {"type": "bytes", "logicalType": "decimal", "precision": 11, "scale": 2} I am looking for some guidance on how I can convert a…
danik
  • 191
  • 3
  • 13
3
votes
2 answers

Kafkajs - get statistics (lag)

In our nest.js application we use kafkajs client for kafka. We need to get chance monitor statistic. One of metrics is lag. Trying to figure out if kafkajs provides any and nothing interesting. (The most interesting thing in payload are: timestamp,…
Sergii
  • 7,044
  • 14
  • 58
  • 116
3
votes
1 answer

Nestjs kafka implementation

I've read nestjs microservice and kafka documentation but I couldn't figure out some of it. I'll be so thankful if you can help me out. So as the docs says I have to create a microService in main.ts file as follows: const app = await…
Nima Soufiloo
  • 224
  • 4
  • 18
3
votes
1 answer

Kafka Producer: Disconnect after sending message vs keeping connection open

I've not been able to find an answer to this in the kafkajs docs or from skimming through the official Apache Kafka design docs, but in their producer examples, the producer disconnects after sending the messages. However, that could be because it's…
Honinbo Shusaku
  • 1,411
  • 2
  • 27
  • 45
3
votes
1 answer

How to make consumer subscribe the dynamically created topics automatically in nodejs

There are several similar questions available on stackoverflow itself but still I am asking again because I want solution in NodeJS. The scenario is - (I am using kafkaJS library) I have two topics topic-A and topic-B, so I subscribe to /topic-.*/,…
P N Jain
  • 302
  • 1
  • 13
3
votes
1 answer

NestJs EventBus duplicates event at EventHandler

I am trying out Event-Sourcing and CQRS using NestJs using Kafka as the Event Store. The application is a small and simple one having 2 parts, customer and order. You first create a customer with some initial balance and then using the customer id…
Ashniu123
  • 347
  • 4
  • 13
3
votes
0 answers

how to make autocommit false work in kafkajs

scenario: im setting autocommit false, producing 12 messages consuming them.. (say from offset 100) shutting down consumer start a new consumer in that stage, I expect to the 2nd consumer to read all messages again starting offset 100 (because no…
yuria
  • 541
  • 2
  • 7
  • 22
1
2 3
…
13 14