Questions tagged [kafka-node]
32 questions
3
votes
1 answer
How to make consumer subscribe the dynamically created topics automatically in nodejs
There are several similar questions available on stackoverflow itself but still I am asking again because I want solution in NodeJS.
The scenario is -
(I am using kafkaJS library)
I have two topics topic-A and topic-B, so I subscribe to /topic-.*/,…
P N Jain
- 302
- 1
- 13
2
votes
0 answers
Apache kafka frequently connect and close connection in nodejs
In my node application I am using kafka with kafka-node npm package and I have custom connection check flow like below,
client.on("close", () => {
console.log("kafka close");
});
client.on("connect", function () {
console.log("kafka…
Koperumsozhan VR
- 55
- 7
2
votes
1 answer
How do you use a nodejs Lambda in AWS as a producer to send messages to MSK topic without creating EC2 client server?
I am trying to create a Lambda in AWS that serves as a producer to an MSK topic. All the AWS docs say to create a new EC2 instance, but as my Lambda is in the same VPC I feel like this should work. I am very new to this and I notice my log statement…
drumurr
- 33
- 1
- 4
2
votes
1 answer
Create multiple partitions in kafka topic and publish message to all of them using kafka-node
I am new to kafka and implementing it in nodeJS using kafka-node. I want to create 3 partitions in one topic and publish messages to all the topics at the same time. I tried following code, but here only one partition is creating and all messages…
Madhur Jain
- 23
- 6
1
vote
0 answers
Unable to delete topics in kafka-node
I want to delete all the topics present in kafka using kafka-node but admin.deleteTopic() or admin.deleteTopic() , both are not working giving it is not defined.
I'm expecting to a js file by running that I can delete all the kafka-topics present…
Saurav Thakur
- 13
- 3
1
vote
0 answers
How to write the Unit test cases for Node JS application which uses kafka-node and HTTP call in a single file?
Below is my sample code. Could someone please tell me how to start the Unit test code for testing and coverage?
const kafka = require('kafka-node');
const express = require('express');
const app =…
Sathyamoorthy
- 11
- 2
1
vote
1 answer
Kafka Consumer not consuming messages from all partitions
I am noticing something weird happening with my system. So, I am using Kafka to send and receive messages between different systems. I have around 6 or 7 topics each with 10 partitions.
I have an external system that is sending messages on my Kafka…
JThind
- 13
- 2
1
vote
1 answer
Is there any way to consume ksqldb table (push queries) in Node.js?
I have test Kafka cluster with topic 'topic1' (with schema in Schema Registry).
I created stream from this topic:
CREATE STREAM topic1_basic_stream
WITH (KAFKA_TOPIC='topic1', VALUE_FORMAT='AVRO', TIMESTAMP='triggertime',…
deeplay
- 376
- 3
- 20
1
vote
0 answers
Kafka-node keeps disconnecting and reconnecting with "Kafka producer error: NestedError refreshBrokenMetadata"
The nodejs server is launched with pm2 as a one instance fork.
My kafka options:
export const kafkaTopics = ['chat'];
export const kafkaOptions: ConsumerGroupOptions = {
kafkaHost,
groupId: 'test-chat',
batch: undefined,
sessionTimeout:…
Kingusha23
- 37
- 2
- 7
1
vote
1 answer
Kafka-node: add consumers to consumer group
So I have a consumer group running and listening to messages from a topic. How do I add more instances of consumers to the consumer group? If this is how i listen to the messages
let consumer = new kafka.ConsumerGroup(options,…
Mister Spurious
- 271
- 2
- 11
1
vote
0 answers
Kakfa consumer is processing next message even before commit first message on same topic
I created single connector to push all insert/update event of the collection to the same topic (1 partition). The consumer code to process the message for insert event will take some time than update event.
The problem here is that the messages are…
Nandy
- 666
- 1
- 12
- 27
1
vote
0 answers
Does Kafka-node has similar config as 'max.poll.records' options in consumer writing in java?
Recently I am writing some coding under nodeJS using Kafka-node module, and I want to control max poll numbers for each consume. From most of the blogs I can find the solution in java using 'max.poll.records' but hard to find one for nodeJS.
Anyone…
ytsm
- 21
- 5
1
vote
0 answers
Listener for kafka partition change - Kafka node
I am currently working with Nodejs and Kafka, and I was looking for a way to detect changes in number of partitions in topics (currently using kafka-node)
Is there any API by which I can achieve this functionality?
I was thinking of calling…
Mister Spurious
- 271
- 2
- 11
1
vote
1 answer
Difference between npm library kafka-node and kafka-streams
In a project, Node.js application connects to Kafka message queue and get all messages from queue. I searched and found two packages kafka-node and kafka-streams. Which package will be suitable…
Manas Kumar
- 2,411
- 3
- 16
- 23
0
votes
0 answers
kafka-node : unable to pause consumer topic wise
I already have a kafka consumer group running in multiple instances. I have a requirement to pause the consumption of a particular topic whenever I reach a threshold. When I try to pause the kafka consumer group for a topic, the entire group gets…
Siddharth Panigrahi
- 17
- 7