Questions tagged [node-kafka]
20 questions
5
votes
1 answer
How do I set up Elastic Node APM distributed tracing to work with Kafka and multiple Node services?
I'm using Kafka for a queue, with Node services producing and consuming messages to Kafka topics using Kafka-Node.
I've been using a home-brewed distributed tracing solution, but now we are moving to the Elastic APM.
This seems to be tailored to…
Boris K
- 3,442
- 9
- 48
- 87
3
votes
1 answer
Kafka-node suddenly consumes from offset 0
Sometimes, kafka-node consumer starts consuming from offset 0, while its default behavior it to consume only newer messages. Then it will not switch back to its default behavior. Do you know how to solve this and what happens and its behavior…
Cap Barracudas
- 2,347
- 4
- 23
- 54
3
votes
1 answer
Creating partition for topic in kafka-node
I have a created a HighLevelProducer to publish messages to a topic stream that will be consumed by a ConsumerGroupStream using kafka-node. When I create multiple consumers from the same ConsumerGroup to consume from that same topic only one…
Dan W
- 138
- 10
2
votes
1 answer
How to implement parallelism in kafka using nodejs consumers?
Theoratically speaking, since nodejs is single threaded how can I achieve parallelism when I define multiple consumers to increase throughput?
For eg, If I have a kafka topic that has 4 partitions, on the consumer end how would I be able to consume…
laxman
- 1,781
- 4
- 14
- 32
2
votes
1 answer
kafka-consumer-groups CLI not showing node-kafka consumer groupf
I have a kafka consumer group running on node.js powered by node-kafka. When this consumer group is active or in-active, I expect to see it reported by the kafa-consumer-groups CLI.
The kafka-consumer-groups CLI does show the console consumers and…
cubsnlinux
- 365
- 3
- 15
2
votes
1 answer
Kafka Node High Level Producer writes to even partitions only
I am using the Kafka Node library, and testing the high level producer.
I've created a topic with 10 partitions, 'HLPTestInput', and written a function to produce to it every second.
The producer writes to partitions 0,2,4,6 and 8, but not to the…
Boris K
- 3,442
- 9
- 48
- 87
1
vote
0 answers
The Kafka consumer automatically stops listening for messages after a certain period of time
I am implementing Kafka in Nodejs using node-Kafka/kafkajs. However, I am facing an issue with the consumer. The Kafka consumer stops listening for messages after some time, typically between 1.4 to 2.4 hours. And also we did not get any error or…
Chetan Parmar
- 31
- 5
1
vote
1 answer
node-rdkafka erorr while installing
I'm using Wind 10, from long time a tried to install node-rdkafka, but it didn't work,I tried to find a solution for it but failed, so i used another client(node-kafka) but it not strong like nod-rdkafka,
I tried all the possible solutions but…
mahmoud farag
- 11
- 2
1
vote
1 answer
Node kafka consumer is receiving messages but not consumer group
I am trying to connect to my local kafka. I created a topic "test_producer" and created a producer for it using:
bin/kafka-console-producer.sh --topic test_producer --bootstrap-server localhost:9092
Now i am trying to connect this using node-kafka…
alan trevor
- 23
- 7
1
vote
0 answers
How to avoid data loss in HighlevelProducer rdkafka
Using npm librdkafka i am producing large volume of data to the Kafka. I am using HighlevelProducer.
For 50K messages this configuration is working fine. But for larger volume like 100K messages, there is data loss. Data loss is varying from 200-1K…
jagadish
- 25
- 1
- 4
1
vote
0 answers
Kafka consumer in Angular component
I have written produce in C# and i am trying to consume in angular app. I have installed node module 'no-kafka'
when node module is imported in ts component like
import * as kafka from "no-kafka";
i am getting compilation error as follows
Date:…
Ravi Belkhindi
- 43
- 2
- 8
1
vote
1 answer
Kafka node throws "TypeError: Client is not a constructor" using Node-RED
Full flow I have installed the kafka node using
npm install -g node-red-contrib-kafka-node
But it throws
TypeError: Client is not a constructor
Can anyone help me to resolve this issue?
kafka node properties settings
Sudhakar
- 45
- 6
1
vote
1 answer
kafka client is sending request to partition where broker which went down
I am using kafka-node module to send message to kafka.
In a clustered environment where I have a topic with 3 partitions and replication factor as 3.
Topic describe is -
Topic:clusterTopic PartitionCount:3 ReplicationFactor:3 …
Anand Jain
- 603
- 7
- 20
1
vote
1 answer
Receiving Junk Characters when messages are consumed using kafka-node
I Am using the npm package kafka-node version 3.0.1.
However, am receiving junk characters in the message -
""M`@$�q��1��N$907959dc-30e9-4e5c-af44-09a4f9062fe1�{"header":{"eventName":"myevent","producer"�'INE",DETECTED"}}"
Any help is…
Vivek
- 61
- 1
- 8
1
vote
0 answers
ConsumerGroup consumerOptions Configuration error @types/kafka-node
I'm trying to use consumerGroup instead of HighLevelConsumer in a typescript project, but I'm not able to configure consumerOptions for consumerGroup.
I am getting the following error when I assign
1) fromOffset: 'earliest'
Argument of type '{…
SathyaJ
- 11
- 2