0

I am running a plain Apache Kafka server (version 3.4.1) which I would like to connect to a Telegraf consumer. The Telegraf [[inputs.kafka_consumer]] plugin has the option to consume by Kafka "group". When staring Telegraf, I get an error message

[inputs.kafka_consumer] Error in plugin: consume: kafka server: Request was for a consumer group that is not coordinated by this broker

Hence, I started to investigate my setup by using the Kafka console tools and found that when executing

 ./kafka-console-consumer.sh --bootstrap-server myserver:9092 --topic test --partition 0

and sending messages via kafka-console-producer.sh, these messages pop up in the console "consumer" window as expected.

In contrast to this, when I run

 ./kafka-console-consumer.sh --bootstrap-server myserver:9092 --topic test --group my-group

nothing happens in the "consumer" window. Furthermore, the command

./kafka-consumer-groups.sh --bootstrap-server myserver:9092 --list

yields nothing.

What do I have to do to cause the consumer with the "group" option to "see" the messages produced to the topic "test"? Ultimately, how can I solve the Telegraf error?

Update: it would be great if you provide a set of Kafka console commands which reliably produces and consumes messages to a consumer group.

WolfiG
  • 1,059
  • 14
  • 31
  • 1) Using `--partition 0` does not use consumer groups, which is why listing does nothing. 2) Telegraf should work, but the error seems to suggest the group needs to be created ahead of time – OneCricketeer Jun 13 '23 at 19:36
  • @OneCricketeer according to https://stackoverflow.com/questions/61770993/how-to-create-a-new-consumer-group-in-kafka I understood that consumer groups are not explicitly created, but get created on the fly when the consumer establishes it's connection. Or am I wrong? In addition to that I did not find a way to create a user group separated from the consumer call... – WolfiG Jun 14 '23 at 07:14
  • They only get created when not using partition assignment – OneCricketeer Jun 14 '23 at 13:14
  • So in my second case Kafka should create groups, right? But I don‘t see any when executing ‚Kafka-consumer-groups‘ – WolfiG Jun 16 '23 at 06:14
  • You'll only be able to describe the group while the consumer group is active, otherwise, the output will say there's no active consumers – OneCricketeer Jun 16 '23 at 12:17

1 Answers1

0

I can't really comment on why Telegraf is not working, however....

If you have a large number of partitions in your topic, it can take a really long while before you start seeing messages on the console.

So, using the partition id is the right approach. But just need to be patient at the command-line.

Out of interest, how long did you wait for the output from console consumer ?

If you need to know the partition id, you will need to use a custom program to compute it based on the key. (You could have a look at the murmur2 source code on the Kafka github repository and try to create a simple command line tool to compute the partition id using the key).

However, using --group option will only set the consumer group id of your instance of the kafka-console-consumer.sh. If you have other consumers that are part of this consumer group, then I believe some rebalancing should occur as some partitions will get moved to your kafka-console-consumer.sh client.

Neeraj
  • 69
  • 7
  • Hi Neeraj, I do not have a large number of partitions. Actually I am running a littele toy instance of Kafka with one topic and three partitions. Concerning latency between producing and consuming: the delay is negligible, not noticable. – WolfiG Jun 14 '23 at 07:09