9

I am running kafka locally following instructions on quick start guide here,

and then I defined my consumer group configuration in config/consumer.properties so that my consumer can pick messages from the defined group.id

Running the following command,

bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

results in,

test-consumer-group  <-- group.id defined in conf/consumer.properties
console-consumer-67807 <-- when connecting to kafka via kafka-console-consumer.sh

I am able to connect to kafka via a python based consumer that is configured to use the provide group.id i.e test-consumer-group

First of all, I am not able to understand how/when kafka creates consumer groups. It seems it loads the conf/consumer.properties at some point of time and additionally it implicitly creates consumer-group (in my case console-consumer-67807) when connecting via kafka-console-consumer.sh.

How can I explicitly create my own consumer group, lets say my-created-consumer-group ?

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
laxman
  • 1,781
  • 4
  • 14
  • 32

1 Answers1

16

You do not explicitly create consumer groups but rather build consumers which always belong to a consumer group. No matter which technology (Spark, Spring, Flink, ...) you are using, each Kafka Consumer will have a Consumer Group. The consumer group is configurable for each individual consumer.

It seems it loads the conf/consumer.properties at some point of time and additionally it implicitly creates consumer-group (in my case console-consumer-67807) when connecting via kafka-console-consumer.sh

If you do not tell your console consumer to actually make use of that file it will not be taken into consideration.

There are the following alternatives to provide the name of a consumer group:

Console Consumer with property file (--consumer.config)

This is how the file config/consumer.properties should look like

# consumer group id
group.id=my-created-consumer-group

And this is how you would then ensure that the console-consumer takes this group.id into consideration:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning --consumer.config /path/to/config/consumer.properties

Console consumer with --group

For console consumers the consumer group gets created automatically with prefix "console-consumer" and suffix something like a PID, unless you provide your own consumer group by adding --group:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning --group my-created-consumer-group

Standard code-based consumer API

When using the standard JAVA/Scala/... Consumer API you could provide the Consumer Group through the properties:

Properties settings = new Properties();
settings.put(ConsumerConfig.GROUP_ID_CONFIG, "basic-consumer");
// set more properties

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(settings)) {
consumer.subscribe(Arrays.asList("test-topic")
Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • 1
    will the above cmd work if I don't already have `my-created-consumer-group` created. In other words, if a consumer connects with any random group.id will kafka accept it creating that consumer group ? – laxman May 13 '20 at 10:08
  • 2
    Yes and yes :) If a consumer with a random group.id consumes messages for the first time, Kafka checks if this consumer group already exists. If this consumer group did not exist before, the consumer will read the topic from beginning (unless otherwise stated). – Michael Heil May 13 '20 at 10:10
  • the console consumer is exactly doing this: creating random group.id and is able to read the data, – Michael Heil May 13 '20 at 10:11
  • 1
    You can have a closer look [here](https://docs.confluent.io/current/clients/consumer.html#consumer-groups) to understand more on the concept of Consumer Groups in Kafka. As this is quite essential when working with Kafka I highly recommend to get a good understanding of it! – Michael Heil May 13 '20 at 10:13