13

I changed the consumer-group-id of my web service listening to a Kafka topic. Now, the old group id is still registered to the topic, but there is no consumer with that group id. Therefore, it is lagging. How can I remove a specific consumer group from a specific topic?

I tried this:

kafka-consumer-groups --bootstrap-server kafka01.myserver.com:9092 --topic notification-topic --delete --group old-consumer-group --execute

But it returns: "The consumer does not support topic-specific offset deletion from a consumer group."

Should I remove the consumer group totally? I use the same group id listening to other topics, are they going to be affected?

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
Cagatay
  • 305
  • 1
  • 3
  • 10

2 Answers2

27

Starting with Kafka 2.4.0 it is possible to delete individual consumer group id offsets from a topic.

The call is very close to what you have already tried, but requires --delete-offsets instead of --delete:

./kafka-consumer-groups.sh \
  --bootstrap-server <bootstrap-server-url> \
  --delete-offsets \
  --group <my-group> \
  --topic <topic-name>

To date this is not reflected in the official documentation (Kafka 2.7.0 and below). However, it is described in a confluence document of improvement proposal KIP-496 that has been implemented and released.

summon
  • 418
  • 5
  • 11
1

You cannot remove a specific group on its own, nor a topic from a group.

Consumer groups remain available based on the retention you've defined for the offsets topic on the broker. There is no performance impact by having lagging groups that are not being consumed.

Note: Every time you use kafka-console-consumer, a brand new group is created.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • 7
    Yes, there is no performance impact. But consumer-lag graphs can become noisy. Also, some people might have alerts in place for consumer lags (despite the consumer-consumer ones) surpassing some threshold for some timespan. Both are currently true in my use-case. So for me, it would be really helpful if there was some way (even a hack) to delete the offsets for a group for a specific topic. – Tobias Hermann Dec 10 '20 at 08:29
  • 1
    The AdminClient can now delete empty groups https://kafka.apache.org/23/javadoc/org/apache/kafka/clients/admin/AdminClient.html#deleteConsumerGroups-java.util.Collection- – OneCricketeer Dec 10 '20 at 13:49
  • Deleting a whole consumer group was already possible with `kafka-consumer-groups.sh`. Instead, op and I would like to delete only the offsets regarding one specific topic for a consumer group that is still actively consuming other topics. Is this possible with `AdminClient` too`? I can't find this in the docs you linked to. – Tobias Hermann Dec 10 '20 at 13:53
  • Groups as a whole are managed by messages in the offsets topic. It's not possible to target one topic without affecting the others in the group. For example, if you grab one message from the offsets topic, it'll have a list of topics and offsets they've been committed to; if you removed a topic from the list and sent that message back to the offsets topic to update the group, and meanwhile another consumer has already pushed new offsets into the same group, you'd effectively cause the group to rollback and poll duplicate messages if it didn't otherwise throw an error. – OneCricketeer Dec 10 '20 at 14:49
  • Ah, ok. Thanks for the explanation. But it sounds like it would be possible to do this safely (without accidental duplicate message consumption) if all consumers from this consumer group are stopped completely before doing this operation, right? Asking, because this would be a feasible option in my case. – Tobias Hermann Dec 10 '20 at 16:27
  • Correct, if all consumers are inactive, it could be done, though, it'd still require finding the most recent message out of that topic for that group, changing it, then pushing it back. Meanwhile, the delete request from the method above simply appends a null value for the group key and doesn't do any such lookups – OneCricketeer Dec 10 '20 at 18:15
  • The delete request from the method above would delete the consumer offsets of that group for *all* topics, which is not what I want because it would break a lot of things in my use-case. :/ – Tobias Hermann Dec 10 '20 at 19:59
  • @TobiasHermann - this is correct. The only way you can manage this is to set the group_id and manage the offsets that way (much like with console consumer). I've found giving them unique names across all topics allows for better control - without having to stop the other members of the same group to do so you are able to reset offsets. The accepted answer does look like this use case has been resolved - and I believe ensuring group id is set should allow your monitoring to work uninhibited. – bob dylan May 12 '21 at 11:15