1

I'm wonder what could cause this and how to fix.
kafka __consumer_offsets disk usage is huge (134 GB)
and asymmetric (mostly on broker 3, and mostly a single partition).
ReplicationFactor=3 and there are 3 brokers so I would at least expect symmetry,
although I am more concerned about reducing the size.

MSK 2.8.1 and confluent 6.2.10 for the command-line.

$ kafka-log-dirs --describe --bootstrap-server $BOOTSTRAP --topic-list __consumer_offsets | grep '^{' | jq -r '.brokers[] | ["broker", .broker, "=", (([.logDirs[].partitions[].size] | add // 0) | . / 10000 | round | ./ 100), "MB" ] | @tsv' | paste -sd , | tr '\t' ' '
broker 1 = 459.72 MB,broker 2 = 218.95 MB,broker 3 = 134346.48 MB

$ kafka-log-dirs --describe --bootstrap-server $BOOTSTRAP --topic-list __consumer_offsets | grep '^{' | jq -r '.brokers[] | ["broker", .broker, "=", (.logDirs[].partitions[].size / 1000000 | round)] | @tsv' | tr '\t' ' '
broker 1 = 52 1 0 0 1 0 0 243 102 0 2 0 3 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 47 4 0 0 0 0 0 0 5 0 0 0 0 1 2 3 1 0 0 0
broker 2 = 52 1 0 0 1 0 0 2 102 0 2 0 3 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 47 4 0 0 0 0 0 0 5 0 0 0 0 1 2 3 1 0 0 0
broker 3 = 133907 1 0 0 8 3 1 31 10 4 2 0 27 0 2 0 14 8 4 4 1 0 3 2 0 10 0 0 3 14 35 123 0 0 2 0 0 0 23 0 0 0 0 25 26 39 9 3 6 5

$ kafka-topics --bootstrap-server $BOOTSTRAP --describe --topic __consumer_offsets
Topic: __consumer_offsets   TopicId: ...    PartitionCount: 50  ReplicationFactor: 3    Configs: compression.type=producer,min.insync.replicas=2,cleanup.policy=compact,segment.bytes=104857600,message.format.version=2.8-IV1,max.message.bytes=10485880,unclean.leader.election.enable=true
...
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
dlipofsky
  • 289
  • 1
  • 4
  • 18

1 Answers1

1

There's not much that can really be done with this, at this point.

tl;dr You have consumer groups names that are hashed to the same partition, or you have one really large consumer group that does very frequent commits.

  1. The topic is compact, so data stays around. Frequent consumer commits can happen faster than compaction happens, causing that partition to grow quickly.
  2. You can consume that topic to inspect it (make sure you add --property print.key=true), you will notice that the keys are by the group.id set in your consumer code bases.
  3. If you change group.id for your consumers (assuming you can track them down), then they will all lose any existing offsets for the data they've already consumed without some manual intervention to migrate offsets using a combination of seek and commitSync consumer API; there is no built-in script to do this for you.

If you end up "moving" that partition, or segments of the logs, for example, then it'll start causing errors in consumers since the broker will still try to use "partition X" to fetch/commit offsets.

In practice, people often have some consumer / topic onboarding form, associated with ACL policies and expected usage quotas, and that "Kafka onboarding processTM " can enforce specific consumer group naming conventions.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thank you for the link to how to inspect it. I was wondering about that so the link helps. `--property print.key=true` doesn't appear to do anything for this topic (I did test it against other topics and verified that's the right syntax) but this works `kafka-console-consumer --bootstrap-server $BOOTSTRAP --topic __consumer_offsets --from-beginning --formatter 'kafka.coordinator.group.GroupMetadataManager$OffsetsMessageFormatter'` – dlipofsky Jun 15 '23 at 21:33
  • The key should be available. You can check source code how it is used https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L1236 – OneCricketeer Jun 15 '23 at 23:16
  • In any case, even without `print.key` the first thing on each line is [consumer group,topic,partition] and I can determine that one consumer group is the majority of the records. Also that CG is used on 64 topics, and each of those topics is 1 partition. I don't think that's good design. I plan to delete that CG when everything is scaled down and see if that clears it up the issue, but it's prod so it may be a while before I can get around to doing that (interestingly UAT appears to have the identical config but does not have the disk space problem). – dlipofsky Jun 16 '23 at 00:12
  • 1 partition topic would be necessary for ordering. I've seen larger consumer groups, but maybe that consumer is using a regex and subscribing to more topics than necessary? Hard to really know. – OneCricketeer Jun 16 '23 at 12:15
  • This fixed itself; after a month of monotonically increasing disk usage, broker 3 is now back to just a few hundred MB, same as broker 1 & 2. I have no idea why. UAT and prod are essentially identical config and apps, but UAT never had the issue. – dlipofsky Jul 12 '23 at 17:29