0

Broken topic cluster1

Broken topic cluster2

I have a problem with some Kafka topics and couldn't find an answer to it yet.

While adding more partitions to __confluent.support.metrics shouldn't be a problem (I know how to do that), I wonder if it is possible to tell it to use brokers which obviously can not be seen by this topic?

Also I'd love to understand why these topics only inherit some brokers instead of all available 5 brokers in their cluster.

I'd love to fix these topics. But I fear that if I tell it to add (or use) partitions on brokers the topic can't "see", that it might not work or even destroy the topic, which would be rather bad.

How can I instruct these topics, that there are 5 available brokers? Can I do it with one of the Kafka tools?

How could that have happened in the first place?

Why does the __consumer_offsets topic only "see" 4 brokers instead of 5 like all other topics in this cluster do?

FYI: I didn't setup any of this, but I have to cleanup/revamp the running clusters and am stuck now, I never came across this sort of problem before

Phantômaxx
  • 37,901
  • 21
  • 84
  • 115
s1mmel
  • 101
  • 3
  • Why are you trying to modify the Confluent support topic? If you are actively using Confluent support, it may break things. If anything, you should increase the replication factor – OneCricketeer Aug 04 '18 at 18:48

2 Answers2

0

The reason this has happened is because you have only one partition and one replica for the __confluent.support.metrics topic. In a 5-node cluster, this means you will only be using 20% of the available brokers in the cluster, which corresponds with the image you've posted. A topic with replication-factor 1 and 1 partition will only ever hold data on one broker.

On the other hand, it is unusual that your __consumer_offsets topic would be using only 4 out of 5 brokers. My guess would be that your 5th broker was not online at the time of creation of __consumer_offsets (this is created when you consume from any topic for the first time) and thus no partitions were created on this broker.

However, this is probably nothing to worry about, as the spread of partitions across the cluster is generally handled by Kafka itself rather than being a user problem. There is no concept of a topic "seeing" a broker per se; rather, the brokers hold the data for the topics, and the topics will know which brokers they reside on. A topic doesn't generally need to concern itself with other brokers.

Simon Clark
  • 624
  • 3
  • 11
0

Both the consumer offsets and Confluent metrics topics have line items in the server properties file that determines what configurations those topics will be created with.

To improve the health of those topics, you can attempt to increase the replication factor, which will spread your topic over more brokers and provide fault tolerance. Also see Kafka Tools Wiki

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thank you, I'm going to try that. I just wanted to have some feedback on this, before changing them. I was a little bit confused. As I said I never encountered this odd behaviour before. My next step would've been to re-replicate this to a factor of 5 with 50 partitions. I guess this is what I'm going to do now, starting with my test clusters. If this works well, I'm going for the production ones. Thansk again for the feedback. – s1mmel Aug 06 '18 at 07:57