1

I am noticing something weird happening with my system. So, I am using Kafka to send and receive messages between different systems. I have around 6 or 7 topics each with 10 partitions. I have an external system that is sending messages on my Kafka topics. So this external system will send messages initially to a topic for eg. "XYZ" and will wait for a response from the Server. Once the Server reads and responds back to the external system then only it will continue further.

Now in our scenario when the external system sends messages to topic "XYZ" it is always sending on partition no 6. This is happening even after restarting the entire system multiple times. Messages on XYZ topic are always being sent to Partition 6.

Now on the Server side, I am using kafka-node to create clients, consumer and producer to consume and produce the messages to kafka. But in this case, it is not consuming from the topic "XYZ".

As a workaround, I tried to test everything by deleting the topics and creating them again but only with a single partition, and this time it worked fine. The entire system worked without any problem.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
JThind
  • 13
  • 2
  • have u subscribed right as a consumer ? can u paste your code. – Amin S Jan 26 '23 at 12:52
  • It won't be possible to paste the code since it is a production environment and I can't add it here. What does this line mean? - "have u subscribed right as a consumer?" – JThind Jan 26 '23 at 14:11
  • you should define topics for your consumers. handling partitions are automatic by kafka as i know. so i guess there should be problem in your subscription functions and your topic list. – Amin S Jan 26 '23 at 16:16

1 Answers1

0

and creating them again but only with a single partition, and this time it worked fine

Unclear what this scenario is testing... If you wanted 1 partition, then why did the topics get created with 10?

The only reason, in theory, why this would happen is if you are closing and re-creating the producer instance, and it is not correctly randomly seeding the round-robin distribution of the sent events, and always picking the same value. Or, you have defined a key for your records, and it's always hashed to partition 6.

in this case, it is not consuming from the topic "XYZ".

Only one consumer can be active on any partition at a time. If all data ends up in partition 6, then you can only have one consumer... So, sounds like something is reading it, just not what you expect.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245