2

I have configured my producer as :

spring.cloud.stream.bindings.pc-abc-out-0.destination=pc-abc-my-topic
spring.cloud.stream.bindings.pc-abc-out-0.producer.partition-count=5
spring.cloud.stream.bindings.pc-abc-out-0.producer.header-mode=headers
spring.cloud.stream.bindings.pc-abc-out-0.producer.partition-count=10
spring.cloud.stream.bindings.pc-abc-out-0.producer.partitionKeyExpression=payload.key
spring.cloud.stream.kafka.bindings.pc-abc-out-0.producer.sync=true

However, in the kafka logs , I keep getting the error :


o.s.kafka.support.LoggingProducerListener - Exception thrown when sending a message with key='byte[14]' and payload='byte[253]' to *topic pc-abc-my-topic and partition 8*: org.apache.kafka.common.errors.TimeoutException: Topic pc-abc-my-topic not present in metadata after 60000 ms.

The highlight in the error message being : topic pc-abc-my-topic and partition 8

Why is it looking for partition 8 even though I have defined number of partitions as 5. Shouldn't the numbering be betweenn 0-4. I have several other error messages with partition number more than 5.

Earlier in my configuration I had added

spring.cloud.stream.kafka.binder.auto-add-partitions=true

But I removed it and we scaled down and scaled up the services. The issue still exists. Is this the case of stale config?

FenderBender
  • 87
  • 1
  • 9

2 Answers2

1

The issue turned out to be this configuration:

spring.cloud.stream.bindings.pc-abc-out-0.producer.partitionKeyExpression=payload.key
spring.cloud.stream.kafka.bindings.pc-abc-out-0.producer.sync=true

The partition number being calculated because of payload.key is somwhow greater than the partitionCount value we configured and created on our infrastructure. Removing these two configuration stopped the issue.

FenderBender
  • 87
  • 1
  • 9
0

i guess you create topic with 8 partition first. if topic already exist spring-kafka doesn't create new one with your partition configs.

Here doc https://docs.spring.io/spring-cloud-stream/docs/Brooklyn.RELEASE/reference/html/_apache_kafka_binder.html#:~:text=Default%3A%20Empty%20map.-,The,-Kafka%20binder%20will

The Kafka binder will use the partitionCount setting of the producer as a hint to create a topic with the given partition count (in conjunction with the minPartitionCount, the maximum of the two being the value being used). Exercise caution when configuring both minPartitionCount for a binder and partitionCount for an application, as the larger value will be used. If a topic already exists with a smaller partition count and autoAddPartitions is disabled (the default), then the binder will fail to start.
If a topic already exists with a smaller partition count and autoAddPartitions is enabled, new partitions will be added. If a topic already exists with a larger number of partitions than the maximum of (minPartitionCount and partitionCount), the existing partition count will be used.

Your real problem is Topic pc-abc-my-topic not present in metadata after 60000 ms. don't care about partition count.

Here solution for this

org.apache.kafka.common.errors.TimeoutException: Topic not present in metadata after 60000 ms

https://developpaper.com/topic-xxx-not-present-in-metadata-after-60000-ms/

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
divilipir
  • 882
  • 6
  • 17
  • Thank you for your response. I already tried adding jackson libraries , it was not useful, the error was still there .Why I think partition count is the real problem is because when I describe the topic only 5 partitions exist as configured and hence the error seems queer , because it is looking for partition number 8 , which does not exist. For the part , "topic already exists with a larger number of partitions" , it is not applicable in my scenario , in our environment ,pre-created topic exists with 5 partitions. – FenderBender Jan 25 '22 at 17:02