Spring-Boot and Kafka : How to handle broker not available?

Question

While the spring-boot app is running and if I shutdown the broker completely ( both kafka and zookeeper ) I am seeing this warn in console for infinite amount of time.

[org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN o.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=ResponseReceiveConsumerGroup] Connection to node 2147483647 could not be established. Broker may not be available.

Is there a way in Spring Boot to handle this gracefully instead of infinite logs on console ?

I believe those are kafka library logs, when consumer or producer starts it will try to identify brokers for every time intervals — Ryuzaki L, Aug 11 '18 at 02:44

score 3 · Accepted Answer · answered Aug 11 '18 at 18:46

3

Increase the reconnect.backoff.ms property (see Kafka docs).

The default is only 50ms.

answered Aug 11 '18 at 18:46

Gary Russell

166,535
14
146
179

Thanks, Instead of this constant back off , How would we handle longer broker outages ? – lazydev Aug 11 '18 at 20:47
2

That's really a question for the Kafka folks since it's out of Spring's control. Fortunately, the new 2.0.0 kafka-clients no longer block when this occurs (supported by spring-kafka 2.2 which will be released late September/early October, 2.2..0.M1 milestone is available now). From a Spring perspective, you will then be able to `stop()` the message listener container(s). All that being said, with the whole resiliency model for Kafka (multiple replicas) the intention is that the broker should never be "down". – Gary Russell Aug 11 '18 at 20:57

Spring-Boot and Kafka : How to handle broker not available?

1 Answers1

Linked