1

The consumer does not work sometimes and sometimes it does, not able to figure the issue.

spring boot - 1.5.21.release
kafka client - 0.11.0.0
kafka servers - IBM event stream service on cloud.

application properties:

#Producer
spring.kafka.template.default-topic=distitopic
spring.kafka.producer.client-id=event-streams-kafka
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

#Consumer
tier2.disti.search.topic=distitopic
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.group-id=distigroup
spring.kafka.consumer.auto-offset-reset=latest
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer

And in my rest controller, I have this method 

@KafkaListener(topics = "${tier2.disti.search.topic}")
public void listen(ConsumerRecord<String, String> cr) throws Exception {

In the system out logs of WebSphere Application server, I see these messages very often 

[7/8/19 13:17:04:662 GMT] 00000347 SystemOut     O 2019-07-08 13:17:04.662  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 5 could not be established. Broker may not be available.
[7/8/19 13:17:24:246 GMT] 0000033e SystemOut     O 2019-07-08 13:17:24.246  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:19:12:022 GMT] 00000347 SystemOut     O 2019-07-08 13:19:12.022  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 8 could not be established. Broker may not be available.
[7/8/19 13:19:32:630 GMT] 0000033e SystemOut     O 2019-07-08 13:19:32.630  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:21:19:510 GMT] 00000347 SystemOut     O 2019-07-08 13:21:19.510  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 4 could not be established. Broker may not be available.
[7/8/19 13:21:41:014 GMT] 0000033e SystemOut     O 2019-07-08 13:21:41.014  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:23:26:870 GMT] 00000347 SystemOut     O 2019-07-08 13:23:26.870  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 7 could not be established. Broker may not be available.
[7/8/19 13:23:49:526 GMT] 0000033e SystemOut     O 2019-07-08 13:23:49.526  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:25:34:230 GMT] 00000347 SystemOut     O 2019-07-08 13:25:34.230  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 5 could not be established. Broker may not be available.


[7/8/19 13:25:58:038 GMT] 0000033e SystemOut     O 2019-07-08 13:25:58.038  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:27:41:590 GMT] 00000347 SystemOut     O 2019-07-08 13:27:41.590  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 7 could not be established. Broker may not be available.
[7/8/19 13:28:06:550 GMT] 0000033e SystemOut     O 2019-07-08 13:28:06.550  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:29:48:950 GMT] 00000347 SystemOut     O 2019-07-08 13:29:48.950  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 4 could not be established. Broker may not be available.
[7/8/19 13:30:14:934 GMT] 0000033e SystemOut     O 2019-07-08 13:30:14.934  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:32:23:062 GMT] 0000033e SystemOut     O 2019-07-08 13:32:23.062  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:34:31:446 GMT] 0000033e SystemOut     O 2019-07-08 13:34:31.446  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.

[7/8/19 13:36:39:702 GMT] 0000033e SystemOut     O 2019-07-08 13:36:39.702  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:38:47:958 GMT] 0000033e SystemOut     O 2019-07-08 13:38:47.958  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:40:56:086 GMT] 0000033e SystemOut     O 2019-07-08 13:40:56.086  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:41:26:678 GMT] 00000347 SystemOut     O 2019-07-08 13:41:26.678  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 6 could not be established. Broker may not be available.
[7/8/19 13:43:04:342 GMT] 0000033e SystemOut     O 2019-07-08 13:43:04.342  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:45:12:854 GMT] 0000033e SystemOut     O 2019-07-08 13:45:12.854  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.


[7/8/19 13:47:21:110 GMT] 0000033e SystemOut     O 2019-07-08 13:47:21.110  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:48:34:070 GMT] 00000347 SystemOut     O 2019-07-08 13:48:34.070  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 4 could not be established. Broker may not be available.
[7/8/19 13:49:29:622 GMT] 0000033e SystemOut     O 2019-07-08 13:49:29.622  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:50:41:558 GMT] 00000347 SystemOut     O 2019-07-08 13:50:41.558  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 6 could not be established. Broker may not be available.
[7/8/19 13:51:38:006 GMT] 0000033e SystemOut     O 2019-07-08 13:51:38.006  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:52:48:918 GMT] 00000347 SystemOut     O 2019-07-08 13:52:48.918  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 7 could not be established. Broker may not be available.
[7/8/19 13:53:46:518 GMT] 0000033e SystemOut     O 2019-07-08 13:53:46.518  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.

[7/8/19 13:55:55:030 GMT] 0000033e SystemOut     O 2019-07-08 13:55:55.030  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:58:03:670 GMT] 0000033e SystemOut     O 2019-07-08 13:58:03.670  WARN 32021 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=distigroup] Connection to node -5 could not be established. Broker may not be available.
[7/8/19 13:59:56:310 GMT] 00000347 SystemOut     O 2019-07-08 13:59:56.310  WARN 32021 --- [t-streams-kafka] org.apache.kafka.clients.NetworkClient   : [Producer clientId=event-streams-kafka] Connection to node 8 could not be established. Broker may not be available.
aryanRaj_kary
  • 503
  • 2
  • 7
  • 28
  • try updating your dependencies. kafka client 0.11.0.0 is somewhat old. If the broker already uses a newer version you could experience problems like you describe. – sschrass Jul 08 '19 at 08:16
  • tried with 2.0.0, still same – aryanRaj_kary Jul 08 '19 at 08:59
  • Can you add some details regarding the failures you see? How often do they appear? How long do they last? When did last one occur? We'll try our best to help you out here but if the issue persist, please open a support ticket: https://cloud.ibm.com/docs/services/EventStreams?topic=eventstreams-report_problem – Mickael Maison Jul 08 '19 at 10:14
  • Based on the logs I can see that it is happening every minute. Have updated the question with more logs(with timestamps). – aryanRaj_kary Jul 08 '19 at 14:06
  • Your consumer is trying to consume messages from broker but for some reason broker is not available as message say. When this message occurs you can try to list your brokers to see which brokers are up. Here is how to list brokers: https://stackoverflow.com/questions/40146921/command-to-get-kafka-broker-list-from-zookeeper – Spasoje Petronijević Jul 08 '19 at 14:52
  • I am using a windows 10 machine, have installed the zookeeper and Kafka on my machine but I am not sure how to get the list of brokers using the zookeeper-shell on a windows machine. – aryanRaj_kary Jul 09 '19 at 06:59
  • Opened a ticket - CS0336197 – aryanRaj_kary Jul 09 '19 at 07:02

1 Answers1

0

Are you specifying the actual comma separated host names in "bootstrap.servers" or the fqdn? I faced the similar issue when used the fqdn instead of directly using the host names.

Nidhin jose
  • 60
  • 5