1

Hi I am trying to run this code in but it is working fine in another EC2 Azkaban instance but not giving below error for another instance.

private val adminprops = new Properties() 
adminprops.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,"Kafka Endpoint")
private val admin = AdminClient.create(adminprops)

def topicExist(topicName: String): Boolean = {
 val result = admin.listTopics.names.get.contains(topicName)
 result
}

"Kafka Exception java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics"

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Amarjeet Singh
  • 11
  • 1
  • 2

1 Answers1

2

Network problem or setting problem.

  1. make sure "bootstrap.servers" is correct，right host and port.
  2. make sure the network is ok between the client and the broker server.

The default timeout of adminClient is 120000ms(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG). Normally,list topics return immediately.

allen
  • 21
  • 4
  • Some time the network this error is coming and sometime not. So is there any solution on it – Amarjeet Singh Mar 15 '22 at 09:34
  • Kafka's bottleneck often occurs in disk capacity and bandwidth. I suggest you first check to see if bandwidth usage is consistently high, and if so, check if the producer has message compression turned on, which will significantly reduce bandwidth usage. – allen Mar 21 '22 at 01:30
  • Regardless of the cause of the network problem, look at the scenario you are using, there are some examples of solutions: Method 1: The request timeout can be reduced here, for example, to 2s, and catch the timeout exception to identify the network problem, try again after a while. Method 2: Get the topic list periodically to allow short-term data inconsistency. When you get the topic, use the cache Method 3: Query the zookeeper of kafka directly by yourself, if there is no way. Kafka keep its meta data in zookeeper. It is recommended that you locate and resolve network problems first. – allen Mar 21 '22 at 01:30