Kafka throws java.nio.channels.ClosedChannelException

Question

When i try to consume messages from the kafka server which is hosted in ec2 with kafka console tool (V 0.9.0.1 , i think this uses old consumer APIs) I get following exception. How can i overcome this?

#./kafka-console-consumer.sh --zookeeper zookeeper1.xx.com:2181 --topic MY_TOPIC --from-beginning

[2016-04-06 14:34:58,219] WARN Fetching topic metadata with correlation id 0 for topics [Set(MY_TOPIC)] from broker [BrokerEndPoint(1014,kafka3.xx.com,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
    at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-04-06 14:34:58,222] WARN Fetching topic metadata with correlation id 0 for topics [Set(MY_TOPIC)] from broker [BrokerEndPoint(1013,kafka22.xx.com,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
    at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-

score 6 · Accepted Answer · answered Apr 06 '16 at 06:47

6

The reason for the original closed channel exception is, i had some DNS issue, which i have solved by editing my local hosts file

answered Apr 06 '16 at 06:47

Ratha

9,434
17
85
163

1

Thanks a lot for hint. I had same problem for a while both in my centos and windows 10. As soon as I fixed ´hosts´ file in both machines error disappeared! – Abu Shumon Sep 01 '16 at 10:49

score 1 · Answer 2 · answered Apr 13 '18 at 05:13

1

I was able to solve this issue by setting advertised.host.name to the config file

answered Apr 13 '18 at 05:13

Denis Sborets

11
1

score 0 · Answer 3 · answered Apr 06 '16 at 06:02

0

This is actually a WARNING - no big deal. May be your topic is corrupted? Try to recreate topic.

answered Apr 06 '16 at 06:02

ponkin

2,363
18
25

I don't think for all topics it gives same type issue pls have a look http://stackoverflow.com/questions/36442668/failed-to-add-leader-for-partitions-in-kafka – Ratha Apr 06 '16 at 06:10
I run into several issues with new consumer, it goes infinite loop without polling messages. So, choking with older consumer APIs(the consumer console tools with the kafka distribution) and now stated to get this type closed channel issues – Ratha Apr 06 '16 at 06:11
Here are some issues i face http://stackoverflow.com/questions/36393867/if-my-producer-producing-then-why-the-consumer-couldnt-consume-it-stuck-pol http://stackoverflow.com/questions/36439818/warn-error-while-fetching-metadata-with-correlation-id-1-my-topic-invalid-to – Ratha Apr 06 '16 at 06:13
2

@Ratha Ok. If you running kafka on EC2 there may be an issue with private IP's - it can be resolved by setting advertised.host.name=, try to search "kafka on EC2 advertised.host.name" or on http://stackoverflow.com/questions/30880811/kafka-quickstart-advertised-host-name-gives-kafka-common-leadernotavailableexce – ponkin Apr 06 '16 at 06:16
The reason for the original closed channel exception is, i had some DNS issue, which i have solved by editing my local hosts file. And the EC2 case, yes i believe my settings are right – Ratha Apr 06 '16 at 06:47
Check this thread at user kafka list //Is there any behavioural change to connect local server and remote server?// – Ratha Apr 06 '16 at 06:49

Kafka throws java.nio.channels.ClosedChannelException

3 Answers3