2

I have a Kafka cluster with 3 brokers configured as follows:

Broker-0 : broker.id=0 ; listeners=PLAINTEXT://localhost:9092 ; log.dirs=/opt/data/kafka/logs-0
Broker-1 : broker.id=1 ; listeners=PLAINTEXT://localhost:9093 ; log.dirs=/opt/data/kafka/logs-1
Broker-2 : broker.id=2 ; listeners=PLAINTEXT://localhost:9094 ; log.dirs=/opt/data/kafka/logs-2

The producer targets Broker-2 to publish messages

sh-3.2# ./kafka-console-producer.sh --broker-list localhost:9094 --topic clusterTopic

I started testing with all brokers up and running

sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic  PartitionCount:1    ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0    Leader: 0   Replicas: 0,1,2 Isr: 2,0,1
sh-3.2#

A this point, all messages published by the producer are received by the consumer

Then I killed Broker-0 (the current leader)

sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic  PartitionCount:1    ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0    Leader: 1   Replicas: 0,1,2 Isr: 2,1
sh-3.2#

At this point, no messages published by the producer are received by the consumer anymore

Then, I enabled back Broker-0

sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic  PartitionCount:1    ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0    Leader: 1   Replicas: 0,1,2 Isr: 2,1,0
sh-3.2#

At this point, all messages published by the producer are received properly by the consumer

Then I killed Broker-1 (the current leader)

sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic  PartitionCount:1    ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0    Leader: 0   Replicas: 0,1,2 Isr: 2,0
sh-3.2#

At this point, all messages published by the producer are still received properly by the consumer.

Why is the behavior so different from broker to broker?

Update #1

I noticed something that might be helpful:

When I kill Broker-1, I see the following warning on the consumer console (a bunch of times)

[2018-06-07 13:59:00,866] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20882] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

But when I kill Broker-0, I see these warning messages (a bunch of times)

[2018-06-07 14:04:08,433] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20882] Connection to node 2147483647 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-06-07 14:04:08,434] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20882] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

I wonder what is the broker with ID=2147483647?

rh4games
  • 962
  • 2
  • 15
  • 38
  • I think this could be related to unclean leader election. Have u got only 1 partition?(seems like it from ur question) – Indraneel Bende Jun 06 '18 at 02:46
  • Also, whats ur value for min insync replicas? – Indraneel Bende Jun 06 '18 at 03:11
  • Thanks @IndraneelBende for your response. Yes, I have only one partition. I'll check the value of min insync repicas, but I did not change it from the default, and the cluster works fine with brokers 0 and 2 but not with 1 and 2 – rh4games Jun 06 '18 at 16:37
  • @IndraneelBende how do I check the value of min insync replicas? – rh4games Jun 07 '18 at 21:27
  • if you have not set it explicitly , it will pick up the default. It can be set when you configure the topic . https://kafka.apache.org/documentation/#topicconfigs – Indraneel Bende Jun 08 '18 at 01:04
  • Hi @rh4games, did you find what was your issue? If yes, care to share? :) – jumping_monkey Nov 15 '19 at 08:17
  • @rh4games Hi did you find the solution to this. Same thing happening with me - When Broker ID 0 goes down then I am to produce Data but NOT able to consume it. Ironic in 2020 same issue... – Arpan Sharma Nov 19 '20 at 19:11

0 Answers0