2

I am facing an issue with the Kafka clustering setup that I have. I have a Kafka cluster with two broker that are connected to two zookeepers. I am posting data to a topic that have replication factor and partition two each with a spring boot Kafka producer and consuming the same with another spring boot app.

I found one strange behavior when testing the cluster in the following manner -

 Turned off node1 and node 2
 Turned on node 1
 Turned off node 1
 Turned on node 2

After turning on node 2 Kafka cluster got failed and I am not able to produce data to Kafka. My consumer started throwing the message continuously as given below.

[Producer clientId=producer-1] Connection to node 1 (/server1-ip:9092) could not be established. Broker may not be available.

Issue is visible in both nodes. But if I kept both system up for a while issue will get resolved and I can turn off any of the node without breaking the cluster.

My broker configuration is as below.


broker.id=0
listeners=PLAINTEXT://server1-ip:9092
advertised.listeners=PLAINTEXT://serever1-ip:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/user/kafka/data/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=serever1-ip:2181,serever2-ip:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=3000
auto.leader.rebalance.enable=true
leader.imbalance.check.interval.seconds=5


Zookeeper configuration


dataDir=/home/user/kafka/data
clientPort=2181
maxClientCnxns=0
initLimit=10
syncLimit=5
tickTime=2000
server.1=server1-ip:2888:3888
server.2=server2-ip:2888:3888

Is this is an expected behavior of Kafka or am I doing something wrong with this configuration ?

Can somebody help me with this issue ..

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
chikku
  • 21
  • 1
  • 2
  • add all the bootstrap server url to the config. `listeners=PLAINTEXT://server1-ip:9092,{server2}` – Prog_G Feb 17 '20 at 10:00
  • You should never use an even number of Zookeepers – OneCricketeer Feb 17 '20 at 14:45
  • So this issue might be solved if I have odd number of zookeepers? – chikku Feb 17 '20 at 15:41
  • Possibly answered by answers to this question discussing Zookeeper quorum and numbers of In Sync Replicas (ISRs) https://stackoverflow.com/questions/58761164/in-kafka-ha-why-minimum-number-of-brokers-required-are-3-and-not-2 – Kevin Hooke Feb 17 '20 at 19:41
  • I am using 3 zookeepers now. But even after that, I am able to replicate the issue. – chikku Feb 18 '20 at 12:03
  • @KevinHooke So is it is necessary that a minimum number of brokers should be running to make Kafka stable (based on min-in-sync replicas)? In my case, when I try to start the nodes after making all nodes down, Kafka producer throwing error as above until I turned on the last active node. Even if I start other nodes, Kafka is not getting stable. Kafka producers stop throwing error only when I turned on the last active node. – chikku Feb 18 '20 at 12:22
  • @chikku yes, but the minimum number is related to replication factor for each topic and min.insync.replicas. If replication factor is 3, you need 3 brokers. If min.insync.replicas is 2, at most 1 broker can be down. If producer acks=all then any more than 1 down in this example and the Producer will not be able to send new messages to the cluster (because the Producer requires min 2 ISRs to acknowledge a message successfully written to the topic. This is described here https://kafka.apache.org/documentation/#brokerconfigs – Kevin Hooke Feb 18 '20 at 18:33
  • @KevinHooke So if I put min-in-sync as 2 and replication factor as 3 and if I turn all the broker down one by one and after that if I turn on any of two, the cluster should became stable, right? – chikku Feb 19 '20 at 12:08
  • @chikku I think the catch here is "if I turn all the broker down one by one and after that if I turn on any two" ... going down in this config 3 nodes can survive a failure of one node. This is different though than if you bring them all down and restart them one by one. I would guess in this case (why don't you try it and see) the cluster is not stable again until you have 3 nodes up. It think other issues would come into play here, like how the Zookeeper nodes establish a quorum. – Kevin Hooke Feb 19 '20 at 19:05

2 Answers2

1

You should add all broker addresses to bootstrap.servers properties in both producer and consumer configs. By this way you can connect to Kafka cluster in case of failure of one or more servers.

bootstrap.servers: A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).

H.Ç.T
  • 3,335
  • 1
  • 18
  • 37
-1

Add all the bootstrap server IPs in your properties file. If anyone of the server is down the Kafka consumer will try to connect to Kafka with an]]]other bootstrap servers. Add server 2 url in the below line:

EDIT:

spring.kafka.bootstrap-servers={SERVER1_HOST},{SERVER2_HOST}
Prog_G
  • 1,539
  • 1
  • 8
  • 22
  • I added SERVER2 address in listeners field of my server.properties of both brokers. But it is throwing error as follows `Socket server failed to bind to server2-ip:9093: Cannot assign requested address.` It is also failing the broker startup. – chikku Feb 17 '20 at 13:36
  • No. Listeners is the bind address for the current server only – OneCricketeer Feb 17 '20 at 14:45
  • add the java snippet where you are reading `server.properties` and also where you are creating the connection to Kafka. – Prog_G Feb 18 '20 at 05:01
  • 1
    I am using spring kafka. So I just have to provide this property in applicaion.property file spring.kafka.bootstrap-servers=server1-ip:9092,server2-ip:9092 – chikku Feb 18 '20 at 09:00