0

I followed this link to install Kafka + Zookeeper. It all works well, yet I am setting up Kafka + Zookeeper on 2 servers.

I have setup the kafka/config/server.properties to have:

  • Server 1: broker.id = 0
  • Server 1: zookeeper.connect = localhost:2181,99.99.99.91:2181
  • Server 2: broker.id = 1
  • Server 2: zookeeper.connect = localhost:2181,99.99.99.92:2181

I am wondering the follwing:

  • When I publish a topic, does it go to both Instances, or just the server it's loaded on?
  • In order to use multiple servers like this, would I be required to use something like HAProxy with say 3 servers?
  • Is there anything important I am missing with using 3 servers?

Thanks for any answers.

JREAM
  • 5,741
  • 11
  • 46
  • 84

1 Answers1

2

At the very beginning, I would be assuming that the instance mentioned here refers to a Kafka server instance.

Question #1: Information for the new-created topics will be stored in Zookeeper and some key information will be loaded into all brokers' metadata cache buffer.

Question #2: No need to configure any kind of proxy servers for Kafka cluster. It's self managed to implement the fail-over and load-balancing.

Question #3: Assigning an unused id and data directory like what you did for those two brokers is enough.

At last but not least, due to the fact that ZAB requires a majority, it is best to use an odd number of machines as the zookeeper quorum.

amethystic
  • 6,821
  • 23
  • 25
  • Thanks! i understand the endpoint to Server 1, and it goes to 2 or 3 with Kafka/ZK. But if server 1 had an outage, then the endpoint for all Kafka/ZK would have a single point of failure, that's why I was wondering how people skirt around that, maybe a DNS heartbeat or something? – JREAM Dec 07 '16 at 16:46
  • One of the brokers will be elected as a controller which is in charge of topic/partition management. If controller is crashed, Kafka initiates an automatic election for a new controller. Users do not have to worry about anything during this process. – amethystic Dec 07 '16 at 22:58