3

I am going to implement a kafka cluster consisting of 3 machines, one for zookeeper and other 2 as brokers. I have about 6 consumer machines and about hundred producers.

Now if one of the broker fails data loss is avoided thanks to replication feature. But what if zookeeper fails and the same machine cannot be started? I have several questions:

  1. I noticed that even after zookeeper failure producers continued to push messages in designated broker. But they could no longer be retrieved by consumers. Because Consumers got unregistered. So in this case is data lost permanently?
  2. How to change zookeeper ip in broker config in run time? Will they have to be shutdown to change zookeeper ip?
  3. Even if new zookeeper machine is somehow brought into the cluster previous would the previous data be lost?
Shades88
  • 7,934
  • 22
  • 88
  • 130

1 Answers1

0

Running only one instance of Zookeeper is not fault-tolerant and the behavior cannot be predicted. According to HBase reference, you should setup an ensemble with at least 3 servers.

Have a look at the official documentation page: Zookeeper clustered setup.

Markon
  • 4,480
  • 1
  • 27
  • 39
  • so suppose in the setup mentioned above I also add 2 more zookeeper machines and if one of them fails then any one of remaining two can be connected? Will they have all the up-to-date data? – Shades88 Oct 27 '15 at 07:56
  • If you add two more machines, then only one can fail. Have a look at this: http://stackoverflow.com/questions/4228227/what-does-2n-1-quorum-mean Yes, data will be in sync, because these machines are replicated. – Markon Oct 27 '15 at 09:07
  • on development environment I took two machines and added this config in each machine's zookeeper.config file `initLimit=5 syncLimit=2 server.1=x.x.x.x:2888:3888 server.2=localhost:2888:3888` Server.2 being the server from which I am about to start zookeeper. So when I hit the startup script, I get error that server.1 with x.x.x.x:3888 cannot be connected. – Shades88 Oct 27 '15 at 10:09
  • Well, have you tried to debug first? e.g., telnet x.x.x.x 3888 ? – Markon Oct 27 '15 at 10:12
  • Ya sorry, I didn't mention everything in previous comment. This is the very thing I am surprised about. port 3888 is not being run by any process on either machines. So how are they supposed to connect to that port? Do I have to run anything prior to starting zookeeper server? – Shades88 Oct 27 '15 at 10:44
  • I have just noticed that you don't have a clientPort specified. Have you tried following a tutorial like http://myjeeva.com/zookeeper-cluster-setup.html? – Markon Oct 27 '15 at 11:21
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/93481/discussion-between-shades88-and-markon). – Shades88 Oct 27 '15 at 11:24
  • Yes, I have mentioned client port in config. Here's entire config `dataDir=/tmp/zookeeper # the port at which the clients will connect clientPort=2181 # disable the per-ip limit on the number of connections since this is a non-production config maxClientCnxns=0 initLimit=5 syncLimit=2 server.1=x.x.x.x:2888:3888 server.2=localhost:2888:3888` – Shades88 Oct 27 '15 at 12:14
  • Thanks a lot, you pointed me in right direction. I was using zookeeper utility provided by kafka but one cannot create zookeeper cluster using that. That's why I was facing all above problems. I downloaded zookeeper separately started 3 instances, now everything works find – Shades88 Oct 28 '15 at 09:35
  • 1
    Great! I am glad it helped :) – Markon Oct 28 '15 at 11:48