1

I've started learning some big data tools for a new project, and right now I'm on Kafka and Zookeeper.

I have them both install on my local machine, and I can start them up and start producing and consuming messages just fine. Now, I want to try it having two machines, one with a kafka broker, zookeepr and a producer, and the other with a consumer. Lets call them Machine A and Machine B.

Machine A has runs the Zookeeper server, the broker and a producer. Machine B runs a consumer. From what I think I understand, I should be able to setup the consumer to listen to a topic from the producer on Machine A, using Zookeeper. Since both machines are on the same network (i.e. my local home network), I thought I could change the kafka broker server.properties to use my static ip address for Machine A, and then have the consumer on Machine B connect to it.

My problem, is that zookeeper keeps spinning up on localhost, and connecting to 0.0.0.0/0.0.0.0:2181 so when my broker tries to connect to it using my static ip address (i.e 192.168.x.x), it times out. I have looked all over for a solution, but I cannot find anything that tells me how to configure the Zookeeper sever to start on a different ip address.

Maybe my understanding of these technologies is simply wrong, but I thought this would be a fairly simple thing to do. Does anyone know any way to resolve this? Or else if I'm doing it completely wrong, what is the correct approach

Eoin
  • 330
  • 1
  • 3
  • 15
  • *consumer to listen to a topic from the producer on Machine A, using Zookeeper* -- Consumers should be setup to listen via Kafka, not Zookeeper – OneCricketeer Aug 17 '18 at 23:54

2 Answers2

3

zookeeper keeps spinning up on localhost, and connecting to 0.0.0.0/0.0.0.0:2181

Well, that is the bind address.

You need to also (preferably) have a static IP for Zookeeper, then set zookeeper.connect within the server.properties file of Kafka to reach to that other machine's external address.

From the Zookeeper configuration file, you would make sure you have the myid file and have a line in the property file that looks like this (without the double brackets)

server.{{ myid }}={{ ip_address }}:2888:3888

You wouldn't find this in the Kafka documentation, but it is in the Zookeeper documentation

However, if Kafka and Zookeeper are on the same machine, this isn't necessary.

Your external consumer should be setting bootstrap.servers property and the Kafka IP address(es) w/ port 9092.

Your problem might me related instead to the advertised.listeners setting within Kafka.

For example, start with listeners=PLAINTEXT://:9092

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • So in my server.properties file I have `listeners=PLAINTEXT://myStaticIPAddress:9092` and the `zookeeper.connect=localhost:2181` and when I start the consumer on the other machine I do start it up pointing the `bootstrap.server` property to the other machines static ip address. But it will tell me that the brooker may be unavailable.. I've also checked the `server.properties` file and I don't have a `advertised.listeners` property anywhere in there. Could this be the reason? – Eoin Aug 18 '18 at 09:43
  • Start by setting the listeners property to `PLAINTEXT://0.0.0.0:9092`, then make sure Kafka is actually running and then from the other machine, try Netcat or telnet commands to test the port connection – OneCricketeer Aug 18 '18 at 15:42
  • Ok so when I set the listeners property to `PLAINTEXT://0.0.0.0:9092`, I get an error starting kafka saying `Cannot use the nonroutable meta-address 0.0.0.0. Use a routable address` By the looks of it is has to `localhost` or the static ip – Eoin Aug 19 '18 at 20:55
  • You can also use `PLAINTEXT://:9092` – OneCricketeer Aug 19 '18 at 21:00
  • That has worked! Thank you so much! I guess just so I'm clear, does `PLAINTEXT://:9092` mean that the broker is listening for any connections from any server on that port? That part is still a bit unclear to me – Eoin Aug 20 '18 at 21:12
  • I think so ;) I just copied that from our running brokers – OneCricketeer Aug 20 '18 at 21:43
  • Unfortunately, while this solution works on my home network, it does not work at my office. Once I start up the broker, it still tells me that the `broker may be unavailable`, and I cannot connect any consumers to it. Any final suggestions? It could be our internal network, but I'm not really sure – Eoin Aug 21 '18 at 08:53
  • If it works in one network, but not another, I would think that's an external firewall problem – OneCricketeer Aug 21 '18 at 13:10
2

As of Zookeeper 3.3.0 (see Advanced Configuration):

clientPortAddress : New in 3.3.0: the address (ipv4, ipv6 or hostname) to listen for client connections; that is, the address that clients attempt to connect to. This is optional, by default we bind in such a way that any connection to the clientPort for any address/interface/nic on the server will be accepted

So you could use:

clientPortAddress=127.0.0.1
Adrian
  • 3,321
  • 2
  • 29
  • 46