1

I have created a cluster (google cloud) with 3 nodes. Zookeeper is running on all nodes and I have started Kafka on one of the nodes. I can communicate (publish/consume) from any machine on the cluster but when i try to connect from a remote machine i get a NoBrokersAvailable exception.

I have opened ports in the firewall for testing and I have tried messing around with advertised_host and port in the Kafka config but I am unable to connect.

What is the expected configuration? - I would have expected, having suitable defaults, that my configuration would work in both the internal and remote case but it does not. I am not sure what part of the configuration of zookeeper/kafka would allow me to tweak this. What is to be done?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
ismisesisko
  • 151
  • 3
  • 10
  • 1
    There are just way too many things it could be -- the machines could be running `iptables`, for example. Or maybe Zookeeper and Kafka are listening on non-routable IP addresses (e.g. `10.x.x.x1` or `192.168.x.x`) or who knows what it could be. – David Griffin Mar 26 '16 at 21:01
  • Thanks David. Noted. I will look into the networking and if i (ever) get to the bottom of it I will note it here for the record. – ismisesisko Mar 27 '16 at 11:39
  • update but no joy: I simplified matters. i used one node with zookeeper and kafka. i used the confluent kafka distro which provides some benefits. i tested locally and remotely with the REST server which verifies zookeeper is configured properly and that I can externally connect to said rest server. When i start up Kafka, i note the ouput line:> Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(cluster-3-m,9092,PLAINTEXT). Where it says 'cluster-3-m', i have primarily been using the PUBLIC IP. Clearly, broker is started and bound - python client says otherwise! – ismisesisko Mar 27 '16 at 13:23
  • 1
    Are you referencing the broker using the fully qualified host name? When the brokers respond to the clients they'll reference brokers by their fqdn which might not be how you're set up to access them on subsequent internal calls. – Chris Gerken Mar 28 '16 at 15:55

1 Answers1

1

Set the advertised.listeners=PLAINTEXT://<broker_ip>:9092 in server.properties file, and make sure this advertised address ingress is allowed through the GCP VPC firewall. Restart kafka-server and producer as well as consumer (whichever or if both is running)

Please check my answer to the same problem in another thread

NoBrokersAvailable: NoBrokersAvailable-Kafka Error

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
somnathchakrabarti
  • 3,026
  • 10
  • 69
  • 92