kafka configuration + zookeeper cli + get the right info for kafka host

Question

As all know when is need to print the kafka broker id’s we can use the following cli

zookeeper-shell.sh zoo_server1:2181 <<< "ls /brokers/ids"

this cli print the following

WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[1018, 1017, 1016]

Its means that we have kafka with id’s

1018 
1017
1016

But our kafka names are

Kafka_confluent01
Kafka_confluent02
Kafka_confluent03

So how to know which kafka broker id ( 1018 , 1017 , 1016 ) is belong to the real host ( Kafka_confluent01 / Kafka_confluent02 / Kafka_confluent03 )

score 2 · Answer 1 · answered May 11 '20 at 13:57

You can use kafkacat for this with the -L operator:

$ kafkacat -b kafka-01.foo.bar:9092 -L
Metadata for all topics (from broker 1: kafka-01.foo.bar:9092/1):
 3 brokers:
  broker 2 at kafka-02.foo.bar:9092
  broker 3 at kafka-03.foo.bar:9092
  broker 1 at kafka-01.foo.bar:9092 (controller)

score 1 · Accepted Answer · answered May 11 '20 at 13:19

1

You can get the list of brokers dynamically, using the following code.

public class KafkaBrokerInfoFetcher {

    public static void main(String[] args) throws Exception {
        ZooKeeper zk = new ZooKeeper("localhost:2181", 10000, null);
        List<String> ids = zk.getChildren("/brokers/ids", false);
        for (String id : ids) {
            String brokerInfo = new String(zk.getData("/brokers/ids/" + id, false, null));
            System.out.println(id + ": " + brokerInfo);
        }
    }
}

After running the code, you will get the broker id and corresponding host.

1: {"jmx_port":-1,"timestamp":"1428512949385","host":"192.168.0.11","version":1,"port":9093}
2: {"jmx_port":-1,"timestamp":"1428512955512","host":"192.168.0.11","version":1,"port":9094}
3: {"jmx_port":-1,"timestamp":"1428512961043","host":"192.168.0.11","version":1,"port":9095}

answered May 11 '20 at 13:19

Rohit Yadav

2,252
16
18

is it possible to write it in shell-script / bash / python , ( because we want to run it on linux machine ) ? ( +1 ) – jessica May 11 '20 at 13:29
You can create this and invoke using the shell-script. this will help you on this.https://stackoverflow.com/questions/32804020/create-a-shell-script-to-run-a-java-program-on-linux – Rohit Yadav May 11 '20 at 13:32
yes but how to create jar file from your code , can you give the all steps? – jessica May 11 '20 at 13:36
Please use the following link https://stackoverflow.com/questions/17981958/how-to-create-jar-file-using-eclipse – Rohit Yadav May 11 '20 at 13:40

kafka configuration + zookeeper cli + get the right info for kafka host

2 Answers2