List all Kafka 0.10 topics using - -zookeeper flag without access to Zookeeper

Question

I'm using kafka 0.10 without zookeeper. I want to get kafka topics list. This command is not working since we're not using zookeeper: bin/kafka-topics.sh --list --zookeeper localhost:2181. How can I get the same output without zookeeper?

```/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181``` — Adiii, Nov 05 '20 at 08:37

score 171 · Accepted Answer · answered Jun 08 '17 at 12:52

171

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one.

If you do not want to install and have a separate zookeeper server, you can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance.

Starting the single-node Zookeeper instance:

bin/zookeeper-server-start.sh config/zookeeper.properties

Starting the Kafka Server:

bin/kafka-server-start.sh config/server.properties

Listing the Topics available in Kafka:

bin/kafka-topics.sh --list --zookeeper localhost:2181

answered Jun 08 '17 at 12:52

Daniccan

2,755
1
18
27

2

I need a solution without zookeeper. Is there any other command than this to list all topics without zookeeper? – SSR Jun 12 '17 at 10:07
7

I don't think its possible to even use kafka without zookeeper as the state of kafka is stored in zookeeper. How were you able to achieve it ? – Daniccan Jun 12 '17 at 11:05
If you are looking for a command to do it then https://stackoverflow.com/questions/40034074/kafka-topics-sh-delete-topic-testtopic-is-not-working-for-kafka-v-0-10 – Peru Feb 10 '19 at 14:49
If your zookeeper server is a named host, you'll need to give the host name instead of `localhost` e.g. `./kafka-topics.sh --list --zookeeper zookeeper:2181`. Or depending on your configuration you may need to use the IP address e.g. `./kafka-topics.sh --list --zookeeper 10.10.10.180:2181` – Chris Halcrow Feb 10 '20 at 23:44
You can find more details here https://reachmnadeem.wordpress.com/2020/08/30/kafka-2-6-up-and-running-in-windows-10/ – craftsmannadeem Aug 30 '20 at 07:42
If you are on windows, use `bin/windows` instead of `bin` – Jahir Jan 05 '21 at 16:48

score 63 · Answer 2 · answered Jun 14 '19 at 09:26

63

Kafka 2.2 and up

Newer versions of Kafka no longer requires ZooKeeper connection string to list topics, but can directly go via the Kafka brokers. kafka-topics.sh is provided in the bin/ folder when downloading Kafka. To list topics, do the following:

bin/kafka-topics.sh --list --bootstrap-server <BROKER-LIST>

answered Jun 14 '19 at 09:26

Paul

770
1
7
12

3

assuming the OP meant "without the zookeeper cli option" - which is likely, since kafka doesn't work without zookeeper - this answer should be the accepted answer for Kafka 2.2+. The cli docs now indicate that --zookeeper is deprecated in favor of --bootstrap-server. – Luke W Aug 10 '19 at 15:38
This is actually the best way to do it if you're working in AWS with their MSK. Otherwise you need to open up your ZooKeeper service, which can be a potential security issue if done incorrectly. – Michael Robinson Oct 04 '19 at 14:32
@MichaelRobinson granted that you use MSK with v2.2.1 and not v1.1.1. – Paul Oct 04 '19 at 14:54
Yes, my comment was localised to this post. – Michael Robinson Oct 04 '19 at 16:24
this must be the accepted answer as of 2022 – nomadSK25 Aug 17 '22 at 16:06

score 31 · Answer 3 · answered Nov 22 '18 at 20:16

31

For dockerized kafka/zookeeper

docker ps

find you zookeeper container id

docker exec -it <id> bash

cd bin

./zkCli.sh

ls /brokers/topics

answered Nov 22 '18 at 20:16

MagGGG

19,198
2
29
30

4

`./zkCli.sh` does not exists... in the zookeeper docker image – Kristoffer Tølbøll Dec 13 '20 at 18:57

score 20 · Answer 4 · answered Dec 14 '17 at 01:22

20

to see that topic if we run the list topic command:

$ bin/kafka-topics.sh --list --zookeeper localhost:2181

To check if the data is landing in Kafka:

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic twitterstream --from-beginning

answered Dec 14 '17 at 01:22

Enggar R Hariawan

379
3
5

score 16 · Answer 5 · edited Jun 08 '21 at 18:51

16

We can use the following command:

kafka-topics.sh --list --bootstrap-server localhost:9092

to list down all topics

edited Jun 08 '21 at 18:51

valdeci

13,962
6
55
80

answered Dec 16 '20 at 15:01

Rajnish Kumar

2,828
5
25
39

score 7 · Answer 6 · edited Jun 16 '21 at 22:46

Commands:

To start the kafka:

 $ nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &

To list out all the topic on on kafka:

$ bin/kafka-topics.sh  --list --zookeeper localhost:2181

To check the data is landing on kafka topic and to print it out:

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic your_topic_name --from-beginning

score 4 · Answer 7 · answered Jun 18 '17 at 12:39

Kafka requires zookeeper and indeed the list of topics is stored there, hence the kafka-topics tool needs to connect to zookeeper too. kafka-clients apis in the newer versions no longer talk to zookeeper directly, perhaps that's why you're under the impression a setup without zookeeper is possible. It is not, as kafka relies on it internally. For reference see: http://kafka.apache.org/documentation.html#quickstart Step 2:

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one

Michael Heil · Answer 8 · 2020-09-10T19:58:09.230

4

Using Confluent's REST Proxy API:

curl -X GET -H "Accept: application/vnd.kafka.v2+json" localhost:8082/topics

where localhost:8082 is Kafka Proxy address.

edited Sep 10 '20 at 19:58

answered Sep 10 '20 at 19:38

Michael Heil

16,250
3
42
77

score 4 · Answer 9 · answered Feb 24 '22 at 05:31

The --zookeeper is no longer a recognized option in version 3.1.0

Hence, we can use

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

or

curl -X GET -H "Accept: application/vnd.kafka.v2+json" localhost:9092/topics

to get the topics per broker level

score 3 · Answer 10 · answered Dec 14 '17 at 06:26

The Kafka clients no longer require zookeeper but the Kafka servers do need it to operate.

You can get a list of topics with the new AdminClient API but the shell command that ship with Kafka have not yet been rewritten to use this new API.

The other way to use Kafka without Zookeeper is to use a SaaS Kafka-as-a-Service provider such as Confluent Cloud so you don’t see or operate the Kafka brokers (and the required backend Zookeeper ensemble).

For example on Confluent Cloud you would just use the following zookeeper free CLI command:

ccloud topic list

score 2 · Answer 11 · edited Sep 24 '21 at 18:15

2

To read messages you should use:

kafka-console-consumer.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic messages --from-beginning

--bootstrap-server is required attribute. You can use only single kafka1:9092 node.

edited Sep 24 '21 at 18:15

OneCricketeer

179,855
19
132
245

answered Dec 17 '18 at 20:41

Dzintars

1,410
21
29

score 1 · Answer 12 · answered Jun 08 '17 at 07:52

1

Kafka is a distributed system and needs Zookeeper. you have to start zookeeper too. Follow "Quick Start" here : https://kafka.apache.org/0100/documentation.html#quickstart

answered Jun 08 '17 at 07:52

Calden

76
5

score 1 · Answer 13 · answered Jul 25 '17 at 06:08

1

Zookeeper is required for running Kafka. zookeeper is must. still if you want to see topic list without zookeeper then you need kafka monitoring tool such as Kafka Monitor Tool, kafka-manager etc.

answered Jul 25 '17 at 06:08

Amol Suryawanshi

86
4

score 1 · Answer 14 · answered Oct 24 '19 at 12:28

You need to start the zookeeper server first. So first go to kafka/bin/windows and run

zookeeper-server-start.bat ../../config/zookeeper.properties

then in the same folder with a new cmd windows start the kafka servers by running

kafka-server-start.bat ../../config/server.properties

Note: if you starting it for the first time then there are certain changes to be made in these files

then inside kafka/bin/windows run

kafka-topics.bat --zookeeper localhost:2181 --list

to list down all the topics existing.

score 0 · Answer 15 · answered Dec 19 '19 at 16:37

0

You have a stale version of the package with commands that no longer accept zookeeper but rather bootstrap-server as the connection. Confluent will then connect with Zookeeper internally.

https://www.confluent.io/download/ (5.3 or greater)

answered Dec 19 '19 at 16:37

Rick O'Shea

1,410
19
15

score 0 · Answer 16 · edited Jun 16 '21 at 22:44

0

You can try using the below two command and list all Kafka topic

 - bin/kafka-topics.sh --describe --zookeeper 192.168.0.142:2181,192.168.9.115:2181,192.168.4.57:2181

- bin/kafka-topics.sh --zookeeper 192.168.0.142:2181,192.168.9.115:2181,192.168.4.57:218 --list

edited Jun 16 '21 at 22:44

valdeci

13,962
6
55
80

answered Aug 31 '20 at 08:14

Kewal Agarwal

1
4

score 0 · Answer 17 · answered Feb 09 '22 at 21:58

0

Try this, I was having issues with the --zookeeper flag.

bin/kafka-topics.sh --list --bootstrap-server my-cluster-kafka-bootstrap:9092

answered Feb 09 '22 at 21:58

Kevin Baker

1,187
3
12
22

score 0 · Answer 18 · answered Aug 06 '23 at 07:09

0

For windows

.\bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092

For linux or mac platform look for kafka-topics.sh file and then run

answered Aug 06 '23 at 07:09

KESHAV KUMAR

140
1
7

List all Kafka 0.10 topics using - -zookeeper flag without access to Zookeeper

18 Answers18

Linked