5

I am new to Confluent/Kafka and I want to find metadata information from kafka

I want to know

  1. list of producers
  2. list of topics
  3. schema information for topic

Confluent version is 5.0

What are classes (methods) that can give this information?
Are there any Rest API's for the same
Also is zookeeper connection necessary to get this information.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
user1393608
  • 1,299
  • 3
  • 16
  • 29
  • Hi All, I could list all topics through ZkUtils and get schema information using CachedSchemaRegistryClient.getAllSubjects(). Can someone tell me how is a topic related to getLatestSchemaMetadata(subjectName). I want to know given a topic name find its schema? – user1393608 Aug 28 '18 at 08:38

1 Answers1

4

1) I don't think that Kafka brokers are aware of producers that produce messages in topics and therefore there is no command line tool for listing them. However, an answer to this SO question suggests that you can list producers by viewing the MBeans over JMX.


2) In order to list the topics you need to run:

kafka-topics --zookeeper localhost:2181 --list

Otherwise, if you want to list the topics using a Java client, you can call listTopics() method of KafkaConsumer.

You can also fetch the list of topics through ZooKeeper

ZkClient zkClient = new ZkClient("zkHost:zkPort");
List<String> topics = JavaConversions.asJavaList(ZkUtils.getAllTopics(zkClient));


3) To get the schema information for a topic you can use Schema Registry API

In particular, you can fetch all subjects by calling:

GET /subjects HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

which should give a response similar to the one below: 

HTTP/1.1 200 OK
Content-Type: application/vnd.schemaregistry.v1+json

["subject1", "subject2"]

You can then get all the versions of a particular subject: 

GET /subjects/subject-name/versions HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

And finally, you can get a specific version of the schema registered under this subject

GET /subjects/subject_name/versions/1 HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

Or just the latest registered schema: 

GET /subjects/subject-name/versions/latest HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

In order to perform such actions in Java, you can either prepare your own GET requests (see how to do it here) or use Confluent's Schema Registry Java Client. You can see the implementation and the available methods in their Github repo.


Regarding your question about Zookeeper, note that ZK is a requirement for Kafka.

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
  • Thanks Giorgos. I want to know above information programmatically say through java client. So need API's for same. Step 1 : connect to zookeeper Step 2 : get list of producers Step 3 : get topics Step 4 : get their schemas – user1393608 Aug 21 '18 at 12:52
  • You should use `AdminClient.listTopics()`, actually. I think the Consumer method only lists those it subscribed to – OneCricketeer Aug 21 '18 at 18:34
  • @cricket_007 Java docs are not very clear to be honest `listTopics(): Get metadata about partitions for all topics that the user is authorized to view. This method will issue a remote call to the server.` – Giorgos Myrianthous Aug 22 '18 at 06:08
  • Hi Giorgos, I could list all topics through ZkUtils and get schema information using CachedSchemaRegistryClient.getAllSubjects(). Can someone tell me how is a topic related to getLatestSchemaMetadata(subjectName). I want to know given a topic name find its schema? – user1393608 Aug 27 '18 at 10:06