Is it possible to create a topic with Kafka Rest Proxy?

Question

I am trying to create a topic from Kafka Rest Proxy but I don't see any documentation for it. I'm hoping there is a way to do it so I don't need to programmatically create a topic differently than I do all other communication. Does anyone know if it possible?

I don't see any documentation for it here: http://docs.confluent.io/1.0/kafka-rest/docs/api.html.

Thanks for your help.

score 13 · Answer 1 · answered Apr 27 '20 at 21:02

With the version 3 of the REST Proxy API it is now possible.

According to the Confluent REST Proxy API Reference the creation of a topic is possible with the REST Proxy API v3 that is currently available as a preview feature.

An example of a topic creation request is presented below and documented here:

POST /v3/clusters/cluster-1/topics HTTP/1.1
Host: kafkaproxy.example.com
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json

{
  "data": {
    "attributes": {
      "topic_name": "topic-1",
      "partitions_count": 2,
      "replication_factor": 3,
      "configs": [
        {
          "name": "cleanup.policy",
          "value": "compact"
        }
      ]
    }
  }
}

Using curl:

curl -X POST -H "Content-Type: application/vnd.api+json" -H "Accept: application/vnd.api+json" \
          --data '{"data":{"attributes": {"topic_name": "topic-1", "partitions_count": 2, "replication_factor": 1, "configs": [{"name": "cleanup.policy","value": "compact"}]}}}' \
          "http://localhost:8082/v3/clusters/<cluster-id>/topics"

where the cluster-id can be identified using

curl -X GET -H "Accept: application/vnd.api+json" localhost:8082/v3/clusters

score 2 · Accepted Answer · edited May 23 '17 at 11:47

2

No. The problem is there's no protocol support for creating topics (not yet, at least). See my previous question here, and read the comments.

edited May 23 '17 at 11:47

Community

1
1

answered Jun 11 '16 at 14:45

David Griffin

13,677
5
47
65

score 2 · Answer 3 · edited May 23 '17 at 12:02

2

There is no explicit admin API to create a topic in REST proxy yet, but if the Kafka broker property auto.crate.topics.enable is set to true the topic is created when you first post to it, including the case when the post comes from the proxy. More info here How to create topics in apache kafka?

edited May 23 '17 at 12:02

Community

1
1

answered May 11 '17 at 19:47

azary

61
1
3

score 2 · Answer 4 · answered Jun 08 '21 at 12:28

It is possible with the latest version. See here: https://docs.confluent.io/platform/current/tutorials/examples/clients/docs/rest-proxy.html#basic-producer-and-consumer

From the docs:

KAFKA_CLUSTER_ID=$(docker-compose exec rest-proxy curl -X GET \
     "http://localhost:8082/v3/clusters/" | jq -r ".data[0].cluster_id")

Verify the parameter KAFKA_CLUSTER_ID has a valid value. For the example in this tutorial, it is shown as lkc-56ngz, but it will differ in your output.

echo $KAFKA_CLUSTER_ID
Create the Kafka topic test1 using the AdminClient functionality of the REST Proxy API v3.

docker-compose exec rest-proxy curl -X POST \
     -H "Content-Type: application/json" \
     -d "{\"topic_name\":\"test1\",\"partitions_count\":6,\"configs\":[]}" \
     "http://localhost:8082/v3/clusters/${KAFKA_CLUSTER_ID}/topics" | jq .

score 0 · Answer 5 · answered Apr 21 '20 at 20:44

0

You can create a topic with Azure HdInsight Kafka REST proxy. https://learn.microsoft.com/en-us/rest/api/hdinsight-kafka-rest-proxy/admin

More details here: https://learn.microsoft.com/en-us/azure/hdinsight/kafka/rest-proxy

answered Apr 21 '20 at 20:44

Linda

78
7

Is it possible to create a topic with Kafka Rest Proxy?

5 Answers5

Linked