How do you set Kafka Rest Proxy Key Schema with JSON values?

Question

I'm using Kafka 1.1 and Kafka Rest Proxy 4.1.2. I have been keying records from inside Kafka streams using String keys. I want to use Rest Proxy to insert records to be joined, but the keys are getting escaped quotation marks put around them.

I'm sending a POST request to /topics/{someTopic} with Content-Type: application/vnd.kafka.json.v2+json which causes the issue.

With Content-Type: application/vnd.kafka.avro.v2+json and key_schema type: string, the keys do not have extra quotation marks around them, but I'd rather send json values.

This is what I am sending to the /topics endpoint.

{
    "records": [
        { 
            "key": "abc", 
            "value": {"animal": "dog"} 
        }
    ]
}

When I stream the data in Kafka streams, the key is coming out as \"abc\", and obviously isn't being joined with records with string keys abc.

Is there a way to specify a key schema with json values, so that my keys don't get escaped quotation marks around them?

score 2 · Answer 1 · answered May 07 '19 at 15:35

2

As per the Kafka docs, the format of the message is correct for json. I think,you should try with the following message header.

   Content-Type: application/vnd.kafka.json.v2+json
   Accept: application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json

I would suggest, you to go through the following Kafka document.

https://docs.confluent.io/current/kafka-rest/api.html

answered May 07 '19 at 15:35

Rohit Yadav

2,252
16
18

Thanks, @Rohit. That is the same Content-Type I was sending. I added that Accept header, and it still adds the escaped quotation marks around my key. – OldDave2019 May 07 '19 at 15:46

score 2 · Accepted Answer · answered May 07 '19 at 19:14

When using the Content-Type: application/vnd.kafka.json.v2+json header, a JSON key will have escaped quotes around all strings so that it properly deserializes in a streams app. When using a simple key, strings do seem to be escape quoted while numeric keys are unmodified.

The Content-Type: application/vnd.kafka.binary.v2+json will produce your key value pairs exactly as you give them, without adding escaped quotes to string keys. You just need to base64 encode your keys and values.

Your example body becomes:

{
    "records": [{
        "key": "YWJj",
        "value": "eyJhbmltYWwiOiJkb2cifQ=="
    }]
}

How do you set Kafka Rest Proxy Key Schema with JSON values?

2 Answers2