Use kafka-avro-console-producer with a schema already in the schema registry

Question

I would like to use the kafka-avro-console-producer with the schema registry. I have big schemas (over 10k chars) and I can't really past them as a command line argument. Besides that I'd like to use the schema registry directly so I can use a specific schema id.

I'm thinking about something like this, but it doesn't work:

kafka-avro-console-producer \
 --broker-list <broker-list> \
 --topic <topic>  \
 --property schema.registry.url=http://localhost:8081 \
 --property value.schema=`curl http://localhost:8081/schemas/ids/419`

What you wrote looks fine. What does the curl command output on its own? Is that cut off? — OneCricketeer, Jan 04 '20 at 01:11
It's not cut off, but I it looks like the inner payloads are strings rather than nested json objects (note the backslash before the quote) : `{"schema":"{\"type\":\"record\",\"name\":\"...`. Also my schema has got a doc that contains single and double quote as well as * which gets interpreted weirdly by bash. — 0x26res, Jan 06 '20 at 10:08
You need to download `jq`, then you can use `curl http://localhost:8081/schemas/ids/419 | jq .schema ` — OneCricketeer, Jan 06 '20 at 18:21
Interesting, but schema is still a string rather than a json object: `{\"type\":\"record\",\"name\":\"`. I would have to parse it and dump it as json again — 0x26res, Jan 06 '20 at 18:32
OK, so what I need is `curl http://localhost:8081/schemas/ids/419 | jq -r .schema`. I'm still having issue because * are expended to the content of the current directory. — 0x26res, Jan 06 '20 at 18:43
Where are you getting `*` from? Can you try `value.schema="$(curl http://localhost:8081/schemas/ids/419 | jq -r .schema)"`? Basically, you need to quote the `*` somehow — OneCricketeer, Jan 06 '20 at 19:53
It works, thanks. The `*` are coming from the java style comments in the schema, eg: `/** */` — 0x26res, Jan 07 '20 at 09:53

score 22 · Accepted Answer · edited Feb 12 '21 at 17:19

22

For the current version of the CLI tool

kafka-avro-console-producer \
 --broker-list <broker-list> \
 --topic <topic>  \
 --property schema.registry.url=http://localhost:8081 \
 --property value.schema.id=419

For older version

You'll need to extract the schema from the API request using jq, for example

value.schema="$(curl http://localhost:8081/schemas/ids/419 | jq -r .schema)"

edited Feb 12 '21 at 17:19

0x26res

11,925
11
54
108

answered Jan 07 '20 at 09:59

OneCricketeer

179,855
19
132
245

This works fine but weirdly enough, on one of my topics I get "InvalidConfigurationException: Schema being registered is incompatible with an earlier schema; error code: 409" which makes no sense since I am specifically not registering any schema but using an existing one. – cen Mar 05 '21 at 15:15
@cen Are you using a file or the ID property? I would only expect that error when using a file – OneCricketeer Mar 05 '21 at 15:31
ID property. That is what makes it weird. – cen Mar 05 '21 at 15:36

score 8 · Answer 2 · answered Sep 30 '20 at 19:12

8

You can use the property value.schema.id:

kafka-avro-console-producer \
 --broker-list <broker-list> \
 --topic <topic>  \
 --property schema.registry.url=http://localhost:8081 \
 --property value.schema.id=419

answered Sep 30 '20 at 19:12

mathiasfk

1,278
1
19
38

2

Thanks for your suggestion. Unfortunately it doesn't work with my version of `kafka-avro-console-producer`, I suspect it's too old and doesn't support it. – 0x26res Oct 23 '20 at 10:07
Where can I find all available properties for the `kafka-avro-console-producer` and `kafka-avro-console-consumer`? I failed to do that in the official docs – whatsupbros Feb 12 '21 at 16:05
1

Here: https://docs.confluent.io/platform/current/tutorials/examples/clients/docs/kafka-commands.html#produce-avro-records – 0x26res Feb 12 '21 at 17:17
@zza You should be able to pass `--help` to both commands – OneCricketeer Feb 12 '21 at 18:09
Yes, but `--help` doesn't give you the properties available to be configured, only the `--property` parameter itself is mentioned there, as well as in the examples page mentioned above only some properties were demonstrated. That is why I asked where, i.e. you found out about the `value.schema.id` property, because it was not mentioned anywhere – whatsupbros Feb 14 '21 at 10:15
@ZZa personally, I reviewed the PR for the feature, and the code is open source, so you could find it there... I'm not sure about the docs or help output – OneCricketeer Mar 05 '21 at 15:33

Use kafka-avro-console-producer with a schema already in the schema registry

2 Answers2

For the current version of the CLI tool

For older version

Linked