Stack Overflow
Stack Overflow

Questions tagged [confluent-schema-registry]

Schema Registry provides a store, backed by Kafka, for storing and retrieving Avro schemas. It also provides Avro serializers for Kafka clients.

Schema Registry is part of Confluent Open Source Stream Processing Platform.

It provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving Avro schemas. It stores a versioned history of all schemas, provides multiple compatibility settings and allows evolution of schemas according to the configured compatibility setting. It provides serializers that plug into Kafka clients that handle schema storage and retrieval for Kafka messages that are sent in the Avro format.

1076 questions
54
votes
2 answers

What is the use of __consumer_offsets and _schema topics in Kafka?

After setting up the Kafka Broker cluster and creating few topics, we found that the following two topics are automatically created by Kafka: __consumer_offsets _schema What is the importance and use of these topics ?
Anshul Patel
  • 845
  • 3
  • 8
  • 11
34
votes
5 answers

Kafka schema registry not compatible in the same topic

I'm using Kafka schema registry for producing/consuming Kafka messages, for example I have two fields they are both string type, the pseudo schema as following： {"name": "test1", "type": "string"} {"name": "test2", "type": "string"} but after…
Jack
  • 5,540
  • 13
  • 65
  • 113
31
votes
11 answers

Integrating Spark Structured Streaming with the Confluent Schema Registry

I'm using a Kafka Source in Spark Structured Streaming to receive Confluent encoded Avro records. I intend to use Confluent Schema Registry, but the integration with spark structured streaming seems to be impossible. I have seen this question, but…
Souhaib Guitouni
  • 860
  • 1
  • 7
  • 24
29
votes
5 answers

KafkaAvroSerializer for serializing Avro without schema.registry.url

I'm a noob to Kafka and Avro. So i have been trying to get the Producer/Consumer running. So far i have been able to produce and consume simple Bytes and Strings, using the following : Configuration for the Producer : Properties props = new…
scissorHands
  • 337
  • 1
  • 4
  • 11
20
votes
1 answer

How to pass parameters for a specific Schema registry when using Kafka Avro Console Consumer?

I am trying to use Confluent kafka-avro-console-consumer, but how to pass parameters for Schema Registry to it?
Joe
  • 11,983
  • 31
  • 109
  • 183
16
votes
1 answer

Kafka Streams - SerializationException: Unknown magic byte

I am trying to create a Kafka Streams Application which processes Avro records, but I am getting the following error: Exception in thread "streams-application-c8031218-8de9-4d55-a5d0-81c30051a829-StreamThread-1"…
R. B
  • 165
  • 1
  • 2
  • 8
15
votes
2 answers

Use kafka-avro-console-producer with a schema already in the schema registry

I would like to use the kafka-avro-console-producer with the schema registry. I have big schemas (over 10k chars) and I can't really past them as a command line argument. Besides that I'd like to use the schema registry directly so I can use a…
0x26res
  • 11,925
  • 11
  • 54
  • 108
15
votes
2 answers

KafkaAvroDeserializer does not return SpecificRecord but returns GenericRecord

My KafkaProducer is able to use KafkaAvroSerializer to serialize objects to my topic. However, KafkaConsumer.poll() returns deserialized GenericRecord instead of my serialized class. MyKafkaProducer KafkaProducer producer; …
Glide
  • 20,235
  • 26
  • 86
  • 135
12
votes
1 answer

Using AWS glue schema registry with confluent SerDe clients

For supporting schema registry on my MSK topic, I found two options - AWS Glue Schema Registry; and Confluent Schema Registry Since, Glue SR is fully managed by AWS, I would prefer to use that. However, my producer and consumer clients are written…
user12345
  • 121
  • 2
  • 5
12
votes
7 answers

TimeoutException: Timeout expired while fetching topic metadata Kafka

I have been trying to deploy Kafka with schema registry locally using Kubernetes. However, the logs of the schema registry pod show this error message: ERROR Server died unexpectedly: …
Cassie
  • 2,941
  • 8
  • 44
  • 92
12
votes
3 answers

Why use Avro with Kafka - How to handle POJOs

I have a spring application that is my kafka producer and I was wondering why avro is the best way to go. I read about it and all it has to offer, but why can't I just serialize my POJO that I created myself with jackson for example and send it to…
adpap
  • 209
  • 3
  • 10
12
votes
2 answers

How to install schema registry

I am looking options to install confluent schema registry, is it possible to download and install registry alone and make it work with existing kafka setup ? Thanks
Tilak
  • 323
  • 1
  • 5
  • 18
11
votes
3 answers

Schema registry on AWS

I'm evaluating kinesis as replacement for kafka. One of the things I'm missing is Schema registry equivalent solution. In particular I need: schema upgrade - validate compatibility with the previous version version avro schemas in a similar way as…
czajek
  • 714
  • 1
  • 9
  • 23
11
votes
2 answers

Unknown magic byte with kafka-avro-console-consumer

I have been trying to connect with kafka-avro-console-consumer from Confluent to our legacy Kafka cluster, which was deployed without Confluent Schema Registry. I provided schema explicitly using properties like: kafka-console-consumer…
Martin Macak
  • 3,507
  • 2
  • 30
  • 54
10
votes
2 answers

How AWS MSK and Confluent Schema Registry and Confluent Kafka connect recommended to use together?

We are planning to use AWS MSK service for Managed Kafka and Schema Registry and Kafka Connect services from Confluent together to run our connectors (Elasticsearch Sink Connector). We have planned to run Schema Registry and Connectors in EC2. As…
Suresh Sanghani
  • 131
  • 1
  • 8
1
2 3
…
71 72