11

I'm evaluating kinesis as replacement for kafka. One of the things I'm missing is Schema registry equivalent solution. In particular I need:

  • schema upgrade - validate compatibility with the previous version
  • version avro schemas in a similar way as schema registry does

What are the options to handle 2 above? The only thing that I found was glue catalogue, but it doesn't seem to
At the end I also want to use firehose (output to redshift), but from what I understand this is not possible and will require writing custom lambda.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
czajek
  • 714
  • 1
  • 9
  • 23
  • Kinesis only stores bytes. If you want to store Avro schemas somewhere that doesn't necessarily require Kafka, you can try Hortonwork's Registry – OneCricketeer Nov 15 '18 at 23:01
  • Did you ever find a solution to this? This is exactly what I want to do. – mp252 Sep 30 '20 at 13:27

3 Answers3

2

AWS just launched Schema Registry functionality for AWS Kinesis, https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html

blacktooth
  • 176
  • 3
  • 16
0

https://docs.aws.amazon.com/kinesisanalytics/latest/dev/sch-dis-ref.html Is where it gets us - still not as complete / elegant as what the kafka ecosystem by confluent bundle, say is. I am doubting how compatible schema work here too.

Might deviate from your ques - but would you consider MSK, or deploying kafka on AWS (although an overhead for managing it) ?

khanna
  • 718
  • 10
  • 24
0

AWS Glue provide option for schema registry which can be used with AVRO data format. at this moment glue only support java as producer and consumer. it is quite easy to use

Deepak Gupta
  • 387
  • 2
  • 17