0

I have a following use case :

  • parse the json from stream (kafka topic)
  • extract some fields (likely 35 out of 100 fields)
  • Build a json out of those fields
  • Publish it to pub/sub for further processing

My implementation is very much java language bound. Can anyone suggest optimal solution for this ? and why is it optimal ?

For json parsing, I am thinking of https://bolerio.github.io/mjson/

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Snehal S
  • 21
  • 2
  • There's an example of building your own Json Serializer/Deserializer here https://stackoverflow.com/questions/50524867/parsing-json-data-using-apache-kafka-streaming – Kevin Hooke Jan 21 '20 at 01:46
  • 1
    Alternatively Spring Kafka will handle it for you, transparently mapping json to pojos and back https://spring.io/projects/spring-kafka – Kevin Hooke Jan 21 '20 at 01:47
  • I don't want json to be bound to pojos. That's the reason I am asking for any sort of dynamic transformation. – Snehal S Jan 21 '20 at 18:10

1 Answers1

0

Kafka includes Jackson JSON library and includes its own JSON Deserializer that returns a JsonNode class

Alternatively,as listed in the comments, you can use higher level frameworks such as Spring, Vertx, Quarkus, etc to build Kafka consumers

For the listed use case, I would opt for Spark, Flink, or NiFi for integration with PubSub. Each also offering JSON proessing, with NiFi being more advanced in that it can do JSONPath

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245