1

Im trying to getting data from a kafka, but that data is in thrift.

If I want to transform it to json in order to change some values, how can I do it?

Is there any processor that transforms from thift to json?

Thanks

Bentipe
  • 423
  • 3
  • 17
  • Can You made some negotiation with server administrator / developer? Thrift package has JSON options too – Jacek Cz Jun 20 '17 at 11:00
  • hey Jacek, yeah, its one of the options that we have, but they prefer it to be in thrift, so I'm looking all the options. Thanks – Bentipe Jun 20 '17 at 12:09
  • What Jacek is saying that Thrift already supports a [`TJSONProtocol`](https://github.com/apache/thrift/blob/master/lib/java/src/org/apache/thrift/protocol/TJSONProtocol.java) (in fact there are two, but one is write-only) – JensG Jun 21 '17 at 14:17

1 Answers1

1

As Jacek pointed out, it would be ideal to consume JSON directly from Kafka, but in the event this is not possible, here are a few options:

  1. Perform a command-line transformation -- write a simple compiled Java client that accepts Thrift input and converts it using a JSON serializer to JSON. Invoke this client using the ExecuteStreamCommand processor. This has the benefit of not bringing any Thrift dependencies into NiFi's JVM.
  2. Perform the transformation in NiFi using ExecuteScript -- write the Thrift transform directly in a Groovy script (you can use pure Java here as well) and either reference the file or paste the script body into the processor. This will require the Thrift libraries to be available in the processor's module directory.
  3. Write a custom processor -- following the instructions in the Apache NiFi Developer Guide, you can write your own custom processor and bundle the NAR together and drop it into your NiFi instance. You'll have access to the complete processor lifecycle, framework API, controller services, unit test harness, etc.

Good luck.

Andy
  • 13,916
  • 1
  • 36
  • 78
  • Thanks Andy, yeah the best option is to be directly from the server in Json, It will take the less time. Thanks. – Bentipe Jun 20 '17 at 12:10