Convert Arrow Flight Response to JSON

Question

I have been looking into Dremio and Arrow Flight recently and intrigued by the possibilities. One thing I’m not sure on is can you convert an Arrow Flight response to JSON easily with Java? Use case would be a service that interacts with Dremio using Arrow Flight to take advantage of the speed, but the client wants the response in JSON. I have been using this repo: https://github.com/dremio-hub/arrow-flight-client-examples and a local Dremio set up via docker.

Answer 1

Nothing is preventing you from converting Arrow data into JSON, you just need to decide how you will represent each of the format's type elements (primitives, binary, fixed/variable lists, structs, unions & null) in JSON. Most of them will have an obvious JSON counterpart, however binary will need to be encoded, e.g. as Base64. Once you have defined your mapping you need to implement it by using the Arrow API to extract the data and convert it.

However, converting the data to JSON does negate one of the primary benefits of using Arrow - bypassing serialisation/deserialisation costs. I.e. the goal should generally be to use Arrow end-to-end rather than converting it to another format like JSON.