1

All I have is raw protobuf binary data, I don't have access to .proto file, and I need to convert it to json string in Java. So is there is a way to do that in java? i.e. something similar to protoc tool

Michał
  • 2,202
  • 2
  • 17
  • 33
Anthony J.
  • 375
  • 1
  • 5
  • 14
  • What have you tried so far – SamHoque Dec 11 '18 at 09:17
  • to be honest, my information about protobuf is limited. I tried several things I found on different tutorials on internet, but I couldn't find any easy way to directly convert say byte array to json. like what protoc --decode_raw is doing – Anthony J. Dec 11 '18 at 09:20
  • I don't think you can decode the protobuf without having the .proto schema. You need to know which fields it contains. – marstran Dec 11 '18 at 09:24
  • nopes, you can't. – Mangat Rai Modi Dec 11 '18 at 09:25
  • "protoc --decode_raw" is doing exactly this. I'm thinking there should be away – Anthony J. Dec 11 '18 at 09:25
  • @AnthonyJ. The protobuf itself doesn't contain any field names. You can't reconstruct those. What exactly is your expected output? – marstran Dec 11 '18 at 09:32
  • similar to protoc output: 12612 { 1: "http://www.google.com" 2: 13065730694127675 4: "http://www.google.com/favicon.ico" } – Anthony J. Dec 11 '18 at 09:35
  • if I can get output like protoc --decode_raw, it will be enough for me. I found one answer https://stackoverflow.com/questions/13937882/parsing-protocol-buffers-without-knowing-the-proto but they guy there built his own binary parser which isn't recommended in my case. – Anthony J. Dec 11 '18 at 09:37
  • @AnthonyJ. do you by any chance have the .class/.java files for your proto type? If not, then protobuf is schemaless, and you can only have a guess at the format – Michał Dec 11 '18 at 10:50
  • @Michał unfortunately I don't have .class/.java. However, like I said, I'm fine with results like what protoc generates for schemaless protobuf – Anthony J. Dec 11 '18 at 12:01
  • 1
    @AnthonyJ. In such case, take a look here: https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/CodedInputStream. This implements the majority of the logic from the Python answer you mentioned before. If that answer is correct, you will need to switch on the last 3 bits of the field tag to learn the type. However, be aware of the limitations of this approach as listed here: https://stackoverflow.com/a/7343928/2946480. After all, protobuf is a schemaless data format. – Michał Dec 11 '18 at 12:27
  • thanks @Michał, I will take a look, it seems it has what i need but requires some work. – Anthony J. Dec 11 '18 at 13:48

2 Answers2

4

This won't be possible without the .proto schema. For multiple reasons:

  • the raw binary doesn't include field names, just numbers; you can of course create JSON with integer-looking properties, but:
  • the data format is ambiguous without a schema:
    • "varint" (simple integers) can mean multiple different things in different contexts, including signed, unsigned, zig-zag signed, or Boolean
    • ditto for fixed length, which could be integers (signed or unsigned) or floating-point
    • "length prefix" could be a utf-8 string, a packed array, or a sub-message

So: there is simply no good and reliable way of understanding data without the schema, let alone choosing how to display it as JSON.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
0

.proto file is basically schema of your proto buffer.

protoc.exe tool is used for generating the files (i.e. used for creating the proto buffer by providing setter and getters methods.)

yes their is available some methods which convert the protobuffer into JSON like i am using protobuffer in c++ and it provide some methods to convert it into JSON.

same also will be available in Java as well. so don't mix protoc tool concept with this 'data format conversion'

try to use this api provided by Protobuffer offical

https://developers.google.com/protocol-buffers/docs/reference/java/

https://developers.google.com/protocol-buffers/docs/reference/java/

Hope it will help

Sunny Goel
  • 1,982
  • 2
  • 15
  • 21
  • Thanks @Sunny. but I didn't find anything in API to fulfill my needs – Anthony J. Dec 11 '18 at 12:03
  • 1
    @AnthonyJ. follow the link which i am providing here it will really helpful for you – Sunny Goel Dec 11 '18 at 12:04
  • sorry but which link? I saw you provided two links, which both are the same. and I couldn't find anything that can just convert raw binary data to json like what protoc is doing. I'm sorry if I'm missing something here – Anthony J. Dec 11 '18 at 12:17
  • 1
    yes exactly ,These two will help you :- https://developers.google.com/protocol-buffers/docs/reference/java/ https://developers.google.com/protocol-buffers/docs/reference/java/ – Sunny Goel Dec 11 '18 at 12:19
  • 2
    this are the api's which you have to include in your project and you will be able to do it in `java` – Sunny Goel Dec 11 '18 at 12:20
  • sorry but why are you posting same link twice? and which API i need to use specifically? – Anthony J. Dec 11 '18 at 13:49
  • @AnthonyJ. have you generated `.java` files using `protoc.exe` for `serialization ` and `deserialization ` of buffer. check the code generated i.e. java files , and once you have buffer then import the api i.e. `com.google.protobuf.util.JsonFormat` class which i am specifying here for conversion of raw binary data i.e. `protocoal buffer` into JSON. – Sunny Goel Dec 11 '18 at 14:12
  • I see. However, I don't have the .proto files and I know nothing about messages schema. I only have binary data. I used "protoc --decode_raw < raw_binary_proto.bin" command and I got a json as output. I want to do something similar but by java code. – Anthony J. Dec 11 '18 at 14:19
  • 2
    @AnthonyJ. yes this `Class JsonFormat` used the method `printer` it will return `JsonFormat.Printer` object and then create or build an message of type `MessageOrBuild` from your raw data i.e. proto data vand then pass that to `print` method of `JsonFormat.Printer` . try this it will help you – Sunny Goel Dec 11 '18 at 14:33