Java: JSON -> Protobuf & back conversion

Question

I have an existing system, which is using protobuf-based communication protocol between GUI and server. Now I would like to add some persistence, but at the moment protobuf messages are straight converted to a third-party custom objects.

Is there a way to convert proto messages to json, which could be then persisted to database.

N.B.: I don't much like an idea of writing binary protobuf to database, because it can one day become not backward-compatible with newer versions and break the system that way.

Answer 1

As mentioned in an answer to a similar question, since v3.1.0 this is a supported feature of ProtocolBuffers. For Java, include the extension module com.google.protobuf:protobuf-java-util and use JsonFormat like so:

JsonFormat.parser().ignoringUnknownFields().merge(json, yourObjectBuilder);
YourObject value = yourObjectBuilder.build();

Answer 2

We are currently using protobuf-java-format to convert our Protobuf messages (anything subclass of Message) into a JSON format to send over our web API.

Simply do:

  JsonFormat.printToString(protoMessage)

Answer 3

I don't much like an idea of writing binary protobuf to database, because it can one day become not backward-compatible with newer versions and break the system that way.

Converting protobuf to JSON for storage and then back to protobuf on load is much more likely to create compatibility problems, because:

• If the process which performs the conversion is not built with the latest version of the protobuf schema, then converting will silently drop any fields that the process doesn't know about. This is true both of the storing and loading ends.
• Even with the most recent schema available, JSON <-> Protobuf conversion may be lossy in the presence of imprecise floating-point values and similar corner cases.
• Protobufs actually have (slightly) stronger backwards-compatibility guarantees than JSON. Like with JSON, if you add a new field, old clients will ignore it. Unlike with JSON, Protobufs allow declaring a default value, which can make it somewhat easier for new clients to deal with old data that is otherwise missing the field. This is only a slight advantage, but otherwise Protobuf and JSON have equivalent backwards-compatibility properties, therefore you are not gaining any backwards-compatibility advantages from storing in JSON.

With all that said, there are many libraries out there for converting protobufs to JSON, usually built on the Protobuf reflection interface (not to be confused with the Java reflection interface; Protobuf reflection is offered by the com.google.protobuf.Message interface).

Answer 4

Generics Solution

Here's a generic version of Json converter

package com.github.platform.util;

import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import com.google.protobuf.AbstractMessage.Builder;
import com.google.protobuf.Message;
import com.google.protobuf.MessageOrBuilder;
import com.google.protobuf.util.JsonFormat;

/**
 * Generic ProtoJsonUtil to be used to serialize and deserialize Proto to json
 * 
 * @author Marcello.deeSales@gmail.com
 *
 */
public final class ProtoJsonUtil {

  /**
   * Makes a Json from a given message or builder
   * 
   * @param messageOrBuilder is the instance
   * @return The string representation
   * @throws IOException if any error occurs
   */
  public static String toJson(MessageOrBuilder messageOrBuilder) throws IOException {
    return JsonFormat.printer().print(messageOrBuilder);
  }

  /**
   * Makes a new instance of message based on the json and the class
   * @param <T> is the class type
   * @param json is the json instance
   * @param clazz is the class instance
   * @return An instance of T based on the json values
   * @throws IOException if any error occurs
   */
  @SuppressWarnings({"unchecked", "rawtypes"})
  public static <T extends Message> T fromJson(String json, Class<T> clazz) throws IOException {
    // https://stackoverflow.com/questions/27642021/calling-parsefrom-method-for-generic-protobuffer-class-in-java/33701202#33701202
    Builder builder = null;
    try {
      // Since we are dealing with a Message type, we can call newBuilder()
      builder = (Builder) clazz.getMethod("newBuilder").invoke(null);

    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException
        | NoSuchMethodException | SecurityException e) {
      return null;
    }

    // The instance is placed into the builder values
    JsonFormat.parser().ignoringUnknownFields().merge(json, builder);

    // the instance will be from the build
    return (T) builder.build();
  }
}

Using it is as simple as follows:

Message instance

GetAllGreetings.Builder allGreetingsBuilder = GetAllGreetings.newBuilder();

allGreetingsBuilder.addGreeting(makeNewGreeting("Marcello", "Hi %s, how are you", Language.EN))
        .addGreeting(makeNewGreeting("John", "Today is hot, %s, get some ice", Language.ES))
        .addGreeting(makeNewGreeting("Mary", "%s, summer is here! Let's go surfing!", Language.PT));

GetAllGreetings allGreetings = allGreetingsBuilder.build();

To Json Generic

String json = ProtoJsonUtil.toJson(allGreetingsLoaded);
log.info("Json format: " + json);

From Json Generic

GetAllGreetings parsed = ProtoJsonUtil.fromJson(json, GetAllGreetings.class);
log.info("The Proto deserialized from Json " + parsed);

Answer 5

Adding to Ophirs answer, JsonFormat is available even before protobuf 3.0. However, the way to do it differs a little bit.

In Protobuf 3.0+, the JsonFormat class is a singleton and therefore do something like the below

String jsonString = "";
JsonFormat.parser().ignoringUnknownFields().merge(jsonString,yourObjectBuilder);

In Protobuf 2.5+, the below should work

String jsonString = "";
JsonFormat jsonFormat = new JsonFormat();
jsonString = jsonFormat.printToString(yourProtobufMessage);

Here is a link to a tutorial I wrote that uses the JsonFormat class in a TypeAdapter that can be registered to a GsonBuilder object. You can then use Gson's toJson and fromJson methods to convert the proto data to Java and back.

Replying to jean . If we have the protobuf data in a file and want to parse it into a protobuf message object, use the merge method TextFormat class. See the below snippet:

// Let your proto text data be in a file MessageDataAsProto.prototxt
// Read it into string  
String protoDataAsString = FileUtils.readFileToString(new File("MessageDataAsProto.prototxt"));

// Create an object of the message builder
MyMessage.Builder myMsgBuilder = MyMessage.newBuilder();

// Use text format to parse the data into the message builder
TextFormat.merge(protoDataAsString, ExtensionRegistry.getEmptyRegistry(), myMsgBuilder);

// Build the message and return
return myMsgBuilder.build();

Answer 6

Try JsonFormat.printer().print(MessageOrBuilder), it looks good for proto3. Yet, it is unclear how to convert the actual protobuf message (which is provided as the java package of my choice defined in the .proto file) to a com.google.protbuf.Message object.

Answer 7

Here is my utility class, you may use:

package <removed>;
import com.google.protobuf.Message;
import com.google.protobuf.MessageOrBuilder;
import com.google.protobuf.util.JsonFormat;
/**
 * Author @espresso stackoverflow.
 * Sample use:
 *      Model.Person reqProto = ProtoUtil.toProto(reqJson, Model.Person.getDefaultInstance());
        Model.Person resProto = personSvc.update(reqProto); // service layer call
        final String resJson = ProtoUtil.toJson(resProto);
 **/
public class ProtoUtil {
    public static <T extends Message> String toJson(T obj){
        try{
            return JsonFormat.printer().print(obj);
        }catch(Exception e){
            throw new RuntimeException("Error converting Proto to json", e);
        }
    }
   public static <T extends MessageOrBuilder> T toProto(String protoJsonStr, T message){
        try{
            Message.Builder builder = message.getDefaultInstanceForType().toBuilder();
            JsonFormat.parser().ignoringUnknownFields().merge(protoJsonStr,builder);
            T out = (T) builder.build();
            return out;
        }catch(Exception e){
            throw new RuntimeException(("Error converting Json to proto", e);
        }
    }
}

Answer 8

For protobuf 2.5, use the dependency:

"com.googlecode.protobuf-java-format" % "protobuf-java-format" % "1.2"

Then use the code:

com.googlecode.protobuf.format.JsonFormat.merge(json, builder)
com.googlecode.protobuf.format.JsonFormat.printToString(proto)

Answer 9

Well, there is no shortcut to do it as per my findings, but somehow you
an achieve it in few simple steps

First you have to declare a bean of type 'ProtobufJsonFormatHttpMessageConverter'

@Bean  
@Primary  
public ProtobufJsonFormatHttpMessageConverter protobufHttpMessageConverter() {  
  return new ProtobufJsonFormatHttpMessageConverter(JsonFormat.parser(), JsonFormat.printer());  
}  

Then you can just write an Utility class like ResponseBuilder, because it can parse the request by default but without these changes it can not produce Json response. and then you can write few methods to convert the response types to its related object type.

public static <T> T build(Message message, Class<T> type) {
  Printer printer = JsonFormat.printer();
  Gson gson = new Gson();
  try {
    return gson.fromJson(printer.print(message), type);
  } catch (JsonSyntaxException | InvalidProtocolBufferException e) {
    throw new ApiException(HttpStatus.INTERNAL_SERVER_ERROR, "Response   conversion Error", e);
  }
}

Then you can call this method from your controller class as last line like -

return ResponseBuilder.build(<returned_service_object>, <Type>);

Hope this will help you to implement protobuf in json format.

Answer 10

Latest answer with google protobuf 3.7.0 :
Maven changes -- Add this to your pom.xml :

<dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>3.7.0</version>
</dependency>
<dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java-util</artifactId>
    <version>3.7.0</version>
</dependency>
</dependencies>
<build>
    <extensions>
        <extension>
            <groupId>kr.motd.maven</groupId>
            <artifactId>os-maven-plugin</artifactId>
            <version>1.6.0</version>
        </extension>
    </extensions>
    <plugins>
    <plugin>
        <groupId>org.xolstice.maven.plugins</groupId>
        <artifactId>protobuf-maven-plugin</artifactId>
        <version>0.6.1</version>
        <extensions>true</extensions>
        <executions>
            <execution>
                <goals>
                    <goal>compile</goal>
                    <goal>test-compile</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <additionalProtoPathElements>
                <additionalProtoPathElement>${project.basedir}/src/main/resources</additionalProtoPathElement>
            </additionalProtoPathElements>
            <protocArtifact>com.google.protobuf:protoc:3.7.0:exe:${os.detected.classifier}</protocArtifact>
        </configuration>
    </plugin>

This is the java class :

public class ProtobufTrial {
  public static void main(String[] args) throws Exception {
    String jsonString = "";
    MyProto.MyEventMsg.Builder builder = MyProto.MyEventMsg.newBuilder();
    JsonFormat.parser().ignoringUnknownFields().merge(jsonString, builder);
    MyProto.MyEventMsg value = builder.build();

    // This is for printing the proto in string format
    System.out.println(JsonFormat.printer().print(value));
  }
}

Answer 11

I wrote a simple library (here) to convert protoc generated Java objects to any Jackson supported dataformat (like JSON, YAML or anything... you name it). This allowed me to use all the Jackson supported serialization features on the converted output. It is based on Google's protobuf-java-util library.

Add the dependency and use the library as follows:

MyMessage myMessage = MyMessage.newBuilder().build();
JsonMapper jsonMapper = JsonMapper.builder()
    .addModule(JavaProtoModule.builder().build()).build();
String serialized = jsonMapper.writeValueAsString(myMessage);