14

I have a client-server application where the server transmits serialized objects in protobuf format to a client, and I would like to retire a required field. Unfortunately I can't change both client and server at the same time to use the new .proto definition.

If I change a required field to be optional, but only for code that serializes messages (i.e. deserializing code has not been rebuilt and still thinks it's a required field), can I continue to publish messages that can be deserialized as long as I populate a value for the now-optional field?

(It appears to be fine to do so, at least for a few trivial cases I experimented with (only using Java), but I'm interested if it's a generally sensible approach, and whether there are any edge cases etc I should worry about).

Motivation: My goal is to retire a required field in a client-server application where the server publishes messages that are deserialized by the client. My intended approach is:

  1. Change required field to optional on the trunk.
  2. If it's necessary to deploy new server code (for unrelated features/fixes), ensure that the optional field continues to be populated in the message.
  3. Gradually deploy updated code for all clients (this will take time as it requires involvement of other teams with their own release schedules)
  4. Confirm that all clients have been updated.
  5. Begin to retire (i.e. not populate) the optional field.
bacar
  • 9,761
  • 11
  • 55
  • 75
  • N.B. I know the protobuf docs say [**required is forever**](https://developers.google.com/protocol-buffers/docs/proto), but to me it implies that this is a simplification, since it's qualified with "it will be problematic to change the field to an optional field" - problematic, not impossible...! – bacar Dec 03 '12 at 00:31
  • This is the one reason why I almost always make every field optional, just in case we change the layout in the future... I've been bitten by this feature way too many times. – g19fanatic Dec 06 '12 at 18:15
  • Related https://stackoverflow.com/q/31801257/834521 – TooTone Aug 14 '18 at 10:23

2 Answers2

22

According to the encoding format documentation, whether a field is required or not is not encoded in serialized byte stream itself. That is, optional or required makes no difference in the encoded serialized message.

I've confirmed this in practice, using the Java generated code, by writing serialized messages to disk and comparing the output - using a message containing all of the supported primitive types as well as fields representing other types.

bacar
  • 9,761
  • 11
  • 55
  • 75
3

As long as the field is set, using the parseFrom(byte[]) method to deserialize will still work, because the byte[] will be the same.

However, one would wonder why you would change the field from required to optional until you are ready to allow it to be optional? Basically you are just making it "optional" in the .proto file, but you are enforcing that it is required by always populating it. Just a thought.

CaTalyst.X
  • 1,645
  • 13
  • 16
  • In order to cater for the clients that don't yet know that the field is optional. If I were to make it optional and stop populating the field at the same time (i.e. before all deserializing clients were aware that it could be optional), then deserializing clients will break - that is no good! (I could try to update all clients first, but that would require me to maintain two copies of the .proto - one with `optional` for clients, and one with `required` for the server while I wait for all clients to be upgraded. – bacar Dec 03 '12 at 00:48
  • I suppose if you are sending different protobuf messages to different clients, this would make sense. However, if the server publishes to say, a Topic, and the clients all connect to that topic, then my question still stands. If any one client hasn't been updated to handle the optional field, then it doesnt make any sense to declare it optional in the protobuf. Updating the clients to handle the case where it is optional one at a time, and then declaring it optional in the .proto would make more sense. – CaTalyst.X Dec 03 '12 at 02:33
  • No, I'm not sending different messages to different clients - in fact if I could tailor messages to the version of the client, this question would be redundant. How would it even be possible to "update the clients to handle the case where it is optional" *without declaring it optional in the .proto* that's used by the client? If you don't change the client's .proto, parseFrom would throw if the field is missing. – bacar Dec 03 '12 at 08:15
  • Perhaps your question meant 'why would you change the field from required to optional _and then immediately deploy this pointless change to the server_'. I've updated my original question to clarify that I don't intend to do that. If this addresses your concerns, we should delete our comments :-) – bacar Dec 04 '12 at 17:20