3

Does Microsoft Bond have some best practices on how schemas evolve over time? I want to make certain we follow best practices such that we have 2 way compatibility (i.e. allowing our Bond types to evolve older versions to the current version, as well as backward compatibility allowing conversion from a newer version back to an older version). I don't see this addressed specifically in the documentation (e.g. https://microsoft.github.io/bond/manual/bond_cs.html nor https://microsoft.github.io/bond/manual/compiler.html#idl-syntax ), though other serialization frameworks such as Avro have this explicitly detailed in their documentation.

For what it's worth, we are writing in .NET (C#/F#) and intend to use the CompactBinaryWriter and CompactBinaryReader formats initially.

For example, I imagine some guidance along these lines:

  1. names of fields can change over time since the field ordinals are used for field resolution, not the names directly (except in SimpleJSON). I believe this true, is it?
  2. adding a new "required" field requires you give the field a default value
  3. removing a field in a newer version is okay, provided older versions had a default value assigned
  4. what about changing the type of a field? For example, can a field change from an string field in version 1 to a int64 in version 2? Can it change from a string to a custom union (custom type with optional fields)?
  5. any other recommendations?

Thank you!

would also be interested if there is any active forum community for this type of Microsoft Bond question, i wasn't able to find one...

user1521026
  • 67
  • 7

2 Answers2

3

I am not aware of any explicit schema evolution guidelines either, that's certainly a gap in the Bond documentation. From my own work with Bond, I can answer some of your questions:

  1. Yes, names of fields can change, all that matters for the CompactBinary format is the field ordinal.
  2. The DefaultAttribute is only valid on interface members. You can add required fields without further annotations, but upon deserialization of old records (that do not contain that required field), I expect a runtime error.
  3. You can remove fields. Default values are determined by instantiating the containing object via its default constructors, and reading out the respective field. However, if you remove a required field, the old reader would fail to deserialize it (the new reader would not have included it at all).
  4. That's a big no no. If you need to change the type of a field, leave the old field in place, (give it a suffix like _Obsolete if desired) and introduce a new field of the "correct" type.

For what it's worth: You write that you are using F#, some F# specific changes are in the making, see here. This will support records, unions, and core F# data types.

Update: There is now a section on schema evolution in the Bond documentation.

Anton Schwaighofer
  • 3,119
  • 11
  • 24
  • 1
    Thank you for your comments, Anton ! Interesting to hear about the F# development: as far as you know, is there an F# code generator in the works? We are currently deserializing into the C# auto-generated objects and managing the conversion to F# in our code (i.e. bond binary->C#->F#). Curious whether it's possible today or possibly forthcoming to allow us to remove that intermediary mapping and go directly from binary to F#? – user1521026 Jan 26 '17 at 13:37
  • Code generation is not yet planned, but will have a look. For the sake of completeness, can you file a feature request on the bond repository. – Anton Schwaighofer Jan 26 '17 at 13:40
  • Current work is about allowing annotations in F# code such that you can get (de-serializers) for free. You could use it to read bond payload that some other source generates, but you'd have to keep your F# code in sync with your source bond schemata. – Anton Schwaighofer Jan 26 '17 at 13:42
  • Some correction. Can you edit these in? 2: _de_serialization of old records would fail if a new required field were added. 3: if a required field is removed, the old reader will fail to deserialize it, as the new writer would not have included it at all. It will not be given its default value. – chwarr Jan 31 '17 at 07:01
  • @chwarr: thanks, I fixed #2. I think #3 is complicated enough to warrant its own answer, good to have your input. – Anton Schwaighofer Jan 31 '17 at 09:07
  • @chwarr: For #3, you stated "if a required field is removed, the old reader will fail to deserialize it, as the new writer would not have included it at all. It will not be given its default value.".... If the required field on the old reader had a default value, why wouldn't the old reader say: "i don't see a value on the new writer, so i'll just assign the default value?" Is this different between complex types and basic types? – user1521026 Feb 01 '17 at 14:55
  • @user1521026, the semantics you describe are what happen for `optional` fields. [`required` fields work](https://microsoft.github.io/bond/manual/bond_cpp.html#required-fields) as I described so that you can chose to make a deserialization fail if its missing a field. (If unspecified, a field is `optional`.) – chwarr Feb 03 '17 at 03:47
  • The Bond documentation now has a section about [schema evolution](https://microsoft.github.io/bond/manual/bond_cs.html#schema-evolution). Can you update the answer with a link? – chwarr Feb 13 '17 at 21:42
-1

The Bond schema evolution rules and best practices can now be found here: https://microsoft.github.io/bond/manual/bond_cpp.html#schema-evolution https://microsoft.github.io/bond/manual/bond_cs.html#schema-evolution

Chad Walters
  • 26
  • As a follow-up, are there any recommendations around renaming a custom struct? I don't see that addressed in the docs regarding schema evolution. I created a question on the Microsoft Bond Repo here: https://github.com/Microsoft/bond/issues/595 – user1521026 Sep 06 '17 at 18:59