3

I need to deserialize raw binary data (BinaryFormatter), then serialize into JSON (for editing) and then serialize it back into binary again. Obviously, I lose on floats. Original float value 0xF9FF4FC1 (big endian, roughly -12.9999933) gets rounded to 0xF6FF4FC1 (-12.99999) when I serialize from original binary (correct data and intermediated data is 1:1 in memory) to JSON. I know it is not a big loss and I know floats are problematic but I want to keep the precision as close as possible due to possible incompatibility issues later.

Anyone tackled this problem before with JSON? How can I force it to write float with max precision? I've tried built in option for handling floats as either decimal or double but there is no difference in output, unfortunately, and I cant change target values because they still need to be written as floats when I do binary serialization so there will be rounding regardless during implicit conversion.

The specific type containing floats I am trying to round-trip is Vector2 from https://github.com/FNA-XNA/FNA/blob/master/src/Vector2.cs.

tl:dr have a float, want JsonNET serialize it as precise as possible into final json string.

P.S. I'e read tons of questions here and blog entries elsewhere but haven't found anyone trying to solve the same issue, most of the search hits were with float-reading issues (which I'm gonna need to solve later on too).

UPDATE: As @dbc below pointed out - Jsont.NET respects "TypeConverter" attribute thus I had to make my own converter that overrides it.

dbc
  • 104,963
  • 20
  • 228
  • 340
KreonZZ
  • 175
  • 2
  • 10
  • 2
    Swearing is **not okay** on SO. – T.J. Crowder Aug 06 '18 at 08:32
  • Possible duplicate of [Json.NET serializing float/double with minimal decimal places, i.e. no redundant ".0"?](https://stackoverflow.com/questions/21153381/json-net-serializing-float-double-with-minimal-decimal-places-i-e-no-redundant) – Pete Kirkham Aug 06 '18 at 08:37
  • The question I tagged as a duplicate is the opposite case - restricting the number of decimal places. But both are controlling the number of decimal places. Instead of ` writer.WriteRawValue(JsonConvert.ToString(value));` you'd use ` writer.WriteRawValue(((float)value).ToString('r'));` for the roundtrip format. – Pete Kirkham Aug 06 '18 at 08:39
  • @Pete Kirkham I've just read that few minutes ago and it is not relevant – KreonZZ Aug 06 '18 at 08:41
  • @dbc it needs to be written back as float into binary so there will be loss and incorrect values in the end, that's the problem – KreonZZ Aug 06 '18 at 08:55
  • @dbc It is in my post. ORIGINAL binary has float value of 0xF9FF4FC1 (-12.9999933), when I deserialize it from binary and then serialize into binary - it is 1:1. However, if I deserialzie it form original binary and serialize INTO JSON, it gets written as `-12.99999` which is upon reading that JSON and serializing into binary again will result in float being represented as `0xF6FF4FC1` (as you can see the elder byte is lower/smaller). What I WANT: write float value as it is when serializing into JSON with max precision. – KreonZZ Aug 06 '18 at 09:08
  • 1
    Can't reproduce, see https://dotnetfiddle.net/qBqBdW. I think we may need to see a [mcve] to help you. Also, what version of Json.NET are you using? – dbc Aug 06 '18 at 09:16
  • @dbc My whole example is literally `File.WriteAllText(@"filename", JsonConvert.SerializeObject(object));` there ist much to add. The object in question is a class that has a `Vector2` struct which uses `floats` for X/Y. I use latest 11.0.2 JsonNET, Targeting Framework 4.0, I've updated my post. – KreonZZ Aug 06 '18 at 09:19
  • Are you talking about [`System.Numerics.Vector2`](https://msdn.microsoft.com/en-us/library/system.numerics.vector2.aspx)? Because as I showed in https://dotnetfiddle.net/qBqBdW I cannot reproduce this with a class containing a single float property. Maybe `Vector2` has a dodgy `TypeConverter` somewhere? Again, might you please provide a [mcve] showing a class that demonstrates precision loss when round-tripped to JSON using Json.NET? – dbc Aug 06 '18 at 09:28
  • 1
    Oh darn, you are right it DOES have TypeConverter: `https://github.com/FNA-XNA/FNA/blob/c77c82837af89e28e9e71106a3637236c215ada3/src/Vector2.cs` --- is there a way to bypass it/ignore? – KreonZZ Aug 06 '18 at 09:40
  • Hmmm maybe [Newtonsoft.JSON cannot convert model with TypeConverter attribute](https://stackoverflow.com/q/31325866/3744182) is what you need then. Or if you make a custom `JsonConverter` for `Vector2` I believe it will should supersede the type converter. – dbc Aug 06 '18 at 09:56
  • Yep, this answers my question. Fee free to post as an answer and I will mark and upboat it, cheers! – KreonZZ Aug 06 '18 at 10:18

1 Answers1

3

Json.NET will serialize float values using the round-trip precision format "R" (source). However, the type you are using, https://github.com/FNA-XNA/FNA/blob/master/src/Vector2.cs, has a TypeConverter applied:

    [Serializable]
    [TypeConverter(typeof(Vector2Converter))]
    [DebuggerDisplay("{DebugDisplayString,nq}")]
    public struct Vector2 : IEquatable<Vector2>
    {
        //...

As explained in the Newtonsoft docs, such types will be serialized as a string using the converter:

Primitive Types

.Net: TypeConverter (convertible to String)
JSON: String

And, inspecting the code for https://github.com/FNA-XNA/FNA/blob/master/src/Design/Vector2Converter.cs, it appears this converter is not using round-trip precision format when converting to string, at around line 60:

    return string.Join(
        culture.TextInfo.ListSeparator,
        new string[]
        {
            vec.X.ToString(culture),
            vec.Y.ToString(culture)
        }
    );

Thus the built-in TypeConverter itself is where you are losing precision.

To avoid this problem you can either

  1. Create a custom JsonConverter for Vector2 such as the following:

    public class Vector2Converter : JsonConverter
    {
        class Vector2DTO
        {
            public float X;
            public float Y;
        }
    
        public override bool CanConvert(Type objectType)
        {
            return objectType == typeof(Vector2) || objectType == typeof(Vector2?);
        }
    
        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            // A JSON object is an unordered set of name/value pairs so the converter should handle
            // the X and Y properties in any order.
            var dto = serializer.Deserialize<Vector2DTO>(reader);
            if (dto == null)
                return null;
            return new Vector2(dto.X, dto.Y);
        }
    
        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            var vec = (Vector2)value;
            serializer.Serialize(writer, new Vector2DTO { X = vec.X, Y = vec.Y });
        }
    }
    

    If you add the converter to JsonSerializerSettings.Converters it will supersede the TypeConverter.

    Working sample fiddle here.

    The converter serializes the Vector2 as an object with X and Y properties, but you could also serialize it as an array with two values if you prefer. I would not recommend serializing as a primitive string since Vector2 does not correspond to a JSON primitive.

  2. Use a custom contract resolver that does not create a primitive contract for types with TypeConverter applied, such as the one shown in the answer to Newtonsoft.JSON cannot convert model with TypeConverter attribute.

Notes:

  • The round-trip format "R" apparently does not preserve the difference between +0.0 and -0.0, and so Json.NET does not either, see https://dotnetfiddle.net/aereJ2 or https://dotnetfiddle.net/DwoGyX for a demo. The Microsoft docs make no mention of whether the zero sign should be preserved when using this format.

    Thus, this is one case where round-tripping a float might result in binary changes.

    (Thanks to @chux for raising this issue in comments.)

  • As an aside, Json.NET also uses "R" when writing double (as shown in the source for JsonConvert.ToString(double value, ...)):

    internal static string ToString(double value, FloatFormatHandling floatFormatHandling, char quoteChar, bool nullable)
    {
        return EnsureFloatFormat(value, EnsureDecimalPlace(value, value.ToString("R", CultureInfo.InvariantCulture)), floatFormatHandling, quoteChar, nullable);
    }
    

    For double only this format is documented to possibly lose precision. From The Round-trip ("R") Format Specifier:

    In some cases, Double values formatted with the "R" standard numeric format string do not successfully round-trip if compiled using the /platform:x64 or /platform:anycpu switches and run on 64-bit systems.

    Thus round-tripping a double via Json.NET might result in small binary differences. However, this does not strictly apply to the question here, which is specifically about float.

dbc
  • 104,963
  • 20
  • 228
  • 340
  • 1
    Aside: Does format "R" preserve the difference between `+0.0` and `-0.0`? – chux - Reinstate Monica Aug 06 '18 at 13:23
  • @chux - seems like not, see https://dotnetfiddle.net/DwoGyX. The docs do not mention anything one way or the other: https://learn.microsoft.com/en-us/dotnet/standard/base-types/standard-numeric-format-strings#the-round-trip-r-format-specifier – dbc Aug 06 '18 at 18:25
  • 1
    Well perhaps format "R" is only [_mostly_](https://www.youtube.com/watch?v=xbE8E1ez97M) round-trip-ish. I alerted MS to this before with C++. – chux - Reinstate Monica Aug 06 '18 at 18:28