C# Float formatting quirks

Question

Attention: This is not a question about lack of precision in floats when doing mathematical operations or comparisons. It is about formatting issues, though the confusion caused may leak in the description.

Background: A log entry of our application essentially said that 19 was greater than 19.

Reproduction: The mathematical steps leading to the issue could be found out, and the problem can be reproduced in a simple test:

[TestMethod]
public void WtfTest()
{
    float bottom = -51;
    float top = 19;
    float rowHeight = (float)((top - bottom) / 3.0);
    float line1 = bottom + rowHeight;
    float line2 = line1 + rowHeight;
    float line3 = line2 + rowHeight;
    Assert.IsFalse(line3 > top, $"WTF: line3={line3:n50} is greater than top={top:n50}.");
}

This test fails because line3 is actually greater than top, as can be checked e.g. with BitConverter.GetBytes. That's NOT the issue here. The real issue is the failure message which perfectly reproduces the original problem:

Assert.IsFalse failed. WTF: line3=19,00000000000000000000000000000000000000000000000000 is greater than top=19,00000000000000000000000000000000000000000000000000.

You see: the two numbers shown are perfectly equal. Though they are actually different.

Additional findings: Calculating the difference and formatting it yields 0,00000190734900000000000000000000000000000000000000 which seems correct.

Debugging the test and looking at the values in VisualStudio's "Locals" window shows 19.0000019 for line3.

Environment: .Net Framework 4.8, platform target x64, on Windows 10

Final question: How can I get the number formatting right?

Note that all Net.Core 3,5,6,7 versions seem to be formatting correctly (both n50 and f50), but 4.6 (I think) produces 19 with zeros. Probably some sort of bug. Make sure to specify exact version you are running that code against. — Alexei Levenkov, Nov 04 '22 at 08:24
@Bernhard Hiller: Also note `(float)((top - bottom) / 3.0)` divides a `float` by a `double`, forming a _rounded_ `double` quotient and then the cast to a `float`, forming a _rounded_ `float`. Avoid rounding twice: `(top - bottom) / 3.0f` (note `f`) to get the best result. [Sometimes](https://stackoverflow.com/q/66631288/2410359) it makes a difference. — chux - Reinstate Monica, Nov 04 '22 at 12:46

shingo · Accepted Answer · 2022-11-04T08:39:58.283

According to the documentation, the recommended format for float is "G9" and for double is "G17".

When used with a Double value, the "G17" format specifier ensures that the original Double value successfully round-trips. This is because Double is an IEEE 754-2008-compliant double-precision (binary64) floating-point number that gives up to 17 significant digits of precision.

When used with a Single value, the "G9" format specifier ensures that the original Single value successfully round-trips. This is because Single is an IEEE 754-2008-compliant single-precision (binary32) floating-point number that gives up to nine significant digits of precision.

I've tested the following code on .net framework and .net core

float bottom = -51;
float top = 19;
float rowHeight = (float)((top - bottom) / 3.0);
float line1 = bottom + rowHeight;
float line2 = line1 + rowHeight;
float line3 = line2 + rowHeight;
Console.WriteLine(line3.ToString("G9"));
Console.WriteLine(line3.ToString("N50"));

Here is the result

//.net framework
19.0000019
19.00000000000000000000000000000000000000000000000000

//.net core
19.0000019
19.00000190734863281250000000000000000000000000000000

C# Float formatting quirks

1 Answers1