-2

In the depths of some software I'm working on there is a line of code...

double DataNoise = StatsStuff.MeanofSquares() - average * average;

Example numbers: 

StatsStuff.MeanofSquares() = 1.9739125181231402E-13  
average = -4.3328988592605794E-07
DataNoise = 9.6511265664977283E-15 //(State1)  
DataNoise = 9.6511265664977204E-15 //(State2)

If I relaunch the analysis from the GUI repeatedly, sooner or later the result of this calculation changes, sometimes on the first re-run of analysis, but it will usually give a few consistent results before switching to a different answer (the number of times before the switch is significantly variable). Once the software has switched to returning this second value, it never reverts to returning the first.

I'm using C# and Visual Studio, testing on a Windows 7 machine with an i5 4570 if that helps anyone.

I have seen the problem in both Debug and Release builds.

Each time the analysis is launched all the analysis objects are recreated within the analysis method, so there shouldn't be anything persisting.

I've logged the values going into the calculation and they do not change; I have also used BitConverter.GetBytes() to check the numbers are the same.

I've already seen the question below and many other articles like it online, but they all relate to the differences between two different machines.
Why does this floating point calculation give different results...

The answers in how-deterministic-is-floating-point-inaccuracy seem to suggest that I should be able to expect deterministic behaviour from a single machine and instruction set, yet I don't.

Any help explaining why this happens and/or how to ensure a consistent result would be greatly appreciated.

Some additional byte values from debugging:
Inputs:
average: 48, 51, 51, 18, 221, 19, 157, 190
MeanOfSquares: 205, 250, 200, 243, 196, 199, 75, 61

Outputs:
DataNoise (state 1): 192, 220, 244, 228, 126, 187, 5, 61
DataNoise (state 2): 187, 220, 244, 228, 126, 187, 5, 61

rbren
  • 73
  • 7
  • Also have a squiz at https://stackoverflow.com/questions/2342396/why-does-this-floating-point-calculation-give-different-results-on-different-mac , https://msdn.microsoft.com/en-us/library/c151dt3s.aspx?f=255&MSPPError=-2147217396 and https://stackoverflow.com/questions/328622/how-deterministic-is-floating-point-inaccuracy . – mjwills Feb 14 '18 at 10:33
  • Minute differences in the less-significant-bits in numbers are to be expected when working with IEEE-754 arithmetic - but results *should* be deterministic given identical inputs on the same hardware, environment, etc. Are you **absolutely certain** that `StatsStuff.MeanOfSquares()` is returning *identical* values (all 64 bits of a `System.Double` value)? – Dai Feb 14 '18 at 10:34
  • Unfortunately it's in the depths of software. So far I haven't been able to replicate the effect outside the main software. – rbren Feb 14 '18 at 10:37
  • https://msdn.microsoft.com/en-us/library/a5be4sc9%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396 will help you see the underlying bytes of a `double`. In all likelihood, two numbers that **look** equal aren't **exactly** equal (when you check the bytes). – mjwills Feb 14 '18 at 10:45
  • @Dai Yes, with BitCoverter I've checked the numerical values of the constituent bytes of both values and they are all identical in both cases. – rbren Feb 14 '18 at 10:52
  • I usually see differences between Debug and Release. I believe the Release uses the Microprocessor FPU while debug uses a software simulation. Some Microprocessors have bugs in the FPU so there are patches in that correct for the bugs. I haven't proved it but I think the patches aren't designed to work with all the different micro bugs so sometimes the patches are causing the issue. – jdweng Feb 14 '18 at 10:56
  • @jdweng Sorry, no, that’s all bollocks. The CLR will use the FPU regardless of Debug/Release configuration and there are no noteworthy bugs in Intel or AMD FPU implementations: it’s been a mature feature for over 30 years now. – Dai Feb 14 '18 at 11:11
  • That is what everybody thinks. I've seen the issue go away when going from debug to release. I've seen the msdn patches used incorrectly. I've had PC with good microprocessors and code was using a msdn patch for the FPU bug which was giving the wrong answer. – jdweng Feb 14 '18 at 11:22
  • I've tried to add an image of the byte values, but don't have the rep. for that. – rbren Feb 14 '18 at 11:43
  • I swear I've read in the past about a driver (I'm thinking a display driver) that was rather badly behaved and would change the floating point mode of the CPU. Which would be the sort of thing that could suddenly kick-in in your process and then persist. – Damien_The_Unbeliever Feb 14 '18 at 11:47
  • @Damien_The_Unbeliever This is the kind of thing I am expecting it to be, as when the software is closed and then reloaded the behaviour reverts to the first state. – rbren Feb 14 '18 at 11:52
  • The code is not formatting the result correctly. It displays too many digits, more than a variable of type *double* can store. The two extra ones are just random noise digits, albeit not random enough to implement a reliable RNG. Assuming that the calculation itself doesn't cause more significant digits to be lost. Just comparing the results of the Debug vs Release build of the program is enough to see different randomness. https://stackoverflow.com/questions/14864238/coercing-floating-point-to-be-deterministic-in-net/14865279#14865279 – Hans Passant Feb 14 '18 at 12:40
  • Is any part of the computation multithreaded? – Eric Postpischil Feb 14 '18 at 15:34
  • @HansPassant; `double` objects do not store “digits.” Each `double` that is not a NaN represents a real number exactly. There is no such thing “too many digits” for showing a value. If you want to make the point that results different when computations are performed in different ways, that is fine, but do not blame “too many digits” for the problem. Differences in computation can result in differences of any magnitude, depending on the computation, not just low digits, so it is wrong to characterize floating-point as accurate up to some number of digits or wrong after some number of digits. – Eric Postpischil Feb 14 '18 at 15:39
  • 2
    @Dai: Your claim is absolutely false. The C# language does NOT guarantee that the same computation will give the same result two times in a row even in the same process. Floating point computations are NOT deterministic in C# because *the jitter gets to decide whether to use high precision registers or not at its absolute discretion*. Therefore computations can be done in either 64 bit precision, at a minimum, or some higher amount of precision depending on the register size. This can change the results of computations. – Eric Lippert Feb 15 '18 at 14:51
  • 1
    @Dai: It is unfortunate that floating point arithmetic is not deterministic in C#, but if you want to complain about that unfortunate situation, complain to Intel. They're the ones who gave us a chip set where deterministic computations are slower and less accurate, and thereby incented language developers to enshrine non-determinism into floating point computations. – Eric Lippert Feb 15 '18 at 14:54
  • @rbren: If you are reading one of my answers in one of the questions you linked to and thinking that I meant to say that computations can be different *only* across machines or architectures, well, that was not my intention. The C# specification says that floating point computations can be done in any bit size larger than the required minimum *at any time for any reason whatsoever*, and that this can change observed results. The runtime is permitted to change its mind about register allocations *at any time*. – Eric Lippert Feb 15 '18 at 15:01
  • @Eric Lippert Thanks for that clarification, that helps me a lot. Is the behaviour I've seen where the particular thread appears to switch into a different state and then remain this way fairly typical behaviour then? Today I have tried moving the calculations out into a Task and appear to be returning consistent results. I guess this is due to a new thread (with a new bit size state) being created in every calculation loop. Though from what you state I guess shouldn't discount the possibility of the new thread starting with a different precision state each time. – rbren Feb 15 '18 at 15:37
  • @rbren Do not try to guess at implementation details that C# has *specifically* gone out of its way to tell you aren't deterministic in an attempt to rely on what you think the optimizer will do. Write your code such that it works just fine *regardless* of what optimizations take place and which of multiple possible implementations are used. If you don't, you're only setting yourself up for major problems down the road when you either guess wrong or those implementations change. – Servy Feb 15 '18 at 15:39
  • It appears this is indeed another duplicate of the floating point precision questions out there and I imagine will in time be closed. The help given in this did help me though - particularly the clarifications regarding performing computations on the same machine and the variation in bit size used for calculation. – rbren Feb 15 '18 at 16:14
  • Don't think of this as a question of statefulness; it's a question of storage. There are a limited number of high precision floating point registers available, and the jitter gets to decide which computations are done with all the operands in registers, and which computations move data from high precision register temporaries to lower precision stack temporaries. I have only the vaguest idea of how the jitter makes those decisions, and remember, it has to make those decisions *as code is running*. – Eric Lippert Feb 15 '18 at 16:39
  • @EricLippert: Is C# compiled just-in-time? Is code possibly compiled more than once during some sort of session? If so, that definitely explains the observed behavior. – Eric Postpischil Feb 16 '18 at 08:16
  • @EricPostpischil: jitted, yes. Re-jitted, typically not to my knowledge, but I am not an expert on the jitter. – Eric Lippert Feb 16 '18 at 14:34
  • According to [very old information posted by Jon Skeet](https://stackoverflow.com/a/1255832/18192), The Compact Framework may jit more than once. Given that there exist situations where jitting happens more than once, I wouldn't want my code to rely on an assumption that jitting is a one-time operation unless it was documented (and even then, I still wouldn't want to rely on it). – Brian Feb 16 '18 at 22:08

1 Answers1

2

According to the C# specification, regarding an operation with double, “the operation is performed using at least double range and precision…”

This means that StatsStuff.MeanofSquares() could be computed with extended precision, and this extended precision result could be used directly in StatsStuff.MeanofSquares() - average * average. If the computation of StatsStuff.MeanofSquares() produces results that vary slightly in extended precision, the differences may not be visible when the number is printed with only 17 digits.

A significant clue is that the two results displayed are exactly the results one could get by calculating from the input values shown using separate double multiplication and addition for the first result and either fused double multiplication and addition or long double arithmetic for the second result. This suggests that different instructions are being used to evaluate the two results. Specifically:

  • Let m be the double value that is closest to 1.9739125181231402E-13.
  • Let a be the double value that is closest to -4.3328988592605794E-07.
  • The first result you show, 9.6511265664977283E-15, equals the result of computing a*a in double, subtracting the product from m in double, and converting the result to 17 decimal digits.
  • The second result you show, 9.6511265664977204E-15, equals the result of computing m-a*a with exact mathematics and then rounding to double (as with C’s fma(a, -a, m)) and converting the result to 17 decimal digits. It also equals the result of computing a*a in long double, subtracting the product from m, rounding the result to double, and converting that to 17 decimal digits.

The only way that these different operations would be performed is by different instructions. So this suggests that double DataNoise = StatsStuff.MeanofSquares() is being compiled to different instructions at different times. One possibility is that this statement appears multiple times in the source code. Another is that the compiler inlines the function containing it, so that it is compiled differently in different contexts.

Since the question does not provide a reproducible example or any information about the context of the statement double DataNoise = StatsStuff.MeanofSquares() - average * average, no definitive answer is possible.

While fact that the numbers match a separate-versus-fused and a double-versus-long-double pattern, which strongly suggest different instructions are used for different results, the possibility remains that one instruction sequence is used to calculate StatsStuff.MeanofSquares() - average * average but that the input values to the expression vary, likely with StatsStuff.MeanofSquares() calculated with extra precision that is not visible in the limited digits you printed. If your software is multithreaded, it may partition a problem into subproblems and execute them in parallel with multiple threads. As those threads return results, it may combine the results to produce a final result for StatsStuff.MeanofSquares(). Since the threads may complete in different orders in different runs, the results may be combined in different orders. This means different data is being used in the operations, so the results may be different. (For example, in two-digit decimal, adding 21 + 4.9 + 90 yields 26 + 90 (21 + 4.9 is exactly 25.9, so rounding to two digits produces 26) and then 120, but adding 21 + 90 + 4.9 yields 110 (111 rounds to 110) and then 110 (110+4.9 rounds to 110).

Another possibility is the software has a bug that causes it to use uninitialized data, and this data affects the results.

If the problem is that the expression is being evaluated in different ways, a potential workaround might be to assign intermediate results to temporary variables:

double t0 = average*average;
double t1 = StatsStuff.MeanofSquares();
double Mean = t1 - t0;

I conjecture that such assignments will result in rounding each expression to double. I do not see an explicit statement of this in the C# specification, but it is a rule in C, and the C# compiler may do it as well. If so, this will likely considerably reduce the frequency with which different final results are observed, but it may not completely eliminate them.

(Since C# is not serving your purposes, you should let Microsoft know, and you should seek other languages and other compilers.)

Eric Postpischil
  • 195,579
  • 13
  • 168
  • 312
  • I'm writing in C# and using VisualStudio. Regarding multi-threading, even if this were the case then should I not be able to expect the calculation, double - double * double, to be done on a single thread and return consistent results, given that the inputs are the same? – rbren Feb 15 '18 at 08:27
  • @rbren: Updated. There is a pattern in the numbers you show that strongly suggests the two results you show come from different instances of instructions that evaluate the expression `StatsStuff.MeanofSquares() - average * average`. – Eric Postpischil Feb 15 '18 at 14:36
  • Thanks for the detailed answer! I'm not running this on multiple threads (though I think I should be heading in that direction), but the detail you give regarding relative levels of precision agrees with Eric's comments. – rbren Feb 15 '18 at 15:44