39

I know that floating point numbers have precision and the digits after the precision is not reliable.

But what if the equation used to calculate the number is the same? can I assume the outcome would be the same too?

for example we have two float numbers x and y. Can we assume the result x/y from machine 1 is exactly the same as the result from machine 2? I.E. == comparison would return true

Steve
  • 11,696
  • 7
  • 43
  • 81

6 Answers6

49

But what if the equation used to calculate the number is the same? can I assume the outcome would be the same too?

No, not necessarily.

In particular, in some situations the JIT is permitted to use a more accurate intermediate representation - e.g. 80 bits when your original data is 64 bits - whereas in other situations it won't. That can result in seeing different results when any of the following is true:

  • You have slightly different code, e.g. using a local variable instead of a field, which can change whether the value is stored in a register or not. (That's one relatively obvious example; there are other much more subtle ones which can affect things, such as the existence of a try block in the method...)
  • You are executing on a different processor (I used to observe differences between AMD and Intel; there can be differences between different CPUs from the same manufacturer too)
  • You are executing with different optimization levels (e.g. under a debugger or not)

From the C# 5 specification section 4.1.6:

Floating-point operations may be performed with higher precision than the result type of the operation. For example, some hardware architectures support an "extended" or "long double" floating-point type with greater range and precision than the double type, and implicitly perform all floating-point operations using this higher precision type. Only at excessive cost in performance can such hardware architectures be made to perform floating-point operations with less precision, and rather than require an implementation to forfeit both performance and precision, C# allows a higher precision type to be used for all floating-point operations. Other than delivering more precise results, this rarely has any measurable effects. However, in expressions of the form x * y / z, where the multiplication produces a result that is outside the double range, but the subsequent division brings the temporary result back into the double range, the fact that the expression is evaluated in a higher range format may cause a finite result to be produced instead of an infinity.

Jon Skeet
  • 1,421,763
  • 867
  • 9,128
  • 9,194
  • 3
    I will add that per experience that the difference in processor is not only manufacturer related. An intel core 2 compute different values than I5/i7 and xeon E5 v3 and E7 v3 are different too. – Franck Jan 22 '18 at 15:37
  • @Franck: Thanks, added that. – Jon Skeet Jan 22 '18 at 16:02
  • 4
    "No, not neccessarialy" You're being too kind. I would have opened with "HECK NO!" =) – Cort Ammon Jan 22 '18 at 18:45
  • I guess the jit produces different machine code to take best advantage of different cpus, or do you mean the cpu itself causes the differences? – Deduplicator Jan 22 '18 at 23:50
  • 1
    @Deduplicator: Well the JIT decides whether to use 80-bit or 64-bit operations. I believe that the same CPU floating-point instruction with the same inputs should always give the same result, but I'm not an expert there. – Jon Skeet Jan 23 '18 at 07:19
21

Jon's answer is of course correct. None of the answers however have said how you can ensure that floating point arithmetic is done in the amount of precision guaranteed by the specification and not more.

C# automatically truncates any float back to its canonical 32 or 64 bit representation under the following circumstances:

  • You put in a redundant explicit cast: x + y might have x and y as higher-precision numbers that are then added. But (double)((double)x+(double)y) ensures that everything is truncated to 64 bit precision before and after the math happens.
  • Any store to an instance field of a class, static field, array element, or dereferenced pointer always truncates. (Stores to locals, parameters and temporaries are not guaranteed to truncate; they can be enregistered. Fields of a struct might be on the short-term pool which can also be enregistered.)

These guarantees are not made by the language specification, but implementations should respect these rules. The Microsoft implementations of C# and the CLR do.

It is a pain to write the code to ensure that floating point arithmetic is predictable in C# but it can be done. Note that doing so will likely slow down your arithmetic.

Complaints about this awful situation should be addressed to Intel, not Microsoft; they're the ones who designed chips that make doing predictable arithmetic slower.

Also, note that this is a frequently asked question. You might consider closing this as a duplicate of:

Why differs floating-point precision in C# when separated by parantheses and when separated by statements?

Why does this floating-point calculation give different results on different machines?

Casting a result to float in method returning float changes result

(.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?

Coercing floating-point to be deterministic in .NET?

C# XNA Visual Studio: Difference between "release" and "debug" modes?

C# - Inconsistent math operation result on 32-bit and 64-bit

Rounding Error in C#: Different results on different PCs

Strange compiler behavior with float literals vs float variables

Eric Lippert
  • 647,829
  • 179
  • 1,238
  • 2,067
  • Interestingly, the ECMA standard (at least the upcoming C# 5 one) does mention casting for this, although it's non-normative. At the end of an example in section 9.3.7 it states: "To force a value of a floating-point type to the exact precision of its type, an explicit cast can be used." – Jon Skeet Jan 22 '18 at 20:07
  • So I should uninstall Resharper which thinks those casts are redundant. – Thomas Weller Jan 22 '18 at 20:32
  • 2
    @ThomasWeller: I've never used R# myself, but from what I hear it has a pretty high false positive rate in a lot of its static analysis. Maybe mention it to them and see if they can fix the warning. – Eric Lippert Jan 22 '18 at 20:34
  • Hmm, so implementations should respect rules which are not in the specification? – Paŭlo Ebermann Jan 22 '18 at 21:14
  • 1
    @PaŭloEbermann: Yes, they *should*. An implementation which does not respect these rules is *morally wrong*, and should be sent to its room until it learns to play nicely with others. – Eric Lippert Jan 22 '18 at 21:16
  • Hmm, if I would be making an implementation (I am not), and not had be randomly stumbling onto your post, how would I know about those rules? – Paŭlo Ebermann Jan 22 '18 at 21:20
  • @PaŭloEbermann: Well, if you are making an implementation of the CLI, and then building C# on top of it then you get the "writes to heap locations truncate" behaviour because that's a rule of the CLI spec, not a rule of the C# spec. As Jon noted, newer versions of the C# spec note that a cast should truncate. If you're building an implementation of C# on top of a non-CLI platform then my guess is that there will be incompatibility problems bigger than this one to deal with. – Eric Lippert Jan 22 '18 at 21:28
  • 1
    @PaŭloEbermann for example, the C# specification implies that casting an `int[]` to `object` and then casting the `object` to `uint[]` should fail, but the CLR allows it because the CLR is a bit wacky when it comes to array covariance, and so C# implementations allow it as well even though that rule is found nowhere in the spec. – Eric Lippert Jan 22 '18 at 21:29
  • @PaŭloEbermann: Or, the C# spec says nothing about what to do in cases like you have a generic interface where two methods unify to the same signature under some generic construction, and some generic class that implements the interface has two methods that both match those signatures under construction, and which method is assigned to which interface slot? The CLR has implementation-defined behaviour in that case; a new implementation of C# might or might not match that behaviour. – Eric Lippert Jan 22 '18 at 21:32
  • 2
    @PaŭloEbermann: I could go on. There are a great many places where the C# language is slightly under-specified, and also a great many places where implementations violate the specification in subtle ways to avoid backwards-compat breaking changes. See the Roslyn source code, anywhere that a comment says DELIBERATE SPEC VIOLATION. I left extensive notes behind on many of these scenarios. – Eric Lippert Jan 22 '18 at 21:33
  • So, either the implementations should break back-compat, or the spec needs fixing. Let's hope it doesn't make the spec too ugly. – Deduplicator Jan 22 '18 at 23:57
  • 2
    @Deduplicator: Welcome to every day of my life of those seven years on the C# team. We struggled constantly to find the best possible balance for developers that maintained backwards compat, spec accuracy, spec elegance, implementation performance, and a dozen other factors. *There are no easy choices when implementing a language that millions of people depend on.* – Eric Lippert Jan 23 '18 at 00:00
  • 1
    @Deduplicator: An oldie but a goodie: https://blogs.msdn.microsoft.com/ericlippert/2006/03/29/the-root-of-all-evil-part-two/ -- if the optimization pass runs slightly too soon then some zeros are assignable to enum and some are not. *What's the right thing to do?* We agonized about that one for ages, *and I implemented it wrong* when we finally decided what to do. – Eric Lippert Jan 23 '18 at 00:02
9

No, it does not. The outcome of the calculation can differ per CPU, because the implementation of floating point arithmetic can differ per CPU manufacturer or CPU design. I even remember a bug in the floating point arithmetic in some Intel processors, which screwed up our calculations.

And then there is difference in how the code is evaluated bu the JIT compiler.

Patrick Hofman
  • 153,850
  • 22
  • 249
  • 325
8

Frankly I wouldn't expect two places in the same codebase to return the same thing for x/y for the same x and y - in the same process on the same machine; it can depend on how exactly x and y get optimized by the compiler / JIT - if they are enregistered differently, they can have different intermediate precisions. A lot of operations are done using more bits than you expect (register size); and exactly when this gets forced down to 64 bits can affect the outcome. The choice of that "exactly when" can depends on what else is happening in the surrounding code.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
4

In theory, on an IEEE 754 conforming system, the same operation with the same input values is supposed to produce the same result.

As Wikipedia summarizes it:

The IEEE 754-1985 allowed many variations in implementations (such as the encoding of some values and the detection of certain exceptions). IEEE 754-2008 has strengthened up many of these, but a few variations still remain (especially for binary formats). The reproducibility clause recommends that language standards should provide a means to write reproducible programs (i.e., programs that will produce the same result in all implementations of a language), and describes what needs to be done to achieve reproducible results.

As usual, however, theory is different from practice. Most programming languages in common use, C# included, do not strictly conform to IEEE 754, and do not necessarily provide a means to write a reproducible program.

Additionally, modern CPU/FPUs make it somewhat awkward to ensure strict IEEE 754 compliance. By default they will operate with "extended precision", storing values with more bits than a double internally. If you want strict semantics you need to pull values out of the FPU into a CPU register, check for and handle various FPU exceptions, and then push values back in -- between each FPU operation. Due to this awkwardness, strict conformance has a performance penalty, even at the hardware level. The C# standard chose a more "sloppy" requirement to avoid imposing a performance penalty on the more common case where small variations are not a problem.

None of this is often an issue in practice since most programmers have internalized the (incorrect, or at least misleading) idea that floating-point math is inexact. Additionally, the errors we're talking about here are all extremely small, enough so that they are dwarfed by the much more common loss of precision caused by converting from decimal.

Daniel Pryden
  • 59,486
  • 16
  • 97
  • 135
  • Depending on what you are doing, converting from decimal may easily be the *least* of your worries for loss of precision. My experience was that subtracting two large but similar numbers was much more of a problem. – Martin Bonner supports Monica Jan 22 '18 at 20:52
  • @MartinBonner: Sure, there are lots of ways to lose precision. But I don't think it's controversial to say that converting from decimal is probably the *most common* way precision loss takes place -- wouldn't you agree? In any case, the point I was trying to make is that the error you'll get due to extended-precision calculation and other deviations from the spec on a modern machine will be quite small relative to other sources of error, such as converting from decimal, or subtracting numbers with a large number of common bits, etc. – Daniel Pryden Jan 22 '18 at 21:38
  • @MartinBonner: The error in subtracting two large but similar numbers is exactly zero. Per Sterbenz’s Lemma, if two numbers are each at least half the other, their difference (in IEEE 754 and other floating-point with subnormals) is exactly representable. When people caution about subtracting big numbers, they are usually concerned that the error **already present in the operands** is large **relative to the result**. But this is merely because the result of subtracting similar numbers is small. The subtraction itself introduces no error. – Eric Postpischil Jan 22 '18 at 22:01
2

In addition to the other answers, it's possible x/y != x/y even on the same machine.

Floating-point computations in x86 are done using 80-bit registers, but truncated to 64-bits when stored. So it's possible for the first division to be calculated, truncated, then loaded back into memory and compared against the non-truncated division.

See here for more information (it's a C++ link, but the reasoning is the same)

BlueRaja - Danny Pflughoeft
  • 84,206
  • 33
  • 197
  • 283