305

It is well-known that NaNs propagate in arithmetic, but I couldn't find any demonstrations, so I wrote a small test:

#include <limits>
#include <cstdio>

int main(int argc, char* argv[]) {
    float qNaN = std::numeric_limits<float>::quiet_NaN();

    float neg = -qNaN;

    float sub1 = 6.0f - qNaN;
    float sub2 = qNaN - 6.0f;
    float sub3 = qNaN - qNaN;

    float add1 = 6.0f + qNaN;
    float add2 = qNaN + qNaN;

    float div1 = 6.0f / qNaN;
    float div2 = qNaN / 6.0f;
    float div3 = qNaN / qNaN;

    float mul1 = 6.0f * qNaN;
    float mul2 = qNaN * qNaN;

    printf(
        "neg: %f\nsub: %f %f %f\nadd: %f %f\ndiv: %f %f %f\nmul: %f %f\n",
        neg, sub1,sub2,sub3, add1,add2, div1,div2,div3, mul1,mul2
    );

    return 0;
}

The example (running live here) produces basically what I would expect (the negative is a little weird, but it kind of makes sense):

neg: -nan
sub: nan nan nan
add: nan nan
div: nan nan nan
mul: nan nan

MSVC 2015 produces something similar. However, Intel C++ 15 produces:

neg: -nan(ind)
sub: nan nan 0.000000
add: nan nan
div: nan nan nan
mul: nan nan

Specifically, qNaN - qNaN == 0.0.

This... can't be right, right? What do the relevant standards (ISO C, ISO C++, IEEE 754) say about this, and why is there a difference in behavior between the compilers?

Michael Hoffman
  • 32,526
  • 7
  • 64
  • 86
geometrian
  • 14,775
  • 10
  • 56
  • 132
  • 18
    Javascript and Python(numpy) do not have this behavior. `Nan-NaN` is `NaN`. Perl and Scala also behave similarly. – Paul Aug 25 '15 at 05:18
  • 36
    Maybe you enabled unsafe math optimizations (the equivalent of `-ffast-math` on gcc)? – Matteo Italia Aug 25 '15 at 05:21
  • 2
    @n.m. I didn't say the C/C++ standards. I said "the standard", meaning IEEE 754 and associated implementations within C and C++. I acknowledge that was unclear, however. – geometrian Aug 25 '15 at 05:23
  • 5
    @n.m.: Not true. Annex F, which is optional but normative when supported, and necessary to have floating point behavior specified *at all*, essentially incorporates IEEE 754 into C. – R.. GitHub STOP HELPING ICE Aug 25 '15 at 05:25
  • @n.m. As I wrote, _unclear_. I edited it just now. I think, but am not certain, that 754 is the standard here, so I made it even more general. – geometrian Aug 25 '15 at 05:25
  • 5
    If you want to ask about the IEEE 754 standard, mention it somewhere in the question. – n. m. could be an AI Aug 25 '15 at 05:27
  • 3
    @n.m.: They say a lot about NaNs, actually, all over the place, unless the standard diverges wildly from the draft. [Ctrl-F the C11 final draft for NaN and you'll get 374 hits.](http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf) – user2357112 Aug 25 '15 at 05:27
  • https://software.intel.com/sites/default/files/article/326703/fp-control-2012-08.pdf – Arun Aug 25 '15 at 05:34
  • @user2357112 you are right, sorry, my memory is failing, – n. m. could be an AI Aug 25 '15 at 05:43
  • 68
    I was _sure_ this question was about JavaScript from the title. – Mike G Aug 25 '15 at 12:51
  • GCC 5.1.0 on Ubuntu does the same as your first example output. I haven't used any switches that alter the floating point model, so everything regarding that should be on default. I only used -O3, which does not mess with the floating point model. – notadam Aug 25 '15 at 15:43
  • Why is the `-nan` weird? – ChiefTwoPencils Aug 25 '15 at 18:46
  • 1
    @ChiefTwoPencils I think OP just means "weird" in the generic "this is kind of unintuitive" sense, i.e., "what does it mean for NaN to be negative in the first place," rather than in the sense of "unexpected and possibly not right." My guess is that OP didn't realize prior to this test that `-nan` was defined by the floating-point spec. – Kyle Strand Aug 25 '15 at 18:52
  • 1
    @imallett I've made what I hope you'd agree is a minimal edit to your question in order to clarify the standards aspect of the question (since the top-rated answer is helpful but not quite what you seem to be looking for). – Kyle Strand Aug 25 '15 at 18:56
  • 1
    @KyleStrand Please don't substitute your own judgement for the OP's in defining the question. He says he is interested in "behavior in both C and C++" and you have edited that out to make it a standards question. The OP could have asked on Programming if this was a conceptual standards question. By posting on Stack Overflow, the practical programming Q&A site, he is making this a practical question. How compilers actually behave is a practical programming matter, and thus more on-topic here. – Paul Aug 26 '15 at 21:54
  • @Paul Er...C/C++ questions on this site are quite often about the C/C++ standards, and even when they're not explicitly about the standards they're often answered by reference to the standard(s). In fact, [I have been rebuffed when attempting to ask about practical behavior not covered by the standard](http://stackoverflow.com/q/32132058/1858225). But more to the point, the question (pre-edit!) said "what does the standard say about this". So I did not "make it a standards question"; it was (quite obviously) *already* a standards question. I merely clarified *which* standards are relevant. – Kyle Strand Aug 26 '15 at 21:59
  • @Paul If you wanted to argue that I clouded matters or altered the intent of the question by adding IEEE when the user had only asked about the C/C++ standards, that might be a reasonable position to take. But to say that the question was not about standards before my edit is simply not true. – Kyle Strand Aug 26 '15 at 22:01
  • 3
    Indeed, I was asking about the standards involved. @ouah's is my favored answer at the moment, since it is the only one that actually addresses them. – geometrian Aug 26 '15 at 22:12
  • @imallett I believe you can give Petr their second gold badge if you accept ouah's answer: http://stackoverflow.com/help/badges/62/populist – Kyle Strand Aug 26 '15 at 22:16
  • @KyleStrand I was mostly annoyed that the behavioral curiosity was being edited out. Having provided some insight that was well received earlier, new readers are complaining I didn't answer the question. Certainly both the de-facto working standard (an actual compiler that most developers usually use) and a standards document are important in that code might be safe for a while when both agree and there is precedent that future development should keep them in agreement. – Paul Aug 26 '15 at 22:46
  • @KyleStrand For your earlier question, it is a shame no one answered, but it would be a lot of work to answer that one from nothing, as it needs a test suite, multiple versions, etc. Could be done, though, with a test suite and some docker containers. – Paul Aug 26 '15 at 22:46
  • 1
    @Paul Ah, I hadn't predicted a negative impact on existing answers, since in my mind I preserved the original intent of the question. That makes more sense. I don't think it was unreasonable of me to interpret the parenthetical as merely a clarification of the stated question about "the standard", but you're right that my edit removes all mention of "behavior." I think if it's going to be present at all, it shouldn't be within a parenthetical. I've made another edit. – Kyle Strand Aug 26 '15 at 22:52
  • OP, is the current version acceptable? – Kyle Strand Aug 26 '15 at 22:52
  • @Paul I'm less annoyed that no one answered my question than I am that there was resistance to the idea that an answer might be valuable in the first place. – Kyle Strand Aug 26 '15 at 22:54
  • 2
    @KyleStrand Yes; the current version is acceptable. My original _question_ was about the standards. Since the _reason_ turned out to be mainly about compilers, I think it's best to make the _answers_ address both. – geometrian Aug 27 '15 at 01:24
  • You may want to add some `volatile` here. – curiousguy Aug 27 '15 at 04:55
  • What's weird about negative NaN is that `neg < 0` is `false`: http://melpon.org/wandbox/permlink/o8JwN6RwOmOme9wY – Joseph Aug 29 '15 at 00:42

3 Answers3

304

The default floating point handling in Intel C++ compiler is /fp:fast, which handles NaN's unsafely (which also results in NaN == NaN being true for example). Try specifying /fp:strict or /fp:precise and see if that helps.

Petr Abdulin
  • 33,883
  • 9
  • 62
  • 96
  • 15
    I was just trying this myself. Indeed, specifying either precise or strict fixes the problem. – geometrian Aug 25 '15 at 05:30
  • 69
    I'd like to endorse Intel's decision to default to `/fp:fast`: if you want something _safe_, you should probably better avoid NaNs turning up in the first place, and generally don't use `==` with floating-point numbers. Relying on the weird semantics that IEEE754 assigns to NaN is asking for trouble. – leftaroundabout Aug 25 '15 at 14:24
  • 10
    @leftaroundabout: What do you find weird about NaN, aside from the IMHO horrible decision to have NaN!=NaN return true? – supercat Aug 25 '15 at 16:41
  • 22
    NaNs have important uses - they can detect exceptional situations without requiring tests after every calculation. Not every floating-point developer needs them but don't dismiss them. – Bruce Dawson Aug 25 '15 at 17:03
  • @leftaroundabout Choice of fast vs. safe is consistent with the handling of [FDIV bug](https://en.wikipedia.org/wiki/Pentium_FDIV_bug): something rarely encountered by average users, so go with whatever solution is fastest. – chux - Reinstate Monica Aug 25 '15 at 18:17
  • 6
    @supercat Out of curiosity, do you agree with the decision to have `NaN==NaN` return `false`? – Kyle Strand Aug 25 '15 at 18:52
  • 6
    @KyleStrand: No, but if NaN!=NaN had returned true, code could at least use inverted logic with != to guard against some problem cases. What was really needed (and for some bizarre reason *still* isn't generally available) is a standardized efficient means of performing a fully-ordered relational comparison. While there are times when having NaN be unordered can be useful, there are other times, such as when trying to sort a list of numbers, where it wreaks havoc on everything. If I were designing a language, I would have multiple types of float and double which were mutally-convertible... – supercat Aug 25 '15 at 19:06
  • 2
    ...but used different NaN-comparison semantics (operations would only be allowed between values with the same semantics, except that non-NaN constants would be freely convertible to any other form); I'd have the "general-purpose" type use the fully-ranking operators. – supercat Aug 25 '15 at 19:09
  • @supercat Huh. I like that. – Kyle Strand Aug 25 '15 at 19:17
  • 4
    @supercat NaNs being unordered is important in scientific computing to preserve the correct semantics of indeterminate forms. I'm guessing this use case is more common/more important than the efficiency of sorting lists containing NaN. – user168715 Aug 26 '15 at 04:15
  • @leftaroundabout: Your position is that, if I want something "safe", I better not use any of the features that help me write code that's both fast and safe because you think they're weird? – tmyklebu Aug 26 '15 at 04:40
  • @supercat If you were sorting a list of numbers containing NaN, would you place it towards positive or negative infinity? (And isn't it generally easy enough to just preprocess the list to remove all NaNs?) – Pharap Aug 26 '15 at 08:47
  • @tmyklebu: no, you'd better use _other_ features that help you write code that's both fast and safe. In particular, anything that can be guaranteed by the type system is preferrable to runtime checks (be they via NaN or without). C++ templates give you a lot of possibilities here, Hindley-Milner type systems such as OCaml's or Haskell's are even better. – leftaroundabout Aug 26 '15 at 08:53
  • @user168715: Both forms are needed at different times, but (1) even in scientific computing I would expect that having an operator which regards NaN as being not unequal to anything (including other NaN) would be about as important as one which regards it as not equal to anything, so I don't understand the basis for making NaN return false for everything *but* inequality, and (2) The IEEE should have very strongly from the get-go directed that implementations should provide *both* a fully-ordered means of comparison and a "scientific" one, and also that it provide functions to compute... – supercat Aug 26 '15 at 13:17
  • 1
    ...trig functions in units of circles or quadrants. While there are some purposes for which radians are the natural unit, there are just as many where calculations end up with a factor of pi which could be omitted if circles were used as units. Given that it's necessary for many sine and cosine implementations to convert to a power-of-two fraction of a circle anyway, requiring client code to multiply by pi and library code to multiply by 1/pi is not a recipe for high precision. – supercat Aug 26 '15 at 13:19
  • 4
    @user168715: BTW, I also think there should have been four zero-ish values: zero (*anything* plus or minus zero yields itself), positive and negative infinitesimal (result of underflow on multiplication or division), and unsigned infinitesimal (result of subtraction yielding zero result). Division by a signed infinitesimal should give +/-inf, but division by zero or unsigned infinitesimal should yield NaN. Such an arrangement would make all operations properly symmetric between positive and negative numbers. – supercat Aug 26 '15 at 13:22
  • 5
    @leftaroundabout: Have you ever written numerical code? Fancy type systems will not save you. The way NaNs are handled was carefully designed so that you can still check for errors but you don't have to pollute your code with explicit, rarely-taken conditionals everywhere. – tmyklebu Aug 26 '15 at 14:05
  • 2
    @supercat: IEEE 754-2008 (or however you punctuate it) does recommend that implementations provide both a total order and the usual floating-point comparisons. Unfortunately, they aren't in a position to dictate syntax to the C and C++ committees. I believe it also recommends `sinpi` and `cospi` and friends for your second complaint. I agree, though; pity it wasn't there at the beginning. – tmyklebu Aug 26 '15 at 14:08
  • `sinpi` etc in addition to (but not replacing) the traditional versions in radians is a fine idea but unrelated to the question of how NaNs should be handled. – user168715 Aug 26 '15 at 14:28
  • @tmyklebu: I work on [MHD](https://en.wikipedia.org/wiki/Magnetohydrodynamics) codes, and therefore know for sure that NaNs do not “save you” very efficiently. Of course, when they pop up you can generally be pretty sure that things have gone awry, but not every error manifests itself as a NaN anywhere at all. Yet even without them, there will often be obvious and efficiently-checkable hints that _something_ is wrong (e.g. negative density). – But none of these hints are very helpful for debugging, because the error could usually be anywhere in a very large section of code. For that... – leftaroundabout Aug 26 '15 at 14:44
  • ...kind of debugging, you'll need to turn on your inefficient local checks – which indeed you may as well do anyway for a quick check _before_ each processing-hungry production run! But even better than that is if most bugs are caught right by the compiler, and that's where strong type systems help. Of course they don't find _every_ error, either, but they can find a heck of a lot of errors. Admittedly I've not yet worked with such codes in C++11, OCaml of Haskell to confirm it; but I ascribe the lack of such codes mostly to academic inertia. – leftaroundabout Aug 26 '15 at 14:44
  • 3
    @leftaroundabout I agree with tmyklebu that type inference is useless here. (Almost) nobody writes NaNs at compile time: they arise due to underflow/overflow over the course of many iterations of complex calculation and predicting them is rarely easier than running the program itself. Consider for instance the simulation of a three-body problem: if any two of the three bodies come too close to each other, NaNs will arise (due to overflow in the force proportional to $1/r^2$) but of course the three-body problem is the classic example of a chaotic system that cannot be analyzed at compile time. – user168715 Aug 26 '15 at 14:46
  • 1
    @user168715: well, in that particular example a type system wouldn't help because it's not really a bug in the program, rather in the problem set-up (why are so close, overflow-inducing vicinities possible in the first place?) – leftaroundabout Aug 26 '15 at 14:54
  • 2
    @leftaroundabout Right but unfortunately that's how NaNs arise in practice. One might imagine a code that takes a large number of time steps of numerical integration for the three-body problem (for efficiency) and periodically checks for NaNs and other problems, and if detected, rewinds time (a la Jefferson's Time Warp) and reruns those iterations with special (more expensive) processing that handles body collisions. – user168715 Aug 26 '15 at 14:57
  • 1
    @user168715: right, but you don't need guaranteed “poison-progression” of NaNs to do that. Checking for impossibly-high kinetic energies will achieve much the same result. – leftaroundabout Aug 26 '15 at 15:10
  • 1
    @tmyklebu: I'm aware that IEEE started recommending such things in 2008, but has anyone acted upon that? Breaking the default equality relation rather than having a function to test whether two variables are "scientifically" equal makes it almost impossible for a language to support IEEE semantics while also specifying that the `==` operator will define an equivalence relation for combinations of operands where it compiles [if I were designing a language, I'd include "ieee32" and "ieee64" types which have a `.ieee_eq()` method but disallow the `==` operator, and have `==` test equivalence]. – supercat Aug 26 '15 at 15:46
  • 1
    @leftaroundabout: I have never had a numerical booboo that a fancy type system would catch. (But I've also never worked on anything related to physics. I do optimisation, and we don't have units.) NaN propagation lets you write clear logic for the common case and *provably* detect if something went awry. Checking for "impossibly high kinetic energies" might usually work in a physics simulation, but one can likely design an input for which it fails. – tmyklebu Aug 26 '15 at 16:57
  • @supercat: Some fancy math libraries like CRlibm have `trigpi` functions. You're in a deep state of sin if you need `<` on floating-point numbers both to act as `<` on floating-point numbers should and to act as a total order. – tmyklebu Aug 26 '15 at 17:01
  • @tmyklebu: There are a lot of cases where floating-point algorithms can benefit from memoization. If `f(x)` is expensive, and its output depends only on `x`, then keeping a list of `x` values and the results of calling `f(x)` can often be a useful optimization, but the only ways I know to do that require a means of checking whether two floating-point values are equivalent, and the only ways I know to do it efficiently require means of either hashing or ranking floating-point values. Is memoization obscure, or are there ways of doing it efficiently without the above features? – supercat Aug 26 '15 at 17:28
  • Memoization isn't obscure, but it also doesn't require a total ordering. Convert the `double` via `memcpy` or similar to an integral type of equal size, such as `int64_t`, and use that `int64_t` as a key. (Or `array – tmyklebu Aug 26 '15 at 18:39
  • As a side note, Intel C++ compiler has the (now deprecated in favor of /fp and default off) /Op option, that documentation describes as: *"Enables conformance to the ANSI C and IEEE 754 standards for floating-point arithmetic".* and *"Specifying the /Op option has the following effects on program compilation: [...] Floating-point arithmetic comparisons conform to the IEEE 754 specification except for NaN behavior."* The documentation does not go into much details in the optimizations/non-conformance of /fp option. – ouah Aug 26 '15 at 21:42
  • 1
    @tmyklebu: There are ways of getting a total ordering from `double` in C, but hacks involving `memcpy` or even `union` can end up forcing extra register->memory round trips. Further, I don't think the usefulness of IEEE-754 was supposed to be limited to languages which support such bit-level hackery. – supercat Aug 26 '15 at 22:45
  • Can we all just agree that floating point numbers prove the existence of God by negation, as they are clearly the work of the Devil? :-) – Bob Jarvis - Слава Україні Aug 28 '15 at 11:05
  • 1
    @supercat: Have you looked at Rust? It defines 4 traits (interfaces) for comparison: `PartialEq`, `PartialOrd`, `Eq` and `Ord`, the latter two are your run off the mill providers of `==`, `!=`, `<`, `<=`, `>` and `>=` however floating point types (`f32` and `f64`) do not implement `Eq` and `Ord` but only `PartialEq` and `PartialOrd`: https://doc.rust-lang.org/std/cmp/trait.PartialOrd.html – Matthieu M. Aug 28 '15 at 15:07
  • @MatthieuM.: I've looked very briefly at Rust; I like the conceptual distinction between "let" and "let mut", but don't know much else about it. How are partial orderings used, and what can one do if one needs a total ordering on reals (with NaN mapped either high or low)? – supercat Aug 28 '15 at 16:09
  • @supercat: you can use the partial orderings by calling the methods of the trait: `fn partial_cmp(&self, &other) -> Option` returns either no ordering or an ordering; for floating points, comparing with `NaN` does not yield an ordering. If you wish to define a total ordering, you create a new type: `struct MyF32(f32);` and then you can implement the interface for your type and decide how to proceed with the comparison. The main advantage is that it forces people to explicitly pick the behavior that makes sense for them. – Matthieu M. Aug 28 '15 at 17:34
  • @MatthieuM.: I didn't mean to ask by what means one would ask objects about a partial ordering, but rather how one could usefully employ it. Are there any useful generic algorithms which require any kind of ordering that don't require a full ordering? – supercat Aug 28 '15 at 18:35
  • @supercat: To be honest? I have no idea! – Matthieu M. Aug 28 '15 at 19:27
  • Speaking of Intel's fp:fast default, it matches roughly gcc -ffast-math -fno-cx-limited-range, while fp:fast=2 includes -complex-limited-range and more. I am among the minority who would prefer that abrupt underflow and k&r style disregard of parentheses not be included in these options. I also have reservations about the way reciprocal-math is handled under these options. Yet I use them most of the time. – tim18 Jun 11 '16 at 19:39
54

This . . . can't be right, right? My question: what do the relevant standards (ISO C, ISO C++, IEEE 754) say about this?

Petr Abdulin already answered why the compiler gives a 0.0 answer.

Here is what IEEE-754:2008 says:

(6.2 Operations with NaNs) "[...] For an operation with quiet NaN inputs, other than maximum and minimum operations, if a floating-point result is to be delivered the result shall be a quiet NaN which should be one of the input NaNs."

So the only valid result for the subtraction of two quiet NaN operand is a quiet NaN; any other result is not valid.

The C Standard says:

(C11, F.9.2 Expression transformations p1) "[...]

x − x → 0. 0 "The expressions x − x and 0. 0 are not equivalent if x is a NaN or infinite"

(where here NaN denotes a quiet NaN as per F.2.1p1 "This specification does not define the behavior of signaling NaNs. It generally uses the term NaN to denote quiet NaNs")

Community
  • 1
  • 1
ouah
  • 142,963
  • 15
  • 272
  • 331
21

Since I see an answer impugning the standards compliance of Intel's compiler, and no one else has mentioned this, I will point out that both GCC and Clang have a mode in which they do something quite similar. Their default behavior is IEEE-compliant —

$ g++ -O2 test.cc && ./a.out 
neg: -nan
sub: nan nan nan
add: nan nan
div: nan nan nan
mul: nan nan

$ clang++ -O2 test.cc && ./a.out 
neg: -nan
sub: -nan nan nan
add: nan nan
div: nan nan nan
mul: nan nan

— but if you ask for speed at the expense of correctness, you get what you ask for —

$ g++ -O2 -ffast-math test.cc && ./a.out 
neg: -nan
sub: nan nan 0.000000
add: nan nan
div: nan nan 1.000000
mul: nan nan

$ clang++ -O2 -ffast-math test.cc && ./a.out 
neg: -nan
sub: -nan nan 0.000000
add: nan nan
div: nan nan nan
mul: nan nan

I think it is entirely fair to criticize ICC's choice of default, but I would not read the entire Unix wars back into that decision.

zwol
  • 135,547
  • 38
  • 252
  • 361
  • Notice that with `-ffast-math`, `gcc` is not complying to ISO 9899:2011 with respect to floating point arithmetic any more. – fuz Sep 01 '15 at 11:12
  • 1
    @FUZxxl Yes, the point is that *both* compilers have a noncompliant floating-point mode, it's just that icc defaults to that mode and gcc doesn't. – zwol Sep 01 '15 at 22:06
  • 4
    Just to throw fuel in the fire, I really like Intel's choice to enable fast math by default. The whole point of using floats is to get high throughput. – Navin Sep 14 '15 at 08:44