Out-of-range floating point to integer conversion breaks in VS2022 executable when linking VS2017 or VS2019 libraries

Question

I have a snippet of code that's present in a VS2017 static library. When linked in a 2017 executable, it works as expected. However, if linking into a 2022 executable, it breaks on the cast from double to uint64_t.

The double is below -1.0 so the truncated value is outside the value-range of uint64_t and ISO C doesn't define the behaviour, but MSVC does.

static uint64_t do_something_2(double f1)
{
    return (uint64_t)(f1 - 0.5);
}

uint32_t do_something(void)
{
    double f1 = -3406302.4613481420;
    uint32_t f2 = (uint32_t)do_something_2(f1);

    return f2;
}

When linking this into a 2022 executable, f2 is 0xffffffff, as if from double to uint64_t with AVX-512 vcvttsd2usi rax, xmm0 which produces all-ones for out-of-range values. (And then uint64_t to uint32_t truncation to get a 32-bit 0xffffffff value)

However, if I re-compile the static library using VS2022, I get my expected value of f2 = 4291560994 (0xffcc0622), as if (uint32_t)(uint64_t)double did get modulo reduction of the integer -3406302.

(For values that also fit in int64_t, we can get this result portably and efficiently with (uint64_t)(int64_t)double, especially in x64 code or with x87+SSE3 fisttp. But the question is why existing code written with (uint64_t)double doesn't compile the way I expected when mixing VS versions.)

---

When I compare the disassembly, I see the 2017-generated code just calls __dtoul3 and nothing more.

The 2022 generated assembly is much more involved with many more calls.

Afterwards, I edited my 2017 makefile by adding /arch=IA32 to disable SSE2 and then recompiled. This resulted in the correct value being computed.

So it seems to be an issue with the SSE2 code generation in VS2017 vs VS2022. However, this blog post on Microsoft seems to suggest that 2022 and 2017 should be compatible: https://devblogs.microsoft.com/cppblog/microsoft-visual-studio-2022-and-floating-point-to-integer-conversions/ It mentions changes to the default semantics for out-of-range FP to integer conversions to match AVX-512 instructions.

Only other things I'll add:

1. Tried compiling the library with 2019 and linking into the 2022 executable. Same issue as 2017, results in 0xFFFFFFF

2. The 2017 library linked into a 2017 executable does work.

Any idea why my findings seem to contradict Microsoft's blog post about floating point to integer conversions between 2017 and 2022?

Answer 1

However, if linking into a 2022 executable, it breaks on the cast from double to uint64_t.

Undefined behavior (UB)

This is allowed by C.

When a finite value of real floating type is converted to an integer type ..., the fractional part is discarded (i.e., the value is truncated toward zero). If the value of the integral part cannot be represented by the integer type, the behavior is undefined. C17dr § 6.3.1.4 1

Only floating point values in the [-0.99999.... to 18,446,744,073,709,551,616.99999....] range are defined to convert via (uint64_t) as OP hoped. Since the 3406302.4613481420 - 0.5 is ≤ -1.0, the conversion is UB. @Peter Cordes

double f1 = -3406302.4613481420;
uint64_t y = (uint64_t)(f1 - 0.5);  // UB

---

I'm just curious as to why 2022 isn't compatible with 2017.

OP's code relied on UB and so should not expect the current or prior compiler versions to act consistently. It is not a compiler bug, but an OP one.

---

Other questionable functionality

f1 - 0.5 is not exact with odd values less than and near 2⁵³ and so form an incorrect rounded result.

Further, other values for do_something_2() may not round as OP might hope. To take further, the exact goal of do_something_2() is needed.

---

Alternative

Perform the modulo step with fmod() and a test.

static uint64_t do_something_2(double f1) {
    f1 = fmod(f1, 18446744073709551616.0); // remainder.
    if (f1 < 0.0) f1 += 18446744073709551616.0;
    return (uint64_t)(f1 - 0.5);
}

This may/may not do exactly what OP wants for all f1 as the overall goal of do_something_2() remains unstated.

Answer 2

A MSFT engineer explained the cause for this in a comment here:

https://developercommunity.visualstudio.com/t/Out-of-range-floating-point-to-integer-c/10324943#T-ND10356428

This happens for historical reasons, and I agree that it is not ideal. When targeting 32-bit IA, Visual Studio uses helper functions for various operations, including conversions between integer and floating-point types. In VS2017 the functions for conversion from floating-point to unsigned types are quite slow. For VS2019 we wanted to use faster functions, and the best versions available use Intel AVX-512 instructions for these conversions when possible and return the same values as those instructions when not.

Until VS2019 version 16.7, MSVC did not fully define behavior for conversion of floating-point values to integer types and uses whatever machine instructions are available to implement these conversions efficiently. The 8087 floating-point coprocessor, introduced in 1980, was the base for Intel Architecture floating-point support, and it only supports conversion to signed integer types. For those types it always returns an integer indefinite value for invalid conversions. Since 8087 supports conversion to 64-bit signed integer, and the allowed values for unsigned 32-bit integers are as subset of those values, Microsoft C used conversion to signed 64-bit integer and truncation to 32 bits to convert floating-point values to 32-bit unsigned integer type.

When integer type support was extended to 64-bit signed and unsigned integers, at first all 64-bit conversions were treated as signed conversions, so that floating-point values larger than the maximum 64-bit signed integer were converted to the 64-bit signed integer indefinite value 0x8000000000000000 even when the result type was unsigned, but eventually special code was added to correctly handle values 0x8000000000000000 to 0xFFFFFFFFFFFFFFFF. All invalid conversions continued to be handled as conversions to signed 64-bit integer. This was the defacto behavior of integer conversions up through VS2017.

VS2012 changed the default floating-point support from 8087 to SSE2, and instruction set extensions from SSE to AVX2 support only conversion to 32-bit signed integer on x86, so all other conversions must be emulated by range-checking and emulation using integer instructions. The original helper functions for these conversions did full emulation of the 8087 conversions including certain bugs that have since been fixed, but this made them much slower than the 8087 conversions, which was not ideal. In 2017, Intel launched the Skylake-X processors which support the Intel AVX-512 instruction set extensions and include instructions for converting floating point values to all 32-bit and 64-bit signed and unsigned integer types. For VS2019 we wanted to use these instructions to improve the performance of the floating-point to integer conversions. The AVX-512 conversions from floating-point to unsigned integer types also return integer indefinite values for all invalid conversions, but those values are different from the values for signed integers. For any integer type, the corresponding integer indefinite value is the value farthest from zero. So, for 32-bit signed integers it is 0x80000000, and for 64-bit unsigned integers it is 0xFFFFFFFFFFFFFFFF. If the result is any other value, the conversion was valid. So, the AVX-512 conversions from floating-point to 32-bit unsigned integer return 0xFFFFFFFF for invalid conversions, while the defacto VS2017 conversions can return any value.

For VS2019 we should have introduced yet another new set of helper functions for conversions compatible with AVX-512 instructions and offered a command switch to select between the new functions and the prior functions, but that is not what we did. Because the behavior of the prior routines is not particularly useful and didn’t match any other MSVC target platform, we thought it would be okay to update the existing helpers with new, faster code that used AVX-512 instructions when possible and emulated them when not.

Based on feedback from customers, we realized that we had made a mistake, but our options for correcting it are limited. Reverting the helper functions could change the behavior of VS2019 compiled code similarly to what you are seeing, so that is not an option. In VS2019 version 16.7 we introduced the /fpcvt command switch to select between VS2017-compatible behavior and AVX-512 compatible behavior, which overrides the mix of behaviors exhibited in VS2019 by the 32-bit and 64-bit IA compilers with various /arch settings. However, for x86 this only works when linking with the corresponding helper functions. In VS2022 we default back to the legacy behavior, although the /fpcvt:IA switch will still generate AVX-512 compatible code, and the helper functions are still compatible with VS2019.

The reason VS2022 is generating additional code is that it is trying to emulate the older functionality using the new helper functions. We are adding new helper functions to emulate the older behavior directly, but for reasons I won’t go into, it takes several version updates for those changes to propagate everywhere they need to go. When those functions have propagated, we will update MSVC to directly use them and all conversions will be either generating a conversion instruction or calling the appropriate helper function.

When linking VS2017 code into a VS2022 executable, the call to __dtoul3 will go to the updated VS2022 helper function which returns the 64-bit unsigned integer indefinite value (0xFFFFFFFFFFFFFFFF) instead of the signed 64-bit integer value the VS2017 __dtoul3 helper would have returned. This is expected.

I do not have a specific workaround to offer you. If you need non-standard behavior, I suggest that you should use a conversion function that fully specifies what values should be returned for all argument values. This will give your code maximum portability at some cost in performance. In VS2022 there are intrinsic functions that return integer indefinite values or saturated integer values for all invalid conversion arguments.