1

loss of precision when using sqrtl() with long long int

I need to use a variable to store the value since I would expect the user to type in their own value in range 0 to 10^18, so it makes sense to use long long int in this situation. But this is where the error occurs which I could not solve it properly.

unsigned long long int a = 999999999999999999;
cout << fixed;
cout << (floor(sqrtl(999999999999999999)) == sqrtl(999999999999999999)) << endl;
cout << floor(sqrtl(999999999999999999)) << endl;
cout << (sqrtl(999999999999999999)) << endl;

cout << a << endl;
cout << (floor(sqrtl(a)) == sqrtl(a)) << endl;
cout << floor(sqrtl(a)) << endl;
cout << (sqrtl(a)) << endl;

In my machine, the output is

0
999999999.000000
1000000000.000000
999999999999999999
1
1000000000.000000
1000000000.000000

which shows the loss precision when using sqrtl(a) compare to sqrtl(999999999999999999)

I want to be able to get the correct value while using the variable

Davide Spataro
  • 7,319
  • 1
  • 24
  • 36
Glen Fung
  • 19
  • 1
  • 4
    Floating point variables represent an approximation of the actual value stored. Therefore, unless the value supplied is EXACTLY equal to one of the values that the type can represent, there will be an error introduced. This error may be positive or negative. All you have demonstrated is that the value of `sqrtl(a)` is slightly less than the value you expect i.e. the error is negative for that case. Printing rounds to the nearest value at the specified precision (e.g. `0.12345` printed to four significant figures will produce `0.1235`) whereas `floor()` is specified to round down. – Peter Aug 17 '19 at 05:59
  • Exactly what @Peter mentioned. C++ treats floats and doubles really differently than whole numbers. You cannot compare then using normal means in C++ you need to check against an error of [Epsilon in C++](https://en.cppreference.com/w/cpp/types/numeric_limits/epsilon) to get the closest comparison. – kokeen Aug 17 '19 at 06:10
  • @Peter No, in this case the value is slightly more than the correct value. – user202729 Aug 17 '19 at 06:10
  • @Glen So you want to compute floor(sqrt(x)) exactly for a large value of x? – user202729 Aug 17 '19 at 06:12
  • @user202729 Yes – Glen Fung Aug 17 '19 at 06:15
  • There is https://stackoverflow.com/questions/28993752/approximation-error-when-using-sqrt-and-floor check if it works. – user202729 Aug 17 '19 at 06:24
  • BTW: Formally, you would be required to provide a [mcve] here, though in this case the question is clear without it. As a new user, also take the [tour] and read [ask]. The latter explains about another thing missing from your question: The expected output! – Ulrich Eckhardt Aug 17 '19 at 06:46
  • In this case the code is almost compilable (needs `int main`, some include, some `using`) Not strictly required in all cases but sometimes the bug lies in the part you think is not important. – user202729 Aug 17 '19 at 06:53
  • 1
    @t.niese Not really. It only explains the incorrect behavior but does not provide OP with a working solution. – user202729 Aug 17 '19 at 08:00
  • Does this code really compile? Those numeric literals all need L on them, at least. – user207421 Aug 17 '19 at 08:32
  • I cannot reproduce that on the machine where I tried, please specify the computer/compiler/options you used, at least. – Marc Glisse Aug 17 '19 at 09:01
  • 999999999999999999 cannot be exactly represented in double precision. And this is likely due to the higher precision registers in x87. Avoid it and use SSE instead. There are many possible duplicates: [double variable not equal to double expression?](https://stackoverflow.com/q/37626687/995714), [Apparently identical math expressions with different output](https://stackoverflow.com/q/33795031/995714), [Difference in casting float to int](https://stackoverflow.com/q/54881541/995714), [Difference in floating point arithmetics between x86 and x64](https://stackoverflow.com/q/22710272/995714) – phuclv Aug 17 '19 at 09:29
  • [Why does this float have 2 different values?](https://stackoverflow.com/q/50894217/995714), [How is float variable auto-promoted to double type?](https://stackoverflow.com/q/12118738/995714) – phuclv Aug 17 '19 at 09:29
  • `long double` is not guaranteed to have 64 bits of precision either. Also avoid using `sqrtl`, use `std::sqrt` instead and allow the user to automatically choose the correct overload – phuclv Aug 17 '19 at 09:40
  • You might start with the [Wikipedia page on integer square roots](https://en.wikipedia.org/wiki/Integer_square_root). – Eric Postpischil Aug 17 '19 at 09:45

0 Answers0