1

I'm writing a set of numeric type conversion functions for a database engine, and I'm concerned about the behavior of converting large integral floating-point values to integer types with greater precision.

Take for example converting a 32-bit int to a 32-bit single-precision float. The 23-bit significand of the float yields about 7 decimal digits of precision, so converting any int with more than about 7 digits will result in a loss of precision (which is fine and expected). However, when you convert such a float back to an int, you end up with artifacts of its binary representation in the low-order digits:

#include <iostream>
#include <iomanip>

using namespace std;

int main()
{
    int a = 2147483000;
    cout << a << endl;
    float f = (float)a;
    cout << setprecision(10) << f << endl;
    int b = (int)f;
    cout << b << endl;

    return 0;
}

This prints:

2147483000
2147483008
2147483008

The trailing 008 is beyond the precision of the float, and therefore seems undesirable to retain in the int, since in a database application, users are primarily concerned with decimal representation, and trailing 0's are used to indicate insignificant digits.

So my questions are: Are there any well-known existing systems that perform decimal significant digit rounding in float -> int (or double -> long long) conversions, and are there any well-known, efficient algorithms for doing so?

(Note: I'm aware that some systems have decimal floating-point types, such as those defined by IEEE 754-2008. However, they don't have mainstream hardware support and aren't built into C/C++. I might want to support them down the road, but I still need to handle binary floats intuitively.)

Trevor Robinson
  • 15,694
  • 5
  • 73
  • 72
  • This situation sends up a red flag, asking "Why is there a round trip conversion?" This implies that the value should be stored in the database as an integer and not as a float. Kind of like simplifying an algebraic expression. – Thomas Matthews Dec 10 '09 at 21:18
  • Good question. I certainly don't plan for there to be round-trip conversions, and nothing in the database engine itself will cause this to happen. However, this database is designed for densely packed, in-memory data, so I need to use the smallest types possible. I don't want users (who can submit their own SQL-like queries) to think they're getting a lot more precision than they really are when they convert a binary float to a higher-precision int or decimal float. – Trevor Robinson Dec 11 '09 at 20:17
  • When you use a database, you should carefully select the type in the database and in the code. Most of the time, you know if you want decimals or not. If you don't want decimals, then use integer or long integer. If you want decimal, then you want to use double (and rarely float unless you are really sure that you need very limited precision). **Instead of doing special cases conversions, I would suggest raising an error if C++ type does not match database type.** – Phil1970 Nov 07 '21 at 22:19

2 Answers2

1

std::numeric_limits<float>::digits10 says you only get 6 precise digits for float.

Pick an efficient algorithm for your language, processor, and data distribution to calculate-the-decimal-length-of-an-integer (or here). Then subtract the number of digits that digits10 says are precise to get the number of digits to cull. Use that as an index to lookup a power of 10 to use as a modulus. Etc.

One concern: Let's say you convert a float to a decimal and perform this sort of rounding or truncation. Then convert that "adjusted" decimal to a float and back to a decimal with the same rounding/truncation scheme. Do you get the same decimal value? Hopefully yes.

This isn't really what you're looking for but may be interesting reading: A Proposal to add a max significant decimal digits value to the C++ Standard Library Numeric limits

Community
  • 1
  • 1
Dan
  • 5,929
  • 6
  • 42
  • 52
0

Naturally, 2147483008 has trailing zeros if you write it in binary (1111111111111111111110110000000) or hexadecimal (0b0x7FFFFD80). The most "correct" thing to do would be to track insignificant digits in any of those forms instead.

Alternatively, you could just zero all digits after the first seven significant ones in the int (ideally by rounding) after converting to it from a float, since the float contains approximately seven significant digits.

Josef Grahn
  • 1,585
  • 9
  • 12
  • Excellent clarifying point. However, I don't really expect binary or hex to be used to display data stored in this database. My question really comes down to the wisdom/precedent and most efficient means of implementing your second paragraph. – Trevor Robinson Dec 10 '09 at 19:57