Floating point precision not matching for the value what was actually assigned?

Question

I am working on STM32F777 mcu creating application on top of FreeRTOS. STM32F777 comes with double-precision floating point unit. I found this really annoying issue while using floating point units that when I assign a value to a float using a constant or something, for example:

float x = 78.352361;

the actual value that I see assigned to variable x when debugging came out to be: 78.3523636

Similarly, when I assigned 17.448636 to the variable x, what I actually saw assigned when debugging was 17.4486351.

Any ideas, why this is happening and how to resolve this? I want correct precision to be maintained till 6 digits.

Answer 1

The most common floating-point format, IEEE 754, includes a basic 32-bit format and a basic 64-bit format, and these are commonly used for the float and double types in C. For brevity, I will call them float and double in this answer.

Neither of these types can exactly represent non-integer numbers other than those that are multiples of a power of two (such as ¼, ¾, 1/1024, 73/1048576). Every other number will be changed slightly when it is converted from decimal to float or double.

However, the float has the property that rounding any decimal numeral with six significant digits (such as 1.2345 or 9.87654e23) to float and back to six significant decimal digits returns the original number (provided the number is within normal bounds of the format). In C, this number of digits is reported by the value FLT_DIG, which is defined by float.h. Since your number 78.352361 has eight significant digits, it is not guaranteed to survive a conversion to float and back.

For double, at least 15 digits will survive a round trip, reported by DBL_DIG.

Note that this is the number of decimal digits guaranteed to survive the rounding caused by one conversion to binary floating-point and back to the original number of decimal digits. If additional arithmetic is performed in floating-point, additional roundings occur, which may accumulate more error. And, if a value is formatted with more decimal digits than the original, then the result may differ from the original number. (For example, .9f produces “.9” when converted back to one decimal digit but “0.899999976” when converted to nine decimal digits.)

Since double guarantees that 15 digits survive a round trip, your number 78.352361 would survive a conversion to double and back to eight significant digits unchanged. Additionally, there is enough precision to perform some arithmetic without accumulating so much error that it is visible in eight significant decimal digits. However, floating-point arithmetic can be tricky, and a complete error analysis depends on the operations you perform.

Answer 2

Floating point variables cannot store arbitrary real numbers, they can only store members of a particular subset (of cardinality slightly under 2^32 for a 32-bit IEEE float) of the real numbers. The "changes" you see are because you assigned a value that is not one of the values that a float can store on your system. It was rounded to a value that can be stored.

In order to store 6 decimal places perfectly, for latitude or longitude, you should use a 32-bit integer (representing 1 million times the value to store).

Answer 3

You asked, "isn't there any way to solve this?" There are basically two ways:

1. Understand the limitations of floating point, and learn how to work within them.
2. Use fixed point, not floating point.

The essential thing to understand about floating point is that there's no such number as, say, 17.445652 decimal. There's no such number as 17.445651999999999, either. Internally, the actual binary floating-point number is somewhere in between. That's just the way it is. So if you're using floating point, and you want, say, six digits of precision, and if you're using a format which guarantees six digits of precision (which double does, sort of), you're always going to need to round appropriately. (You may also have to remember not to get confused if, say, your debugger shows you an unrounded value.)

Or, if for whatever reason rounding is unacceptable, you can use fixed point, but you'll have to do it by hand, since it's not built in to the language. Multiply all your latitudes and longitudes by 1000000, and store them internally as long int, and when you print them back out, stick in a '.' 6 places from the end. For example:

long int latitude = 17445652;    /* microdegrees */

printf("latitude = %ld.%06ld\n", latitude / 1000000, latitude % 1000000);

Answer 4

In a comment you asked,

Can you please tell me how do you calculate the precision in terms of distance from the lat/lon degree value? Like what place after the decimal digit accounts for how much precision in terms of distance?

This is obviously a totally separate question, and I probably shouldn't answer it here, but it's a fun problem, so I can't resist. You can actually work it out from first principles.

A nautical mile was originally defined as one minute of longitude at the equator. So the earth's circumference is 360 x 60 = 21600 nautical miles. And a nautical mile is 1852 meters, so that's 40003200 meters, or 40003.2 kilometers. And this is a nice result, because the meter was originally defined as, I think, 1/40000000 of the earth's circumference pole-to-pole, so we're on the right track.

With the earth's circumference (which is obviously 360 degrees) in hand, we can work out these distances:

degrees     distance (m)
1           111120
0.1         11112
0.01        1111.2
0.001       111.12
0.0001      11.112
0.00001     1.1112
0.000001    0.1111
0.0000001   0.0111
0.00000001  0.0011

So 0.000001 degrees is about 0.1 meter, or 10 centimeters. And 0.00000001 degrees is about a millimeter.

Those numbers all apply to longitude at the equator, or to latitude anywhere. For longitudes not at the equator, multiply by the cosine of the latitude. (For example, I live at latitude 42. Here, 0.000001 degrees of longitude is about 0.111 x 0.743 = 0.082 meters = 8 cm.)

These are all approximate results. The earth is not a perfect sphere, and when you start doing precise navigation, it turns out this can make quite a difference, though not too much to the numbers I've shown here.