0

I've been trying to round a float value to 4 precision without success.

float fconv = 1.0f;
float fdata = 39.934543423412f;
float fres = RoundTo(fdata*fconv, -4);

if(fres <= 39.9345f){do something;} //<-- unwanted behavior

Wanted result is 39.934500000000
Actual result is 39.934543423412

I've tried many methods including Round a float to a given precision without success.

I'm working on an AMD FX83xx 64bit. Program is built in 32bit Debug using XE7

Thanks

Community
  • 1
  • 1
Azmaedus
  • 3
  • 4

1 Answers1

0

Your desired precision of 6 decimal digits is very near the precision limits for a float data type. The epsilon, or delta between consecutive representable floating point values, for a number around 40f is about 7.63E-6, so there's only a couple of bits different between the 'best' value and what you're getting. This is possibly due to rounding that close to the limit, but I'm not sure.

1201ProgramAlarm
  • 32,384
  • 7
  • 42
  • 56