0

Simple math or so I thought. I get the correct answer for 99.99% of values but this one was spotted in live environment.

I need to round down a float value to 4 places, dropping anything after the 4th place so this example shouldn't be an issue.

0.6720/0.672 is the correct answer answer, why are example trunc & int returning .6719?

((12.0000 * 5.6000) / 100) = 0.672

Trunc(((12.0000 * 5.6000) / 100) * 10000) / 10000) = .6719

Int(((12.0000 * 5.6000) / 100) * 10000) / 10000) = .6719

I added a listview and button to a form to get below example

procedure TForm1.Button1Click(Sender: TObject);
var
  x, y : Double;
  Itm: TListItem;
begin
  x := 12.0000;
  y :=  5.6000;

  Itm := ListView1.Items.Add;
  Itm.Caption := 'a. FloatToStr((12.0000 * 5.6000) / 100) ';
  Itm.SubItems.Add(FloatToStr((x * y) / 100));

  Itm := ListView1.Items.Add;
  Itm.Caption := 'b. ((12.0000 * 5.6000) / 100) * 10000 ';
  Itm.SubItems.Add(FloatToStr(((x * y) / 100) * 10000));

  Itm := ListView1.Items.Add;
  Itm.Caption := 'c. ((12.0000 * 5.6000) / 100) * 10000 ';
  Itm.SubItems.Add(FloatToStr(((x * y) / 100) * 10000));

  Itm := ListView1.Items.Add;
  Itm.Caption := 'd. Trunc(((12.0000 * 5.6000) / 100) * 10000) / 10000) ';
  Itm.SubItems.Add(FloatToStr(  Trunc(((x * y) / 100) * 10000) / 10000) );

  Itm := ListView1.Items.Add;
  Itm.Caption := 'e. Int(((12.0000 * 5.6000) / 100) * 10000) / 10000) ';
  Itm.SubItems.Add(FloatToStr(  Int(((x * y) / 100) * 10000) / 10000) );
end;

Screen shot of calculation results

david
  • 11

1 Answers1

-1

I once had a similar problem...

   x:=Trunc(12 * 100 * 5.6) ;      // 6720

   x:=Trunc(12 * 5.6 * 100) ;        // 6719

   x:=12*5.6*100;
   x:=trunc(x);       // 6720

Could be used RoundTo

   x:=RoundTo(12*5.6*100,0);    // 6720
Marek_P
  • 9
  • 2