2

I am working on an application that calculates ppm and checks if it is greater than a certain threshold. I recently found out the precision error of floating point calculation. 

double threshold = 1000.0;
double Mass = 0.000814;
double PartMass = 0.814;
double IncorrectPPM = Mass/PartMass * 1000000;
double CorrectedPPM = (double)((decimal)IncorrectPPM);

Console.WriteLine("Mass = {0:R}", Mass);
Console.WriteLine("PartMass = {0:R}", PartMass);
Console.WriteLine("IncorrectPPM = {0:R}", IncorrectPPM);
Console.WriteLine("CorrectedPPM = {0:R}", CorrectedPPM);
Console.WriteLine("Is IncorrectPPM over threshold? " + (IncorrectPPM > threshold) );
Console.WriteLine("Is CorrectedPPM over threshold? " + (CorrectedPPM > threshold) );

The above codes would generate the following outputs:

Mass = 0.000814
PartMass = 0.814
IncorrectPPM = 1000.0000000000002
CorrectedPPM = 1000
Is IncorrectPPM over threshold? True
Is CorrectedPPM over threshold? False

As you could see, the calculated ppm 1000.0000000000002 has a trailing 2 which causes my application to falsely judge that the value is over the 1000 threshold. All inputs to the calculation are given to me as double values so I couldn't use decimal calculation. In addition, I couldn't round the calculated value since it could cause the threshold comparison to be incorrect.

I noticed that if I cast the calculated double number into decimal and then cast it back to double again the 1000.0000000000002 number got corrected into 1000.

Question:
Does anyone know how the computer know in this case that it should change the 1000.0000000000002 value to 1000 when casting to decimal?
Can I rely on this trick to avoid the precision issue of double calculation?

Andre Kampling
  • 5,476
  • 2
  • 20
  • 47
ccyen
  • 33
  • 6
  • `threshold` is too small. In reality any measurement has a tolerance, that's your expected value + threshold. – Sinatr Sep 01 '17 at 07:47
  • The threshold 1000 ppm is decided by an actual regulation so I couldn't adjust it. Also, both Mass and PartMass are designated values so I don't think there is measurement error involved. – ccyen Sep 01 '17 at 08:03
  • If it's not measurements, then you should stick to `decimal`, see [this](https://stackoverflow.com/q/1165761/1997232). – Sinatr Sep 01 '17 at 08:08

3 Answers3

2

Either your threshold is too small or you round up the result to a certain amount of decimals. The more decimals the more precise your evaluation.

double threshold = 1000.0;
double Mass = 0.000814;
double PartMass = 0.814;
double IncorrectPPM = Mass/PartMass * 1000000;
double CorrectedPPM = Math.Round(IncorrectPPM,4); // 1000.0000 will output 1000

You can be as precise as you want.

Bert Sinnema
  • 319
  • 2
  • 14
  • If the calculated ppm value turns out to be something like 1000.000035 and I round it to 4 decimal places, I would end up having the incorrect comparison again because in this case 1000.000035 is legit over the threshold but the rounded value would become 1000, which is not over the threshold. How precise should I round up the result to make sure it won't be affected by this double precision issue? – ccyen Sep 01 '17 at 08:33
  • @ccyen: That depends on your requirements. We can't know how precise you have to be. +1 For this answer! – Andre Kampling Sep 01 '17 at 08:37
  • 1
    4 is just an example. Who decides what is legit over? if 1000.000035 is legit for you Round to 6 decmials. – Bert Sinnema Sep 01 '17 at 08:38
  • I see. I was under the false impression that there exists a specific decimal place to which rounding could always resolve this kind of precision issue. I understand now that it is an decision on how much I should tolerate the precision errors. – ccyen Sep 01 '17 at 09:12
  • @ccyen You are in control. It's the beauty of developing software. – Bert Sinnema Sep 01 '17 at 09:31
2

Does anyone know how the computer know in this case that it should change the 1000.0000000000002 value to 1000 when casting to decimal?

First of all the cast:

(decimal)IncorrectPPM

is equivalent to the constructor call, see here on SO:

new decimal(IncorrectPPM)

If you read on the MSDN page about the decimal constructor you will find the following remark:

This constructor rounds value to 15 significant digits using rounding to nearest. This is done even if the number has more than 15 digits and the less significant digits are zero.

That means

1000.0000000000002 
               ^ ^  
            15th 17th significant digit

will be rounded to:

1000.00000000000 
               ^ 
            15th significant digit

Can I rely on this trick to avoid the precision issue of double calculation?

No, you can't imagine the following result when calculating IncorrectPPM, see online at ideone:

1000.000000000006
               ^  
            15th significant digit

will be rounded to:

1000.00000000001
               ^  
            15th significant digit

To resolve your issue about the comparison with your threshold, you have in general 2 possibilities.

  1. Add a little epsilon to your threshold, e.g.:

    double threshold = 1000.0001;

  2. Change your cast of IncorrectPPM from:

    double CorrectedPPM = (double)((decimal)IncorrectPPM);

    to:

    /* 1000.000000000006 will be rounded to 1000.0000 */
double CorrectedPPM = Math.Round(IncorrectPPM, 4);

    with the Math.Round() function, but be careful Math.Round() means fractional not significant digits

Andre Kampling
  • 5,476
  • 2
  • 20
  • 47
  • I see. So casting to decimal doesn't help in this case. Since I don't have decimal values as my input. Is there still anything I could do to resolve this issue? – ccyen Sep 01 '17 at 08:24
  • As the other answer said use [Math.Round()](https://msdn.microsoft.com/en-us/library/75ks3aby(v=vs.110).aspx) to decide on you own how many digits you want to leave to make it more tolerant for comparison. – Andre Kampling Sep 01 '17 at 08:27
  • Sorry for the late response. I just have one last question. With the example number 1000.000000000006, if I round it to 12 decimal place instead of rounding to 4 decimal places I would end up having the wrong comparison result. So does that mean having more decimal places for rounding doesn't necessary mean I am more tolerant of errors? – ccyen Sep 01 '17 at 09:28
  • 1
    @ccyen: You have to use 10 as second argument because `Math.Round()` doesn't mean siginficant digits. It means resulting fractional digits and the 6 is the 12 fractional digit. See example: https://ideone.com/J8VAb3 – Andre Kampling Sep 01 '17 at 09:33
  • Yes, I was aware of that. I now know that there exist some cases where rounding to 12 decimal places can actually be worse off than rounding to 10 decimal places, even though it is more precise to round to 12 decimal places. – ccyen Sep 01 '17 at 09:53
  • In short: Rounding to less fractional digits makes your comparison more tolerant. – Andre Kampling Sep 01 '17 at 09:54
0

There is a fundamental difference in decimal and double in terms of precision and that is rooted in the way the number is stored:

  • decimal is a fixed point number, which means it provides a fixed number of decimals. When you do calculations with fixed point numbers, you need to perform rounding operations quite frequently. For example if you divide the number 100.0006 by 10, you have to round the value to 10.0001, given your decimal point is fixed with four decimals. This rounding error is usually fine in economical calculations, since you have enough decimals available and the point is fixed to decimal places (i.e. to the base of 10).
  • double is a floating point number, which is stored differently. It has a sign (defining if the number is positive or negative), a mantissa and an expontent. The mantissa is a number between 0 and 1 with a certain amount of significant digits (that is the actual precision we are talking about). The exponent defines where the decimal point is located in the mantissa. So the decimal point moves in the mantissa (hence floating point). Floating point numbers are great for calculating measured values and doing scientific calculations, since the precision stays the same through calculations and rounding errors can be minimized.

The fundamental problem that you are facing is that you can only rely on the value of a floating point number within its precision. That includes the actual precision of the mantissa and the accumulated rounding error through your calculation. Additionally, since the mantissa is stored in binary format and you convert it into a decimal number when you compare it with 1000, you have an additional imprecision through this conversion. You usually don't have this problem in a fixed point number, since the significant decimal digits are clearly defined (and you accept the rounding error during calculation).

In practice this means that when you compare floating point numbers you have to always be aware how many digits are significant. Note that means the total number of digits (i.e. the ones before and after the decimal point). Once you know the precision (or choose one that works for you and provides enough margin of error), you can decide to how many digits you need to round your value for your comparison. Let's say according to your data a precision of six decimal digits is appropriate, you can compare against your threshold like this:

bool isWithinThreshold = Math.Round(PPM, 6) > 1000D;

Note that you only round for the comparison, you don't round your value.

What you are doing with a conversion to decimal is to implicitly apply the precision of the decimal to the floating point number. That's nothing more than the preferred solution of rounding, just with less control over the precision and an additional performance impact. So no, a conversion to decimal is not reliable, especially with large numbers.

Sefe
  • 13,731
  • 5
  • 42
  • 55