-1

I have a lot of data in Science notation when I load this data into my double variable, everything works fine (in VS I see value 0.00000000022). But when I multiply this number by 1000000 I got unrounded value (0.00021999999999999998).

enter image description here

I must multiply this value because I use it for selection filter. After send data from selection filter I again divide this data to my raw format 0.00021999999999999998 / 1000000 = 0.00000000022.

2.20E-10 = 0.00000000022 * 1000000 = 0.00021999999999999998

Expected value is this:

0.00021999999999999998 => 0.00022

When I use a similar number, for example, 2.70E-10 I got after multiple value 0.00027 (In this case conversion work fine).

Values are converted only for use in the selection menu so that no unnecessary zeros are shown and the label indicates which unit they represent. (For this example from Ohm to microOhm in select box)

Is there any way to correctly convert these values?

I use LINQ for convert this values in like this:

var x = y.Select(s => s.Resistance * 1000000).Distinct();
Jan Sršeň
  • 1,045
  • 3
  • 23
  • 46
  • http://floating-point-gui.de/ or Oracle: [What Every Computer Scientist Should Know About Floating-Point Arithmetic](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html) – Fildor Nov 16 '17 at 14:20
  • See also : https://stackoverflow.com/questions/618535/difference-between-decimal-float-and-double-in-net – PaulF Nov 16 '17 at 14:22
  • Won't converting s.Resistance * 1000000 to System.Decimal and applying Round (see [link](https://msdn.microsoft.com/library/6be1edhb(v=vs.110).aspx) do the trick? – Oleksandr Tyshchenko Nov 16 '17 at 14:22

1 Answers1

2

problem is floating point numbers, a good article about this is link

use decimal type instead of double for fast solving :)

mostafa
  • 261
  • 1
  • 2
  • 10
  • I solved this problem like this, how easy :D `var x = y.Select(s => (decimal) s.Resistance * 1000000).Distinct();` – Jan Sršeň Nov 20 '17 at 15:39