1

I have a problem with a double (TaxRate here), that do not have the same value when it pass through functions:

I first call a function

DataContext.UpdateCategory(theCategorie.Code, theCategorie.Description, theCategorie.TaxRate, theCategorie.TaxCode);

When I run the debugger the value of TaxRate is 19.6

Then when I put a stop point in the dbml function that is called:

 [Function(Name="dbo.UpdateCategory")]
 public int UpdateCategory([Parameter(Name="Code", DbType="VarChar(20)")] string code, [Parameter(Name="Description", DbType="NVarChar(512)")] string description, [Parameter(Name="TaxRate", DbType="Float")] System.Nullable<double> taxRate, [Parameter(Name="TaxCode", DbType="NChar(10)")] string taxCode)
 {
     IExecuteResult result = this.ExecuteMethodCall(this, ((MethodInfo)(MethodInfo.GetCurrentMethod())), uNSPSC, description, taxRate, taxCode);
     return ((int)(result.ReturnValue));
 }

here the value is 19.6000003814697. You see the strange added decimal? there are no operations between these two calls, so why these decimals appears?

Gert Arnold
  • 105,341
  • 31
  • 202
  • 291
Gregoire
  • 24,219
  • 6
  • 46
  • 73
  • 1) http://stackoverflow.com/questions/872544/precision-of-floating-point 2) http://stackoverflow.com/questions/1694545/floating-point-again 3) http://stackoverflow.com/questions/590822/dealing-with-accuracy-problems-in-floating-point-numbers – KristoferA Nov 30 '09 at 02:04
  • 3
    This has been beat to death. Read What Every Computer Scientist Should Know About Floating Point (http://docs.sun.com/source/806-3568/ncg_goldberg.html), change to decimal types since you're working with monetary figures and move on. – jason Nov 30 '09 at 02:14
  • Yes, but the problem comes from the dbml generation, that doest not use the same type between the stored procedure parameter (double) and the class property (float) (even if the both are float SqlType) – Gregoire Nov 30 '09 at 02:41
  • Change the SQL type to decimal. – jason Nov 30 '09 at 02:52
  • Yes it what I have done thanks to your advice, but I persist there is a problem with the dbml generation – Gregoire Nov 30 '09 at 02:55
  • The LINQ-to-SQL generator (or sqlmetal) should read the type correctly and use CLR type decimal. Even if it does not, you can manually change the type in the designer to CLR type decimal. – jason Nov 30 '09 at 02:58

1 Answers1

1

Is theCategorie.TaxRate a float? If so, you're assigning a float into a double. The extra decimals is due to higher precision in the double, so the double's nearest to a float's 19.6...

The output of this illustrates it:

float f = 19.6F;
double d = f;
double d2 = 19.6D;

System.Diagnostics.Debug.WriteLine("Float: " + f.ToString());
System.Diagnostics.Debug.WriteLine("Double from float: " + d.ToString());
System.Diagnostics.Debug.WriteLine("Double: " + d2.ToString());

As a workaround, you can either change the type of theCategorie.TaxRate to a double, or if you want to keep it as a float you can cast & round it when calling the stored proc:

DataContext.UpdateCategory(theCategorie.Code, theCategorie.Description, Math.Round((double)theCategorie.TaxRate, 6), theCategorie.TaxCode);
KristoferA
  • 12,287
  • 1
  • 40
  • 62