1

I am working in our Land Management Software which includes a custom field called ESFT_BILLINGEDU, this field is used to combine several other custom fields in the database and then divide the total by 20 to get our billing EDU's for sewer calculations (I work for a public agency). The answer displayed is always off by .2 even if whole numbers divisible by 20 are used.

Our software does not support use of DECIMAL, so I cannot convert to that (which would likely solve the issue). I have tried using CONVERT to INTEGER, not using CONVERT at all and simply dividing, and using CAST and ROUND to see if I can get the correct result. All have failed.

CONVERT([float],CONVERT([float](2),((((((((((((((((((((((((((isnull([EXT1_BAR_SINK],(0))*(2)+isnull([EXT1_BATHTUB],(0))*(4))+isnull([EXT1_CLWASH],(0))*(4))+isnull([EXT1_DEN_LAV],(0))*(1))+isnull([EXT1_DISHWASH],(0))*(4))+isnull([EXT1_FLOOR_DRN],(0))*(1))+isnull([EXT2_KIT_SINK],(0))*(4))+isnull([EXT2_LAUN_TUB],(0))*(4))+isnull([EXT2_LAVY_SNGL],(0))*(2))+isnull([EXT2_LAVY_DBL],(0))*(4))+isnull([EXT2_MOP_SINK],(0))*(4))+isnull([EXT2_SHOW],(0))*(4))+isnull([EXT2_URIN_WALL],(0))*(5))+isnull([EXT2_WATER_CLOSET],(0))*(6))+isnull([EXT2_WCL_FLUSH],(0))*(10))+isnull([ext1_cir_sink],(0))*(4))+isnull([ext1_cup_sink],(0))*(1))+isnull([ext1_den_unit],(0))*(1))+isnull([ext1_drink_foun],(0))*(1))+isnull([ext1_flushing],(0))*(10))+isnull([ext2_rv_dump],(0))*(20))+isnull([ext2_rv_spaces],(0))*(8))+isnull([ext2_urin_flsh],(0))*(3))+isnull([ext2_urin_step_on],(0))*(10))+isnull([ext2_wash_sink_set],(0))*(3))+isnull([EXT1_FLOOR_SINK],(0))*(4))+((((((((((((((((((((((((isnull([ENEW_SINK],(0))*(2)+isnull([ENEW_BATH],(0))*(4))+isnull([ENEW_CLTHWASH],(0))*(4))+isnull([ENEW_DEN_LAV],(0))*(1))+isnull([ENEW_DISHWASH],(0))*(4))+isnull([ENEW_FLR_DRN],(0))*(1))+isnull([ENEW2_KIT_SINK],(0))*(4))+isnull([ENEW2_LAUN_TUB],(0))*(4))+isnull([ENEW2_LAVY_SNGL],(0))*(2))+isnull([ENEW2_LAVY_DBL],(0))*(4))+isnull([ENEW2_MOP_SINK],(0))*(4))+isnull([ENEW2_SHOWER],(0))*(4))+isnull([ENEW2_URIN_Wall],(0))*(5))+isnull([ENEW2_WCL_Home],(0))*(6))+isnull([ENEW2_WCL_FLUSH],(0))*(10))+isnull([enew2_urin_flush],(0))*(3))+isnull([enew_cir_spray],(0))*(4))+isnull([enew_cup_sink],(0))*(1))+isnull([enew_den_unit],(0))*(1))+isnull([enew_drink_foun],(0))*(1))+isnull([enew_flushing],(0))*(10))+isnull([enew2_rv_dump],(0))*(20))+isnull([enew2_rv_space],(0))*(8))+isnull([enew2_urin_step],(0))*(10))+isnull([ENEW_FLR_SINK],(0))*(4)))+isnull([enew2_wash_sink],(0))*(3)))/CONVERT([float],(20),(0)),(0)),(0)))

the fields within the [] are the custom screen fields. The last line of code divides these totals by 20 to get a number. I cannot get the field to correctly divide, it is always off by .2 For example, in one of our permits the number of Existing Fixtures (the first part of the code is 230, the number of the new fixtures (second part of code is 223 for a total of 453. This divided by 20, last line of code..equals 22.65, the current code gives the answer of 22.4500007629395)

Hesperia
  • 11
  • 2
  • 1
    Please format your code correctly so its readable. – Omkar Aug 19 '19 at 20:58
  • 2
    That is a beastly formula. I would put good honest money on this being a floating point precision issue. Since you can't use Decimal, there may be no way around it. – JNevill Aug 19 '19 at 20:59
  • 1
    your software may not support decimals, but your database does. Store the value in DECIMAL and display it as INTEGER. – Omkar Aug 19 '19 at 21:00
  • I can't imagine it's helpful, but for those that want something readable I ran through really big formula through a pretty printer. It's now ugly in a different way. – JNevill Aug 19 '19 at 21:03
  • There has to be a better way. Perhaps if you supply a SMALL/reduced data sample and desired results. – John Cappelletti Aug 19 '19 at 21:06
  • It seems to me like the biggest issue here is the lack of properly normalized tables. You seem to have repeating groups which violates 1NF. From the sounds of things it is years past the ability to fix such a thing though. – Sean Lange Aug 19 '19 at 21:06
  • Did you try `CAST`ing the results instead of `CONVERT`ing them? https://stackoverflow.com/questions/11719044/how-to-get-a-float-result-by-dividing-two-integer-values – Omkar Aug 19 '19 at 21:08
  • 1
    floats / floats = rounding issues like crazy. – Sean Lange Aug 19 '19 at 21:10
  • @SeanLange The floats are all added first, so the division happens only once. Shouldnt be that big of a deal tbh. – Omkar Aug 19 '19 at 21:10
  • @Omkar but they are not all floats added together. Each column has ISNULL with an int so each column may or may not be implicitly converted to a float (or whatever the column datatype is). There is a ton of implicit conversion going on here. – Sean Lange Aug 19 '19 at 21:28
  • Ah, you're right @SeanLange. The `CAST` needs to be given inside the `ISNULL` clause. :/ (Is there a better way to do it though?) – Omkar Aug 19 '19 at 21:36
  • Neither of those expressions is valid. Can you make this into a usable repro. `float` has the highest data type precedence of any number type, and calculations involving `float` should be accurate to 15 significant digits. – David Browne - Microsoft Aug 19 '19 at 21:47
  • Hi All,new to Stackoverflow and the response is amazing! In answer to Omkar, I tried to cast them as DECIMAL and when I visited the backend of our software the field was null and there was an error box that the field type is not supported. I changed them back to FLOAT and the field re-appeared in our DB backend software. I also cleaned up my code above, when I saved it added whitespace, making it hard to read. – Hesperia Aug 19 '19 at 22:35
  • @Hesperia -Welcome to StackOverflow, you have actually removed an edit from another user (@JNevill) who added it in an attempt to make your formula more readable. Can you possibly add a sample of your data structure and sample data. I suspect there is a much better way to do this. – Java Devil Aug 19 '19 at 22:44
  • This formula (the technical term for it is "hairball" :-) is worthy of a usable unit-testable stored function. In that stored function you can use any datatype you wish, as long as its output is in the desired datatype. – O. Jones Aug 19 '19 at 23:18

1 Answers1

2

I made a temporary table to represent the source values, then removed the premature casting, and it just worked. I believe SQL converted the values to a suitable intermediate type and did the correct math because we didn't force the use of a "limiting" datatype that causes rounding issues. I added one cast at the end so your front-end code could read a FLOAT result value.

The values that I inserted into the temporary row exactly fit your specification that the first group of values totaled to 230 and the last group totaled to 223, resulting in 22.65. I removed the extra parenthesis because SQL already multiplies before adding.

create table #temp (
   [EXT1_BAR_SINK]             float null, 
   [EXT1_BATHTUB]              float null,
   [EXT1_CLWASH]               float null,
   [EXT1_DEN_LAV]              float null,
   [EXT1_DISHWASH]             float null,
   [EXT1_FLOOR_DRN]            float null,
   [EXT2_KIT_SINK]             float null,
   [EXT2_LAUN_TUB]             float null,
   [EXT2_LAVY_SNGL]            float null,
   [EXT2_LAVY_DBL]             float null,
   [EXT2_MOP_SINK]             float null,
   [EXT2_SHOW]                 float null,
   [EXT2_URIN_WALL]            float null,
   [EXT2_WATER_CLOSET]         float null,
   [EXT2_WCL_FLUSH]            float null,
   [ext1_cir_sink]             float null,
   [ext1_cup_sink]             float null,
   [ext1_den_unit]             float null,
   [ext1_drink_foun]           float null,
   [ext1_flushing]             float null,
   [ext2_rv_dump]              float null,
   [ext2_rv_spaces]            float null,
   [ext2_urin_flsh]            float null,
   [ext2_urin_step_on]         float null,
   [ext2_wash_sink_set]        float null,
   [EXT1_FLOOR_SINK]           float null,
   [ENEW_SINK]                 float null,
   [ENEW_BATH]                 float null,
   [ENEW_CLTHWASH]             float null,
   [ENEW_DEN_LAV]              float null,
   [ENEW_DISHWASH]             float null,
   [ENEW_FLR_DRN]              float null,
   [ENEW2_KIT_SINK]            float null,
   [ENEW2_LAUN_TUB]            float null,
   [ENEW2_LAVY_SNGL]           float null,
   [ENEW2_LAVY_DBL]            float null,
   [ENEW2_MOP_SINK]            float null,
   [ENEW2_SHOWER]              float null,
   [ENEW2_URIN_Wall]           float null,
   [ENEW2_WCL_Home]            float null,
   [ENEW2_WCL_FLUSH]           float null,
   [enew2_urin_flush]          float null,
   [enew_cir_spray]            float null,
   [enew_cup_sink]             float null,
   [enew_den_unit]             float null,
   [enew_drink_foun]           float null,
   [enew_flushing]             float null,
   [enew2_rv_dump]             float null,
   [enew2_rv_space]            float null,
   [enew2_urin_step]           float null,
   [ENEW_FLR_SINK]             float null,
   [enew2_wash_sink]           float null
) ;

insert into #temp (
[EXT1_BAR_SINK]      ,
[EXT1_BATHTUB]       ,
[EXT1_CLWASH]        ,
[EXT1_DEN_LAV]       ,
[EXT1_DISHWASH]      ,
[EXT1_FLOOR_DRN]     ,
[EXT2_KIT_SINK]      ,
[EXT2_LAUN_TUB]      ,
[EXT2_LAVY_SNGL]     ,
[EXT2_LAVY_DBL]      ,
[EXT2_MOP_SINK]      ,
[EXT2_SHOW]          ,
[EXT2_URIN_WALL]     ,
[EXT2_WATER_CLOSET]  ,
[EXT2_WCL_FLUSH]     ,
[ext1_cir_sink]      ,
[ext1_cup_sink]      ,
[ext1_den_unit]      ,
[ext1_drink_foun]    ,
[ext1_flushing]      ,
[ext2_rv_dump]       ,
[ext2_rv_spaces]     ,
[ext2_urin_flsh]     ,
[ext2_urin_step_on]  ,
[ext2_wash_sink_set] ,
[EXT1_FLOOR_SINK]    ,
[ENEW_SINK]          ,
[ENEW_BATH]          ,
[ENEW_CLTHWASH]      ,
[ENEW_DEN_LAV]       ,
[ENEW_DISHWASH]      ,
[ENEW_FLR_DRN]       ,
[ENEW2_KIT_SINK]     ,
[ENEW2_LAUN_TUB]     ,
[ENEW2_LAVY_SNGL]    ,
[ENEW2_LAVY_DBL]     ,
[ENEW2_MOP_SINK]     ,
[ENEW2_SHOWER]       ,
[ENEW2_URIN_Wall]    ,
[ENEW2_WCL_Home]     ,
[ENEW2_WCL_FLUSH]    ,
[enew2_urin_flush]   ,
[enew_cir_spray]     ,
[enew_cup_sink]      ,
[enew_den_unit]      ,
[enew_drink_foun]    ,
[enew_flushing]      ,
[enew2_rv_dump]      ,
[enew2_rv_space]     ,
[enew2_urin_step]    ,
[ENEW_FLR_SINK]      ,
[enew2_wash_sink]    
)
 values ( 
     1.0, 2.0, 1.0, 2.0, 3.0, 4.0, 3.0, 7.0, 1.0, 3.0
   , 3.0, 2.0, 1.0, 1.0, 2.0, 1.0, 5.0, 3.0, 2.0, 1.0
   , 1.0, 3.0, 3.0, 1.0, 2.0, 3.0, 1.0, 1.0, 2.0, 3.0
   , 1.0, 2.0, 2.0, 2.0, 1.0, 2.0, 3.0, 2.0, 1.0, 3.0
   , 3.0, 2.0, 1.0, 1.0, 1.0, 1.0, 2.0, 1.0, 2.0, 1.0
   , 1.0, 2.0
    ) ;

SELECT convert(float,
      (isnull([EXT1_BAR_SINK], 0)*(2)
      +isnull([EXT1_BATHTUB], 0)*(4)
      +isnull([EXT1_CLWASH], 0)*(4)
      +isnull([EXT1_DEN_LAV], 0)*(1)
      +isnull([EXT1_DISHWASH], 0)*(4)
      +isnull([EXT1_FLOOR_DRN], 0)*(1)
      +isnull([EXT2_KIT_SINK], 0)*(4)
      +isnull([EXT2_LAUN_TUB], 0)*(4)
      +isnull([EXT2_LAVY_SNGL], 0)*(2)
      +isnull([EXT2_LAVY_DBL], 0)*(4)
      +isnull([EXT2_MOP_SINK], 0)*(4)
      +isnull([EXT2_SHOW], 0)*(4)
      +isnull([EXT2_URIN_WALL], 0)*(5)
      +isnull([EXT2_WATER_CLOSET], 0)*(6)
      +isnull([EXT2_WCL_FLUSH], 0)*(10)
      +isnull([ext1_cir_sink], 0)*(4)
      +isnull([ext1_cup_sink], 0)*(1)
      +isnull([ext1_den_unit], 0)*(1)
      +isnull([ext1_drink_foun], 0)*(1)
      +isnull([ext1_flushing], 0)*(10)
      +isnull([ext2_rv_dump], 0)*(20)
      +isnull([ext2_rv_spaces], 0)*(8)
      +isnull([ext2_urin_flsh], 0)*(3)
      +isnull([ext2_urin_step_on], 0)*(10)
      +isnull([ext2_wash_sink_set], 0)*(3)
      +isnull([EXT1_FLOOR_SINK], 0)*(4)
      +isnull([ENEW_SINK], 0)*(2)
      +isnull([ENEW_BATH], 0)*(4)
      +isnull([ENEW_CLTHWASH], 0)*(4)
      +isnull([ENEW_DEN_LAV], 0)*(1)
      +isnull([ENEW_DISHWASH], 0)*(4)
      +isnull([ENEW_FLR_DRN], 0)*(1)
      +isnull([ENEW2_KIT_SINK], 0)*(4)
      +isnull([ENEW2_LAUN_TUB], 0)*(4)
      +isnull([ENEW2_LAVY_SNGL], 0)*(2)
      +isnull([ENEW2_LAVY_DBL], 0)*(4)
      +isnull([ENEW2_MOP_SINK], 0)*(4)
      +isnull([ENEW2_SHOWER], 0)*(4)
      +isnull([ENEW2_URIN_Wall], 0)*(5)
      +isnull([ENEW2_WCL_Home], 0)*(6)
      +isnull([ENEW2_WCL_FLUSH], 0)*(10)
      +isnull([enew2_urin_flush], 0)*(3)
      +isnull([enew_cir_spray], 0)*(4)
      +isnull([enew_cup_sink], 0)*(1)
      +isnull([enew_den_unit], 0)*(1)
      +isnull([enew_drink_foun], 0)*(1)
      +isnull([enew_flushing], 0)*(10)
      +isnull([enew2_rv_dump], 0)*(20)
      +isnull([enew2_rv_space], 0)*(8)
      +isnull([enew2_urin_step], 0)*(10)
      +isnull([ENEW_FLR_SINK], 0)*(4)
      +isnull([enew2_wash_sink], 0)*(3))
      / 20.0) as [ESFT_BILLINGEDU]
from #temp ;

drop table #temp ;

Result:

ESFT_BILLINGEDU
22.65
  • Worked perfectly! Thank you so much! That fixed hundreds of records in our database that were performing incorrectly! – Hesperia Aug 20 '19 at 21:47