2

I have a class that does some length calculations based on a height on a ticket. It's been in place for years and working quite well... Until we got a unique ticket size.

They are entered by sales people in inches and are normally nice numbers like 3, 4 or 3.5 and store in a database - This one is however 3.66666 recurring (or 11/3) But it is being entered as 3.666 and causing the calculation to fail due to lost precision.

I have thought of a bit of a hack to restore precision for certain numbers, but thought maybe someone knows of a better way of getting a 3.666 or a 93.1333 back to it's number + two thirds status?

Thanks, Mick.

Mick m
  • 55
  • 7
  • How would one know if **3.666** in the database was meant to be _recurring_ or if it was just meant to be _"3.666"_? If one can not know the difference how would one know _when_ to attempt to restore precision? –  Sep 28 '15 at 10:43
  • 1
    How does the difference between 3.66666 and 3.666 fail your calculation? – netaholic Sep 28 '15 at 10:44
  • Can you do the processing on the string? It may be easier to notice that it ends in a recurring digit, and append more copies of that digit. – Patricia Shanahan Sep 28 '15 at 10:59
  • Completely unclear what you think should happen. 11/3 is neither 3.666 nor 3.66666. I second the question of how one causes a calculation to "fail". Define "fail". – DonBoitnott Sep 28 '15 at 11:51
  • @ DonBoitnott 11 / 3 is 3.666... recurring. @netaholic - A ticket is less than 4 inch and it's on a roll of paper about 14,000 foot long and there are multiple rolls, so the loss of precision is calculating the tickets knowing the roll length. I thought it maybe an no-go question, but my hack was to check the last few digits and see if they are the same, but just wondered if anyone on here had come across a similar problem and come up with a smarter solution - Thanks anyway, – Mick m Sep 29 '15 at 08:27

2 Answers2

0

It would be difficult to get the accurate value of double as double is floating point.

The MSDN says:

Remember that a floating-point number can only approximate a decimal number, and that the precision of a floating-point number determines how accurately that number approximates a decimal number. By default, a Double value contains 15 decimal digits of precision, although a maximum of 17 digits is maintained internally. The precision of a floating-point number has several consequences:

  • Two floating-point numbers that appear equal for a particular precision might not compare equal because their least significant digits are different.

  • A mathematical or comparison operation that uses a floating-point number might not yield the same result if a decimal number is used because the floating-point number might not exactly approximate the decimal number.

Rahul Tripathi
  • 168,305
  • 31
  • 280
  • 331
0

As you explained in comments I see your point now. I've checked the numbers: 

  168000 / 3.666     = 45826.5139
  168000 / 3.666666  = 45818.1901488
  168000 * 3 / 11    = 45818.1818182

It makes a difference of 8 tickets. I have a feeling that your issue can be solved in many ways. On the side of user input for example. Or on the side of database. But back to your question:

How do I convert 3.666 or a 93.1333 back to it's number + two thirds status?

You are looking for converting decimal (or double) to fraction. There is already a question on SO: Algorithm for simplifying decimal to fractions which has many answeres. I've tested some of them, and none of them were satisfying. Some of them don't even hanlde recurrence. Perhaps I've missed the correct one, you can look by yourself.

Anyway, I believe you don't need to fully implement a conversion from 1.666 to 3/2, since it's not easy and you have a real-world sizes. You've said, that most of the time numbers are aroung 3, 3.5, 4 etc. So I suggest you to take a look at a question I've linked above and search for an algorythm of detecting the recurrence number. It was also discussed here How to know the repeating decimal in a fraction? After what just convert 1.666 to 1.666666, since 1/1000000 of inch won't mess your calculations, as numbers above show.

Community
  • 1
  • 1
netaholic
  • 1,345
  • 10
  • 22
  • @netholic Thank you - That second link is really helpful. I shall work on my searching technique :) Thanks again. – Mick m Sep 30 '15 at 10:24