2

I have a program that reads a number from an Access database. The number is stored in a table as a "single". Now the number is handled by my program and stored in an XML file. Later the program may read the XML file again.

Now my problem is in the database stands the number "3.001277". This is read from the database with help of COleSafeArray::GetElement and the debugger shows the number "3.001277". In hex the value 0x404014eD. When the number is written to the XML file it writes "3.001277" in ascii. But when the number is then read from the XML file the debugger shows it read 3.001277 but the hex value is =x404014eC. So the last hex digit differs with one.

How can I make my conversions so I get the same number?

Jens Mose Pedersen
  • 35
  • 5
  • The single data type can only represent 7 significant digits. Your actual byte value is `3.00127721`, but because it's represented as a single, the last two digits get stripped off. – Robby Cornelissen Jan 05 '18 at 09:13
  • Forgot to mention the conversion from XML is made with Convert.ChangeType where the input is the string with the number. – Jens Mose Pedersen Jan 05 '18 at 11:02
  • Yes I know 3.001277 can't be converted exact, but the question is why does the database conversion end up with 0x404014eD and the XML conversion with 0x404014eC. – Jens Mose Pedersen Jan 05 '18 at 11:05
  • 1
    I have solved the problem now by using string.Format("{0:G9}", value) instead of just Convert.ToString(value, CultureInfo.InvariantCulture) when I write a float to the XML file. This solves my problem! – Jens Mose Pedersen Jan 05 '18 at 12:31
  • If `GetElement` is converting the string “3.001277” to the floating-point value represented by 0x404014ed, then it is not returning the representable value nearest the input. 0x404014ed represents 3.0012772083282470703125 (.208… away relative to the final input digit). 0x404014ec represents 3.00127696990966796875 (.030… away). I expect the Microsoft software is not returning the best result. The debugger may be showing “3.001277” because it defaults to six digits after the decimal point or seven significant digits. If it has formatting controls to show more, it may show you 3.0012772… – Eric Postpischil Jan 05 '18 at 15:00
  • @JensMosePedersen Similar issue in C: [width specifier to maintain precision of floating-point value](https://stackoverflow.com/q/16839658/2410359) A "Single", in general, needs to be printed with 9 significant decimal digits to round-tripped correctly. – chux - Reinstate Monica Jan 18 '18 at 16:59

0 Answers0