Python please don't muck with my number

Question

I'm writing a testing tool in python that downloads REST data, tosses the return value through json.loads() and then compares the value returned from the DB with an expected value. Unfortunately trying to print out that value or compare that value fails. Even though the pretty print of the JSON / Rest data is correct and has the full value. So something as simple as the example below prints lesser precision

Example:

print 1.414213562373095
1.41421356237

Note the reduced precision. Running an equal compare does not work either. In both cases I'm coercing the value to a string since comparing two numbers such as 1.13337 and 1.133333333333337 compare as the same number. Although technically correct we want to be sure that the output from the DB is at the promised precision. I would be grateful for any solutions out there. Thanks in advance.

Answer 1

First, you're not actually losing the precision you think you are in your example. print just truncates more aggressively than you expected on Python 2. Comparisons should work fine on that number, as long as you're not losing more precision somewhere else.

If you have an actual precision limits problem - for example, JSON with 20-digit numbers - you can address that. json.loads defaults to parsing numbers as floats, and floats have limited precision. If you don't want that, change how json.loads parses numbers:

>>> import json
>>> x = '{"a": 1.2345678901234567890}'
>>> json.loads(x, parse_float=str, parse_int=str, parse_constant=str)
{u'a': '1.2345678901234567890'}
>>> from decimal import Decimal
>>> json.loads(x, parse_float=Decimal, parse_int=Decimal, parse_constant=Decimal)
{u'a': Decimal('1.2345678901234567890')}

Answer 2

I would do as user2357112 suggested. Not enough info to tell exactly the procedure you're going through to compare with the DB, but for future reference, you could use format such as:

val = "{0:.15f}".format(1.414213562373095)
print val

Edit: Looks like Zinki beat me to it.

Answer 3

Python (and many other programming languages) inherently have problems representing decimal numbers as floats and floating point arithmetic (including comparing floating point numbers). Please see these pages for an extensive explanation as to why: Floating Point Arithmetic: Issues and Limitations, What Every Computer Scientist Should Know About Floating Point Arithmetic.

If you require high levels of precision in Python, using the Decimal class can help. From the Python docs: "The decimal module provides support for fast correctly-rounded decimal floating point arithmetic". Please see Decimal fixed point and floating point arithmetic for more detail.

Here is an example (from Python docs) showing the user-alterable level of precision:

>>> from decimal import *
>>> getcontext().prec = 6
>>> Decimal(1) / Decimal(7)
Decimal('0.142857')
>>> getcontext().prec = 28
>>> Decimal(1) / Decimal(7)
Decimal('0.1428571428571428571428571429')

EDIT: As per discussion in comments, I was looking further into floating point comparison and it is possible to use math.isclose in Python if you desire comparisons to a specific precision. See What is the best way to compare floats for almost-equality in Python? for more details. For an analysis of floating point comparisons, please see: Comparing Floating Point Numbers

Answer 4

This is complete coded up answer that will correctly print any arbitrarily large decimal number. Unfortunately, you must use the DecimalEncoder class to return the value as a string. When I run this code stand alone I get exactly what I want. (remember this is being used for testing and I want to be sure that python isn't changing the value somehow). So when I get the value back from the database I can compare the value correctly without python rounding or clipping the value.

This solution in my testing environment, for some reason, rounds the last digit but no longer clips to 11 digits of precision. Swapping the json.loads calls will show the original issue.

Unfortunately this changes the type of the data to a string and I still have to figure out why my code is rounding the value for the comparison but I can figure that out on the weekend :). Thanks for everybody's help!!

import json
import decimal  # use decimal to tell python to leave my numbers alone

class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            return str(o)
        return super(DecimalEncoder, self).default(o)

class JSONUtils:
    def __init__( self, response ):
        self.response = response
        self.jsonData = None
        self.LoadData( )

        print 'jsonData: ' + json.dumps( self.jsonData, cls=DecimalEncoder, indent=2 )

    def LoadData ( self ):
        if ( self.jsonData == None ):
            if ( type( self.response ) == str or type( self.response ) == unicode ):
#               self.jsonData = json.loads(self.response )
                self.jsonData = json.loads(self.response, parse_float=decimal.Decimal )

    def GetJSONChunk( self, path ):
        returnValue = ''
        curPath     = ''
        try:
            if ( type( path ) == str ):
                returnValue = self.jsonData[path]
            elif (type( path ) == list):
                temp = ''
                firstTime = True
                for curPath in path:
                    if firstTime == True:
                        temp = self.jsonData[curPath]
                        firstTime = False
                    else:
                        temp = temp[curPath]
                returnValue = temp
            else:
                print 'Unknown type in GetJSONChunk: ' + unicode( type( path ))
        except KeyError as err:
            ti.DBG_OUT( 'JSON chunk doesn\'t have value: ' + unicode( path ))
            returnValue = self.kNoNode
        except IndexError as err:
            ti.DBG_OUT( 'Index does not exist: ' + unicode( curPath ))
            returnValue = self.kInvalidIndex

        return returnValue

myJSON = JSONUtils( '{ "fldName":4.9497474683058327445566778899001122334455667788990011 }' )
value =  str( myJSON.GetJSONChunk ( 'fldName' ))
print str( type( value ))
print value

Output:

<type 'str'>
4.9497474683058327445566778899001122334455667788990011