Full precision numbers in python and JSON library

Question

Having a small issue with the code below with a recursive error:

100s errors printed ending in this:

RuntimeError: maximum recursion depth exceeded while calling a Python object

As you can see below my code is not recursive so something is happening with the DecimalEncoder.

Code

import json
import decimal      # tell json to leave my float values alone

class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            return decimal.Decimal(o)
            #return str(o)
        return super(DecimalEncoder, self).default(o)

class JSONUtils:
    def __init__( self, response ):
        self.response = response
        self.jsonData = None
        self.LoadData( )

        print 'jsonData: ' + json.dumps( self.jsonData, cls=DecimalEncoder, indent=2 )

    def LoadData ( self ):
        if ( self.jsonData == None ):
            if ( type( self.response ) == str or type( self.response ) == unicode ):
                #self.jsonData = json.loads(self.response )
                self.jsonData = json.loads(self.response, parse_float=decimal.Decimal )

    def GetJSONChunk( self, path ):
        returnValue = ''
        curPath     = ''
        try:
            if ( type( path ) == str ):
                returnValue = self.jsonData[path]
            elif (type( path ) == list):
                temp = ''
                firstTime = True
                for curPath in path:
                    if firstTime == True:
                        temp = self.jsonData[curPath]
                        firstTime = False
                    else:
                        temp = temp[curPath]
                returnValue = temp
            else:
                print 'Unknown type in GetJSONChunk: ' + unicode( type( path ))
        except KeyError as err:
            ti.DBG_OUT( 'JSON chunk doesn\'t have value: ' + unicode( path ))
            returnValue = self.kNoNode
        except IndexError as err:
            ti.DBG_OUT( 'Index does not exist: ' + unicode( curPath ))
            returnValue = self.kInvalidIndex

        return returnValue

myJSON = JSONUtils( unicode('{ "fldName":4.9497474683058327445566778899001122334455667788990011 }' ))
value =  str( myJSON.GetJSONChunk ( 'fldName' ))
print str( type( value ))
print value

If I swap return decimal.Decimal(0) for a string. It gets rid of the error BUT the value as you can see, is returned as a string.

#return decimal.Decimal(o)
return str(o)

This output is close but I need a double at the type:

jsonData: {
  "fldName": "4.9497474683058327445566778899001122334455667788990011"
}
<type 'str'>
4.9497474683058327445566778899001122334455667788990011

If I swap these lines you can see the original issue which looses precision.

#self.jsonData = json.loads(self.response )
self.jsonData = json.loads(self.response, parse_float=decimal.Decimal )

Answer 1

You are getting into this recursion error because the default method in DecimalEncoder is returning the same type it's supposed to handle, instead of turning it in a serializable type.

If you debug your code in a debugger, you'll be able to see that in a given moment, a internal call of JSONEncoder is trying to convert the value to one of the default serializable types, and since no one matches, it calls the fallback, which is the default method in DecimalEncoder class.

This is why the statement return str(o) works. It's returning a different type than Decimal which is a serializable one.

I recommend you to follow the user2357112's advice, and consider to encode the Decimal as a string, and convert it back on the other end.

The answer at https://stackoverflow.com/a/38357877/7254201 points out some reasons to use string instead of Decimal in JSON.

Update #1

Following below my suggestion about how to convert the decimal into a string so that you can get the original value later.

Use the DecimalEncoder to turn the decimal value in a new dict where there is a key called value a another called type, so you put both as string. Of course there will be some further work when deserializing the value.

class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            return {
                'value': str(o),
                'type': str(type(o).__name__) # you can use another code to provide the type.
            }
            #return str(o)
        return super(DecimalEncoder, self).default(o)

Following this strategy, the output of your code will looks like below:

jsonData: {
  "fldName": {
    "type": "decimal", 
    "value": "4.9497474683058327445566778899001122334455667788990011"
  }
}

As I see it, this problem is all about keeping the value and its type when serializing it. In those cases I try, when it's possible, to separate the data and its 'meta' information.

---

Update #2

How to decode it

Now, you have your object like:

  "fldName": {
    "type": "decimal", 
    "value": "4.9497474683058327445566778899001122334455667788990011"
  }

You may want to deserialize it as:

  {
    "fldName": Decimal('4.9497474683058327445566778899001122334455667788990011')
  }

So, you can do something based on this gist code: https://gist.github.com/simonw/7000493

I change the gist code a little bit to the below:

class DecimalDecoder(json.JSONDecoder):
    def __init__(self, *args, **kwargs):
        json.JSONDecoder.__init__(self, object_hook=self.object_hook, *args, **kwargs)

    def object_hook(self, obj):
        if 'type' not in obj:
            return obj

        obj_type = obj['type']
        if obj_type == 'decimal':
            return decimal.Decimal(obj.get('value'))
        return obj

My test code was:

    jsonData = json.dumps( self.jsonData, cls=DecimalEncoder, indent=2 )
    print 'jsonData: ' + jsonData
    newObj = json.loads(jsonData, cls=DecimalDecoder)
    print 'newObj: ' + str(newObj)

The output was:

jsonData: {
  "fldName": {
    "type": "decimal", 
    "value": "4.9497474683058327445566778899001122334455667788990011"
  }
}
newObj: {u'fldName': Decimal('4.9497474683058327445566778899001122334455667788990011')}