How to load strings with "nil" into json objects in Python

Question

In this example here, the json module in Python can help load strings with the word "null" into a json object, but this doesn't work with data that has the word "nil" in it:

data = '''
    {
    "abc": nil,
    "def": 9
    }
'''
json.loads(data)

which gives the following error:

JSONDecodeError                           Traceback (most recent call last)
<ipython-input-38-3302f1bae848> in <module>
      5     }
      6 '''
----> 7 json.loads(data)

~/anaconda3/envs/prediction-routine/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    346             parse_int is None and parse_float is None and
    347             parse_constant is None and object_pairs_hook is None and not kw):
--> 348         return _default_decoder.decode(s)
    349     if cls is None:
    350         cls = JSONDecoder

~/anaconda3/envs/prediction-routine/lib/python3.7/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

~/anaconda3/envs/prediction-routine/lib/python3.7/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 3 column 12 (char 18)

What can be a solution? Thanks

`nil` isn't a valid JSON value; `null` is. What is generating the (invalid) JSON? — chepner, Aug 05 '20 at 15:23
`nil` is not a valid value for json, so one way is replacing all instances of `nil` as value with null and then passing the string to `json.loads`. — kaveh, Aug 05 '20 at 15:23
Why don't you use `json.loads(data.replace("nil", "null"))`? — Riccardo Bucco, Aug 05 '20 at 15:23
@RiccardoBucco That's very fragile, as you essentially have to parse the data in the first place to identify the (sub)strings that should actually be replaced. — chepner, Aug 05 '20 at 15:24
@RiccardoBucco you need to do a safer replace than that. `nil` might appear in other places within other keys or values too and you probably don't want to replace those. — kaveh, Aug 05 '20 at 15:24
@chepner True, but since the OP is dealing with a non standard json, the file has to be parsed anyway... Right? I see no easy solutions — Riccardo Bucco, Aug 05 '20 at 15:25
Thanks everyone for the responses! I'm getting the json data from a rails app, and I think I can ask people to replace nil with null. — George Liu, Aug 05 '20 at 15:28
@GeorgeLiu Just to be clear; you aren't getting JSON data. The Rails app is broken if it claims to be sending you JSON, because it is only supplying you with text that *looks* similar to JSON. — chepner, Aug 05 '20 at 15:29

score 1 · Answer 1 · answered Aug 05 '20 at 15:39

You are not dealing with valid JSON data. The only thing I can think of if modifying the Python json module itself.

For example, here you can find the implementation of the function you could edit (Python 3.8). This code (line 41-42)

elif nextchar == 'n' and string[idx:idx + 4] == 'null':
    return None, idx + 4

should become something like this

elif nextchar == 'n' and string[idx:idx + 4] == 'null':
    return None, idx + 4
elif nextchar == 'n' and string[idx:idx + 3] == 'nil':
    return None, idx + 3

I am not saying this is a recommended approach. However, since you are dealing with non valid JSON data, you would need to rewrite a parser from scratch (if you can't modify your data source to get valid JSON data). In this way, you can still use the Python official JSON parser by applying very little modification.

How to load strings with "nil" into json objects in Python

1 Answers1