How can I parse (read) and use JSON in Python?

Question

My Python program receives JSON data, and I need to get bits of information out of it. How can I parse the data and use the result? I think I need to use json.loads for this task, but I can't understand how to do it.

For example, suppose that I have jsonStr = '{"one" : "1", "two" : "2", "three" : "3"}'. Given this JSON, and an input of "two", how can I get the corresponding data, "2"?

---

_{Beware that .load is for files; .loads is for strings. See also: Reading JSON from a file.}

_{Occasionally, a JSON document is intended to represent tabular data. If you have something like this and are trying to use it with Pandas, see Python - How to convert JSON File to Dataframe.}

_{Some data superficially looks like JSON, but is not JSON.}

_{For example, sometimes the data comes from applying repr to native Python data structures. The result may use quotes differently, use title-cased True and False rather than JSON-mandated true and false, etc. For such data, see Convert a String representation of a Dictionary to a dictionary or How to convert string representation of list to a list.}

_{Another common variant format puts separate valid JSON-formatted data on each line of the input. (Proper JSON cannot be parsed line by line, because it uses balanced brackets that can be many lines apart.) This format is called JSONL. See Loading JSONL file as JSON objects.}

_{Sometimes JSON data from a web source is padded with some extra text. In some contexts, this works around security restrictions in browsers. This is called JSONP and is described at What is JSONP, and why was it created?. In other contexts, the extra text implements a security measure, as described at Why does Google prepend while(1); to their JSON responses?. Either way, handling this in Python is straightforward: simply identify and remove the extra text, and proceed as before.}

Answer 1

Very simple:

import json
data = json.loads('{"one" : "1", "two" : "2", "three" : "3"}')
print(data['two'])  # or `print data['two']` in Python 2

Answer 2

Sometimes your json is not a string. For example if you are getting a json from a url like this:

j = urllib2.urlopen('http://site.com/data.json')

you will need to use json.load, not json.loads:

j_obj = json.load(j)

(it is easy to forget: the 's' is for 'string')

Answer 3

For URL or file, use json.load(). For string with .json content, use json.loads().

#! /usr/bin/python

import json
# from pprint import pprint

json_file = 'my_cube.json'
cube = '1'

with open(json_file) as json_data:
    data = json.load(json_data)

# pprint(data)

print "Dimension: ", data['cubes'][cube]['dim']
print "Measures:  ", data['cubes'][cube]['meas']

Answer 4

Following is simple example that may help you:

json_string = """
{
    "pk": 1, 
    "fa": "cc.ee", 
    "fb": {
        "fc": "", 
        "fd_id": "12345"
    }
}"""

import json
data = json.loads(json_string)
if data["fa"] == "cc.ee":
    data["fb"]["new_key"] = "cc.ee was present!"

print json.dumps(data)

The output for the above code will be:

{"pk": 1, "fb": {"new_key": "cc.ee was present!", "fd_id": "12345", 
 "fc": ""}, "fa": "cc.ee"}

Note that you can set the ident argument of dump to print it like so (for example,when using print json.dumps(data , indent=4)):

{
    "pk": 1, 
    "fb": {
        "new_key": "cc.ee was present!", 
        "fd_id": "12345", 
        "fc": ""
    }, 
    "fa": "cc.ee"
}

Answer 5

Parsing the data

Using the standard library json module

For string data, use json.loads:

import json

text = '{"one" : "1", "two" : "2", "three" : "3"}'
parsed = json.loads(example)

For data that comes from a file, or other file-like object, use json.load:

import io, json
# create an in-memory file-like object for demonstration purposes.
text = '{"one" : "1", "two" : "2", "three" : "3"}'
stream = io.StringIO(text)
parsed = json.load(stream) # load, not loads

It's easy to remember the distinction: the trailing s of loads stands for "string". (This is, admittedly, probably not in keeping with standard modern naming practice.)

Note that json.load does not accept a file path:

>>> json.load('example.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
AttributeError: 'str' object has no attribute 'read'

Both of these functions provide the same set of additional options for customizing the parsing process. Since 3.6, the options are keyword-only.

For string data, it is also possible to use the JSONDecoder class provided by the library, like so:

import json
text = '{"one" : "1", "two" : "2", "three" : "3"}'
decoder = json.JSONDecoder()
parsed = decoder.decode(text)

The same keyword parameters are available, but now they are passed to the constructor of the JSONDecoder, not the .decode method. The main advantage of the class is that it also provides a .raw_decode method, which will ignore extra data after the end of the JSON:

import json
text_with_junk = '{"one" : "1", "two" : "2", "three" : "3"} ignore this'
decoder = json.JSONDecoder()
# `amount` will count how many characters were parsed.
parsed, amount = decoder.raw_decode(text_with_junk)

Using requests or other implicit support

When data is retrieved from the Internet using the popular third-party requests library, it is not necessary to extract .text (or create any kind of file-like object) from the Response object and parse it separately. Instead, the Response object directly provides a .json method which will do this parsing:

import requests
response = requests.get('https://www.example.com')
parsed = response.json()

This method accepts the same keyword parameters as the standard library json functionality.

Using the results

Parsing by any of the above methods will result, by default, in a perfectly ordinary Python data structure, composed of the perfectly ordinary built-in types dict, list, str, int, float, bool (JSON true and false become Python constants True and False) and NoneType (JSON null becomes the Python constant None).

Working with this result, therefore, works the same way as if the same data had been obtained using any other technique.

Thus, to continue the example from the question:

>>> parsed
{'one': '1', 'two': '2', 'three': '3'}
>>> parsed['two']
'2'

I emphasize this because many people seem to expect that there is something special about the result; there is not. It's just a nested data structure, though dealing with nesting is sometimes difficult to understand.

Consider, for example, a parsed result like result = {'a': [{'b': 'c'}, {'d': 'e'}]}. To get 'e' requires following the appropriate steps one at a time: looking up the a key in the dict gives a list [{'b': 'c'}, {'d': 'e'}]; the second element of that list (index 1) is {'d': 'e'}; and looking up the 'd' key in there gives the 'e' value. Thus, the corresponding code is result['a'][1]['d']: each indexing step is applied in order.

See also How can I extract a single value from a nested data structure (such as from parsing JSON)?.

Sometimes people want to apply more complex selection criteria, iterate over nested lists, filter or transform the data, etc. These are more complex topics that will be dealt with elsewhere.

Common sources of confusion

JSON lookalikes

Before attempting to parse JSON data, it is important to ensure that the data actually is JSON. Check the JSON format specification to verify what is expected. Key points:

• The document represents one value (normally a JSON "object", which corresponds to a Python dict, but every other type represented by JSON is permissible). In particular, it does not have a separate entry on each line - that's JSONL.

• The data is human-readable after using a standard text encoding (normally UTF-8). Almost all of the text is contained within double quotes, and uses escape sequences where appropriate.

Dealing with embedded data

Consider an example file that contains:

{"one": "{\"two\": \"three\", \"backslash\": \"\\\\\"}"}

The backslashes here are for JSON's escape mechanism. When parsed with one of the above approaches, we get a result like:

>>> example = input()
{"one": "{\"two\": \"three\", \"backslash\": \"\\\\\"}"}
>>> parsed = json.loads(example)
>>> parsed
{'one': '{"two": "three", "backslash": "\\\\"}'}

Notice that parsed['one'] is a str, not a dict. As it happens, though, that string itself represents "embedded" JSON data.

To replace the embedded data with its parsed result, simply access the data, use the same parsing technique, and proceed from there (e.g. by updating the original result in place):

>>> parsed['one'] = json.loads(parsed['one'])
>>> parsed
{'one': {'two': 'three', 'backslash': '\\'}}

Note that the '\\' part here is the representation of a string containing one actual backslash, not two. This is following the usual Python rules for string escapes, which brings us to...

JSON escaping vs. Python string literal escaping

Sometimes people get confused when trying to test code that involves parsing JSON, and supply input as an incorrect string literal in the Python source code. This especially happens when trying to test code that needs to work with embedded JSON.

The issue is that the JSON format and the string literal format each have separate policies for escaping data. Python will process escapes in the string literal in order to create the string, which then still needs to contain escape sequences used by the JSON format.

In the above example, I used input at the interpreter prompt to show the example data, in order to avoid confusion with escaping. Here is one analogous example using a string literal in the source:

>>> json.loads('{"one": "{\\"two\\": \\"three\\", \\"backslash\\": \\"\\\\\\\\\\"}"}')
{'one': '{"two": "three", "backslash": "\\\\"}'}

To use a double-quoted string literal instead, double-quotes in the string literal also need to be escaped. Thus:

>>> json.loads('{\"one\": \"{\\\"two\\\": \\\"three\\\", \\\"backslash\\\": \\\"\\\\\\\\\\\"}\"}')
{'one': '{"two": "three", "backslash": "\\\\"}'}

Each sequence of \\\" in the input becomes \" in the actual JSON data, which becomes " (embedded within a string) when parsed by the JSON parser. Similarly, \\\\\\\\\\\" (five pairs of backslashes, then an escaped quote) becomes \\\\\" (five backslashes and a quote; equivalently, two pairs of backslashes, then an escaped quote) in the actual JSON data, which becomes \\" (two backslashes and a quote) when parsed by the JSON parser, which becomes \\\\" (two escaped backslashes and a quote) in the string representation of the parsed result (since now, the quote does not need escaping, as Python can use single quotes for the string; but the backslashes still do).

Simple customization

Aside from the strict option, the keyword options available for json.load and json.loads should be callbacks. The parser will call them, passing in portions of the data, and use whatever is returned to create the overall result.

The "parse" hooks are fairly self-explanatory. For example, we can specify to convert floating-point values to decimal.Decimal instances instead of using the native Python float:

>>> import decimal
>>> json.loads('123.4', parse_float=decimal.Decimal)
Decimal('123.4')

or use floats for every value, even if they could be converted to integer instead:

>>> json.loads('123', parse_int=float)
123.0

or refuse to convert JSON's representations of special floating-point values:

>>> def reject_special_floats(value):
...     raise ValueError
... 
>>> json.loads('Infinity')
inf
>>> json.loads('Infinity', parse_constant=reject_special_floats)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/json/__init__.py", line 370, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
  File "<stdin>", line 2, in reject_special_floats
ValueError

Customization example using object_hook and object_pairs_hook

object_hook and object_pairs_hook can be used to control what the parser does when given a JSON object, rather than creating a Python dict. A supplied object_pairs_hook will be called with one argument, which is a list of the key-value pairs that would otherwise be used for the dict. It should return the desired dict or other result:

>>> def process_object_pairs(items):
...     return {k: f'processed {v}' for k, v in items}
... 
>>> json.loads('{"one": 1, "two": 2}', object_pairs_hook=process_object_pairs)
{'one': 'processed 1', 'two': 'processed 2'}

A supplied object_hook will instead be called with the dict that would otherwise be created, and the result will substitute:

>>> def make_items_list(obj):
...     return list(obj.items())
... 
>>> json.loads('{"one": 1, "two": 2}', object_hook=make_items_list)
[('one', 1), ('two', 2)]

If both are supplied, the object_hook will be ignored and only the object_items_hook will be used.

Text encoding issues and bytes/unicode confusion

JSON is fundamentally a text format. Input data should be converted from raw bytes to text first, using an appropriate encoding, before the file is parsed.

In 3.x, loading from a bytes object is supported, and will implicitly use UTF-8 encoding:

>>> json.loads('"text"')
'text'
>>> json.loads(b'"text"')
'text'
>>> json.loads('"\xff"') # Unicode code point 255
'ÿ'
>>> json.loads(b'"\xff"') # Not valid UTF-8 encoded data!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/json/__init__.py", line 343, in loads
    s = s.decode(detect_encoding(s), 'surrogatepass')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 1: invalid start byte

UTF-8 is generally considered the default for JSON. While the original specification, ECMA-404 does not mandate an encoding (it only describes "JSON text", rather than JSON files or documents), RFC 8259 demands:

JSON text exchanged between systems that are not part of a closed ecosystem MUST be encoded using UTF-8 [RFC3629].

In such a "closed ecosystem" (i.e. for local documents that are encoded differently and will not be shared publicly), explicitly apply the appropriate encoding first:

>>> json.loads(b'"\xff"'.decode('iso-8859-1'))
'ÿ'

Similarly, JSON files should be opened in text mode, not binary mode. If the file uses a different encoding, simply specify that when opening it:

with open('example.json', encoding='iso-8859-1') as f:
    print(json.load(f))

In 2.x, strings and byte-sequences were not properly distinguished, which resulted in a lot of problems and confusion particularly when working with JSON.

Actively maintained 2.x codebases (please note that 2.x itself has not been maintained since Jan 1, 2020) should consistently use unicode values to represent text and str values to represent raw data (str is an alias for bytes in 2.x), and accept that the repr of unicode values will have a u prefix (after all, the code should be concerned with what the value actually is, not what it looks like at the REPL).

Historical note: simplejson

simplejson is simply the standard library json module, but maintained and developed externally. It was originally created before JSON support was added to the Python standard library. In 2.6, the simplejson project was incorporated into the standard library as json. Current development maintains compatibility back to 2.5, although there is also an unmaintained, legacy branch that should support as far back as 2.2.

The standard library generally uses quite old versions of the package; for example, my 3.8.10 installation reports

>>> json.__version__
'2.0.9'

whereas the most recent release (as of this writing) is 3.18.1. (The tagged releases in the Github repository only go as far back as 3.8.2; the 2.0.9 release dates to 2009.

I have as yet been unable to find comprehensive documentation of which simplejson versions correspond to which Python releases.