What's the most Pythonic way to parse out this value from a JSON-like blob?

Question

See below. Given a well-known Google URL, I'm trying to retrieve data from that URL. That data will provide me another Google URL from which I can retrieve a list of JWKs.

>>> import requests, json
>>> open_id_config_url = 'https://ggp.sandbox.google.com/.well-known/openid-configuration'
>>> response = requests.get(open_id_config_url)
>>> r.status_code
200

>>> response.text
u'{\n "issuer": "https://www.stadia.com",\n "jwks_uri": "https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt@system.gserviceaccount.com",\n "claims_supported": [\n  "iss",\n  "aud",\n  "sub",\n  "iat",\n  "exp",\n  "s_env",\n  "s_app_id",\n  "s_gamer_tag",\n  "s_purchase_country",\n  "s_current_country",\n  "s_session_id",\n  "s_instance_ip",\n  "s_restrict_text_chat",\n  "s_restrict_voice_chat",\n  "s_restrict_multiplayer",\n  "s_restrict_stream_connect",\n ],\n "id_token_signing_alg_values_supported": [\n  "RS256"\n ],\n}'

Above I have successfully retrieved the data from the first URL. I can see the entry jwks_uri contains the second URL I need. But when I try to convert that blob of text to a python dictionary, it fails.

>>> response.json()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/saqib.ali/saqib-env-99/lib/python2.7/site-packages/requests/models.py", line 889, in json
    self.content.decode(encoding), **kwargs
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

>>> json.loads(response.text)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

The only way I can get out the JWKs URL is by doing this ugly regular expression parsing:

>>> re.compile('(?<="jwks_uri": ")[^"]+').findall(response.text)[0]
u'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt@system.gserviceaccount.com'

Is there a cleaner, more Pythonic way to extract this string? I really wish Google would send back a string that could be cleanly JSON-ified.

score 4 · Accepted Answer · answered Jun 30 '19 at 20:56

The returned json string is incorrect because last item of the dictionary ends with ,, which json cannot parse.

": [\n  "RS256"\n ],\n}'
                  ^^^

But ast.literal_eval can do that (as python parsing accepts lists/dicts that end with a comma). As long as you don't have booleans or null values, it is possible and pythonic

>>> ast.literal_eval(response.text)["jwks_uri"]
'https://www.googleapis.com/service_accounts/v1/jwk/stadia-jwt@system.gserviceaccount.com'

score 2 · Answer 2 · answered Jun 30 '19 at 20:57

Your JSON is invalid because it has an extra comma after the last value in the claims_supported array.

I wouldn't necessarily recommend it, but you could use the similarity of JSON and Python syntax to parse this directly, since Python is much less picky:

ast.literal_eval(response.tezt)

score 1 · Answer 3 · answered Jun 30 '19 at 22:26

1

As suggested in this answer use yaml to parse json. It will tolerate the trailing comma as well as other deviations from the json standard.

import yaml

d = yaml.load(response.text)

answered Jun 30 '19 at 22:26

Markus Rother

414
3
10

What's the most Pythonic way to parse out this value from a JSON-like blob?

3 Answers3