json() on "requests" response raises UnicodeEncodeError

Question

I'm querying Github's Jobs API with python3, using the requests library, but running into an error parsing the response.

Library: http://docs.python-requests.org/en/latest/

Code:

import requests
import json

url = 'https://jobs.github.com/positions.json?' 

response = requests.get(url)

print(response.json())

Error:

UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 321: ordinal not in range(128)

Using this API in the past with Ruby, I've never run into this issue.

I also tried converting it to a dictionary but it resulted in the same errors.

There's other questions on SO about the UnicodeEncodeError (mostly re: opening files), but I'm not familiar with Python and didn't find them helpful.

Answer 1

First, check that the response is actually JSON. Try printing response.text and see if it looks like a valid JSON object.

Assuming it is JSON: it's very "hack"-ey, but you can replace the non ASCII characters with their escaped Unicode representation:

def escape_unicode(c):
    return c.encode('ascii', 'backslashreplace').decode('ascii')

response = ...

text = response.text
escaped = re.sub(r'[^\x00-\x7F]', lambda m: escape_unicode(m.group(0)), text)
json_response = json.loads(escaped)