Unable to parse Non ASCII URL on Python?

Question

I wanted to query Freebase API to get the list of teams José Mourinho has played for.

So, the URL i used on my browser is

https://www.googleapis.com/freebase/v1/mqlread?query=[{"name": "José Mourinho","/sports/pro_athlete/teams": [{"mid": null,"team": null,"to": null,"optional": true}]}]

However,

import json
import urllib

service_url="https://www.googleapis.com/freebase/v1/mqlread"
query = '[{"name": "' + "José Mourinho" + '","/sports/pro_athlete/teams": [{"mid": null,"team": null,"to": null,"optional": true}]}]'
url = service_url + '?' + 'query='+query
response = json.loads(urllib.urlopen(url).read())

Gives me an error saying,

UnicodeError: URL u'https://www.googleapis.com/freebase/v1/mqlread?query=[{"name": "Jos\xe9 Mourinho","/sports/pro_athlete/teams": [{"mid": null,"team": null,"to": null,"optional": true}]}]' contains non-ASCII characters

What is the solution to this?

[strictly speaking, urls cannot have non-ascii characters](http://stackoverflow.com/questions/4389572/how-to-fetch-a-non-ascii-url-with-python-urlopen) — R Nar, Nov 09 '15 at 18:55
I know that actually. But is there any hack/bypass for this in Python? — John Oliver, Nov 09 '15 at 18:56
@RNar I saw that before posting it. The solution gives me the error where I decode it twice. — John Oliver, Nov 09 '15 at 18:57
did you try using [urllib.parse.quote](http://stackoverflow.com/a/18269491/5323213)? — R Nar, Nov 09 '15 at 19:00
then use [`urllib.quote`](https://docs.python.org/2/library/urllib.html#urllib.quote) — R Nar, Nov 09 '15 at 19:09
Out of curiosity (and love of the library), what happens if you use [`requests`](http://docs.python-requests.org/en/latest/)? — jpmc26, Nov 09 '15 at 22:48

Kirk Strauser · Accepted Answer · 2015-11-09T19:16:13.523

I think you skipped over a little bit of the docs. Try this instead:

# coding=UTF-8

import json
import urllib

service_url = "https://www.googleapis.com/freebase/v1/mqlread"
query = [{
    '/sports/pro_athlete/teams': [
        {
            'to': None,
            'optional': True,
            'mid': None,
            'team': None
        }
    ],
    'name': 'José Mourinho'
}]

url = service_url + '?' + urllib.urlencode({'query': json.dumps(query)})
response = json.loads(urllib.urlopen(url).read())

print response

Rather than building the query string yourself, use json.dumps and urllib.urlencode to create it for you. They're good at this.

Note: if you can use the requests package, that last bit could be:

import requests
response = requests.get(service_url, params={'query': json.dumps(query)})

Then you get to skip the URL construction and escaping altogether!

Fantastic Solution. Bravo – John Oliver Nov 09 '15 at 19:13 — John Oliver, Nov 09 '15 at 19:13

Unable to parse Non ASCII URL on Python?

1 Answers1