urlopen.request with an umlaut in the url

Question

i want to scrape a website with a german umlaut in the url. Here is my code in python 3.3, that works very fine without any umlauts.

def numResults(keyword):
try:
    page_google = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=' +keyword
    print(page_google)
    req_google = Request(page_google)
    req_google.add_header('User Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20120427 Firefox/15.0a1')
    html_google = urlopen(req_google).read()
    soup = BeautifulSoup(html_google)
except URLError as e:
    print(e)
return soup

But when i request something like:

print(numResults('älterer'))

i get the following error because urllib cannot handle the umlaut i guess:

Traceback (most recent call last):
File "C:\Users\zwieback86\Desktop\programming\scrape.py", line 137, in <module>
print(numResults('älterer'))
File "C:\Users\zwieback86\Desktop\programming\scrape.py", line 73, in numResults
html_google = urlopen(req_google).read()
File "c:\python33\lib\urllib\request.py", line 156, in urlopen
return opener.open(url, data, timeout)
File "c:\python33\lib\urllib\request.py", line 469, in open
response = self._open(req, data)
File "c:\python33\lib\urllib\request.py", line 487, in _open
'_open', req)
File "c:\python33\lib\urllib\request.py", line 447, in _call_chain
result = func(*args)
File "c:\python33\lib\urllib\request.py", line 1268, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "c:\python33\lib\urllib\request.py", line 1248, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "c:\python33\lib\http\client.py", line 1061, in request
self._send_request(method, url, body, headers)
File "c:\python33\lib\http\client.py", line 1089, in _send_request
self.putrequest(method, url, **skips)
File "c:\python33\lib\http\client.py", line 953, in putrequest
self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 38: ordinal not in range(128)

When i type in the adress "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=älterer" into the browser i get the wanted page.

So i assume urllib cannot handle requests with umlauts in the url. But how can i fix it that it will accept the german umlauts? To change the umlauts like ä -> ae is not an option.

Many thanks and regards!

Answer 1

Use the requests module! It is superior to urllib2. http://docs.python-requests.org/en/latest/

>>> import requests
>>> r = requests.get('http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=' +'älterer')
>>> print r.text
{"responseData": {"results":[{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://de.wikipedia.org/wiki/Alter","url":"http://de.wikipedia.org/wiki/Alter","visibleUrl":"de.wikipedia.org","cacheUrl":"http://www.google.com/search?q\u003dcache:xN4gCMgmnZ0J:de.wikipedia.org","title":"\u003cb\u003eAlter\u003c/b\u003e – Wikipedia","titleNoFormatting":"Alter – Wikipedia","content":"Unter dem \u003cb\u003eAlter\u003c/b\u003e versteht man den Lebensabschnitt rund um die mittlere   Lebenserwartung des Menschen, also das Lebensalter zwischen dem mittleren \u003cb\u003e...\u003c/b\u003e"},{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://de.wikipedia.org/wiki/Blumfeld,_ein_%C3%A4lterer_Junggeselle","url":"http://de.wikipedia.org/wiki/Blumfeld,_ein_%25C3%25A4lterer_Junggeselle","visibleUrl":"de.wikipedia.org","cacheUrl":"http://www.google.com/search?q\u003dcache:VtRzZLhU-qkJ:de.wikipedia.org","title":"Blumfeld, ein \u003cb\u003eälterer\u003c/b\u003e Junggeselle – Wikipedia","titleNoFormatting":"Blumfeld, ein älterer Junggeselle – Wikipedia","content":"Blumfeld, ein \u003cb\u003eälterer\u003c/b\u003e Junggeselle ist eine Erzählung von Franz Kafka. Sie wurde   1915 verfasst und postum veröffentlicht. Sie behandelt die skurrilen \u003cb\u003e...\u003c/b\u003e"},{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://www.arbeitsagentur.de/nn_193018/Navigation/zentral/Buerger/Hilfen/Beschaeftigung-Aelterer/Beschaeftigung-Aelterer-Nav.html","url":"http://www.arbeitsagentur.de/nn_193018/Navigation/zentral/Buerger/Hilfen/Beschaeftigung-Aelterer/Beschaeftigung-Aelterer-Nav.html","visibleUrl":"www.arbeitsagentur.de","cacheUrl":"http://www.google.com/search?q\u003dcache:SXb9a3GIufkJ:www.arbeitsagentur.de","title":"Beschäftigung \u003cb\u003eÄlterer\u003c/b\u003e - www.arbeitsagentur.de","titleNoFormatting":"Beschäftigung Älterer - www.arbeitsagentur.de","content":"\u003cb\u003eÄltere\u003c/b\u003e Arbeitnehmer/-innen, die ihre Arbeitslosigkeit durch Aufnahme einer   geringer entlohnten versicherungspflichtigen Beschäftigung beenden oder \u003cb\u003e...\u003c/b\u003e"},{"GsearchResultClass":"GwebSearch","unescapedUrl":"http://www.imdb.com/title/tt0932839/","url":"http://www.imdb.com/title/tt0932839/","visibleUrl":"www.imdb.com","cacheUrl":"http://www.google.com/search?q\u003dcache:V-wfqhR1ABUJ:www.imdb.com","title":"\u0026quot;Monaco Franze - Der ewige Stenz\u0026quot; Ein ernsthafter \u003cb\u003eälterer\u003c/b\u003e Herr - IMDb","titleNoFormatting":"\u0026quot;Monaco Franze - Der ewige Stenz\u0026quot; Ein ernsthafter älterer Herr - IMDb","content":"Directed by Helmut Dietl. With Helmut Fischer, Ruth-Maria Kubitschek, Karl   Obermayr, Christine Kaufmann."}],"cursor":{"resultCount":"1,940,000","pages":[{"start":"0","label":1},{"start":"4","label":2},{"start":"8","label":3},{"start":"12","label":4},{"start":"16","label":5},{"start":"20","label":6},{"start":"24","label":7},{"start":"28","label":8}],"estimatedResultCount":"1940000","currentPageIndex":0,"moreResultsUrl":"http://www.google.com/search?oe\u003dutf8\u0026ie\u003dutf8\u0026source\u003duds\u0026start\u003d0\u0026hl\u003den\u0026q\u003d%C3%A4lterer","searchResultTime":"0.08"}}, "responseDetails": null, "responseStatus": 200}