how to open unicode encoded URL?

Question

This code works as expected:

from urllib.request import urlopen 
with urlopen('https://mr.wikipedia.org/s/4jp4') as f:
    f.read().decode('utf-8')

But similar code returns an error. Both the URL's point to the same wiki article.

from urllib.request import urlopen 
with urlopen('https://mr.wikipedia.org/wiki/किशोरावस्था') as f:
    f.read().decode('utf-8')

UnicodeEncodeError: 'ascii' codec can't encode characters in position 10-20: ordinal not in range(128)

I need to use python built-in modules and therefore can not use requests module.

This works. But in my case the URL is part of an API and I do not know which word to quote. Is there more general solution like requests?

from urllib.parse   import quote
from urllib.request import urlopen

url = 'https://mr.wikipedia.org/wiki/' + quote("किशोरावस्था")
content = urlopen(url).read()

I tried `res = requests.get('https://mr.wikipedia.org/wiki/किशोरावस्था') res.content.decode('utf8')` — Joshua Varghese, Apr 12 '20 at 15:58
The URL is not strictly speaking Unicode-encoded, because Unicode is not an encoding. The page (or the response) is "utf-8" encoded (or should be). — Laurent LAPORTE, Apr 12 '20 at 16:09
Provide the full traceback. It gives us more useful information than you think. — Jongware, Apr 12 '20 at 16:09
There is also an open issue for this - https://bugs.python.org/issue3991 — urban, Apr 12 '20 at 16:18
This edit " But in my case the URL is part of an API and I do not know which word to quote. Is there more general solution like requests" should be a new question I think. — snakecharmerb, Apr 12 '20 at 16:25

score 2 · Accepted Answer · answered Apr 12 '20 at 16:13

2

The url is the error creator here. Try:

from urllib.request import urlopen 
with urlopen('https://mr.wikipedia.org/wiki/'+urllib.parse.quote('किशोरावस्था')) as f:
    f.read().decode('utf-8')

answered Apr 12 '20 at 16:13

Joshua Varghese

5,082
1
13
34

lol ok the question closed with the right answer. I found this answer before that. Sorry if this was late – Joshua Varghese Apr 12 '20 at 16:15

how to open unicode encoded URL?

1 Answers1