how to url-safe encode a string with python? and urllib.quote is wrong

Question

Hello i was wondering if you know any other way to encode a string to a url-safe, because urllib.quote is doing it wrong, the output is different than expected:

If i try

urllib.quote('á')

i get

'%C3%A1'

But thats not the correct output, it should be %E1

As demostrated by the tool provided here this site

And this is not me being difficult, the incorrect output of quote is preventing the browser to found resources, if i try

urllib.quote('\images\á\some file.jpg')

And then i try with the javascript tool i mentioned i get this strings respectively

%5Cimages%5C%C3%A1%5Csome%20file.jpg

%5Cimages%5C%E1%5Csome%20file.jpg

Note how is almost the same but the url provided by quote doesn't work and the other one it does. I tried messing with encode('utf-8) on the string provided to quote but it does not make a difference. I tried with other spanish words with accents and the ñ they all are differently represented.

Is this a python bug? Do you know some module that get this right?

both javascript and python are using the same encoding? Have you tried unicode? `repr('á') == "'\\xc3\\xa1'"` and `repr(u'á') == "u'\\xe1'"` — JBernardo, Jun 14 '11 at 02:34
Related: http://stackoverflow.com/questions/912811/what-is-the-proper-way-to-url-encode-unicode-characters — icktoofay, Jun 14 '11 at 02:38
`0xc3a1` is a [UTF-8 representation of _LATIN SMALL LETTER A WITH ACUTE_](http://www.utf8-chartable.de/unicode-utf8-table.pl). — sarnold, Jun 14 '11 at 02:38
@sarnold oh that helps, now i know i want my urls in unicode not in utf-8, but doing unicode(urllib.quote(string)) is not working. — Guillermo Siliceo Trueba, Jun 14 '11 at 03:51

score 7 · Answer 1 · edited Oct 07 '21 at 05:51

7

According to RFC 3986, %C3%A1 is correct. Characters are supposed to be converted to an octet stream using UTF-8 before the octet stream is percent-encoded. The site you link is out of date.

See Why does the encoding's of a URL and the query string part differ? for more detail on the history of handling non-ASCII characters in URLs.

edited Oct 07 '21 at 05:51

Community

1
1

answered Jun 14 '11 at 02:38

Anomie

92,546
13
126
145

could be, but none of my browsers find the resourse with the "updated" encoding. – Guillermo Siliceo Trueba Jun 14 '11 at 03:20
So, any module that uses the outdated but actually working encoding? – Guillermo Siliceo Trueba Jun 14 '11 at 03:42
2

@Guillermo, can you update your _server_ to allow newer HTTP clients to request resources as specified in the newer RFC? – sarnold Jun 14 '11 at 09:01
I'm using web.py's internal server can't do anything about that for the time being. – Guillermo Siliceo Trueba Jun 14 '11 at 18:15

score 3 · Answer 2 · answered Jun 14 '11 at 04:40

3

Ok, got it, i have to encode to iso-8859-1 like this

word = u'á'
word = word.encode('iso-8859-1')
print word

answered Jun 14 '11 at 04:40

Guillermo Siliceo Trueba

4,251
5
34
47

score 1 · Answer 3 · answered Jun 14 '11 at 02:40

Python is interpreted in ASCII by default, so even though your file may be encoded differently, your UTF-8 char is interpereted as two ASCII chars.

Try putting a comment as the first of second line of your code like this to match the file encoding, and you might need to use u'á' also.

# coding: utf-8

score 0 · Answer 4 · answered Jun 14 '11 at 02:42

0

What about using unicode strings and the numeric representation (ord) of the char?

>>> print '%{0:X}'.format(ord(u'á'))
%E1

answered Jun 14 '11 at 02:42

It's a hack, but a hack may be required for a website that still requires ISO-8859-1. Most webservers are now compliant with UTF-8, as assumed by urllib. – krubo Jun 14 '11 at 03:22
1

Works but looks like black magic, and it doesnt work with more than 1 character, and looping over all my content just doesnt seem like a good idea. – Guillermo Siliceo Trueba Jun 14 '11 at 03:43

score 0 · Answer 5 · edited May 23 '17 at 12:26

0

In this question it seems some guy wrote a pretty large function to convert to ascii urls, thats what i need. But i was hoping there was some encoding tool in the std lib for the job.

edited May 23 '17 at 12:26

Community

1
1

answered Jun 14 '11 at 03:57

Guillermo Siliceo Trueba

4,251
5
34
47

I spoke too soon, those functions do not output the Unicode code point that is needed. – Guillermo Siliceo Trueba Jun 14 '11 at 04:29

how to url-safe encode a string with python? and urllib.quote is wrong

5 Answers5

Linked