Simple ascii url encoding with python

Question

look at that:

import urllib
print urllib.urlencode(dict(bla='Ã'))

the output is

bla=%C3%BC

what I want is simple, I want the output in ascii instead of utf-8, so I need the output:

bla=%C3

if I try:

urllib.urlencode(dict(bla='Ã'.decode('iso-8859-1')))

doesn't work (all my python files are utf-8 encoded):

'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

In production, the input comes unicoded.

It might be worth rephrasing the title. ASCII does not include an 'Ã' character. — ʇsәɹoɈ, Jun 24 '10 at 22:01
"bla=%C3%BC" contains no non-ASCII characters. You need to explain what you really want/need and why you think that you need it. — John Machin, Jun 24 '10 at 22:14
@mykhal: U+00C3 is LATIN CAPITAL LETTER A WITH TILDE. "\xC3" is mapped to U+00C3 in ISO-8859-1 and cp1252. What are you trying to say? — John Machin, Jun 24 '10 at 22:24

score 4 · Answer 1 · answered Jun 24 '10 at 22:08

Have a look at unicode transliteration in python:

from unidecode import unidecode
print unidecode(u"\u5317\u4EB0")

# That prints: Bei Jing

In your case:

bla='Ã'
print unidecode(bla)
'A'

This is a third party library, which can be easily installed via:

$ git clone http://code.zemanta.com/tsolc/git/unidecode
$ cd unidecode
$ python setup.py install

score 2 · Answer 2 · answered Jun 24 '10 at 22:04

I want the output in ascii instead of utf-8

That's not ASCII, which has no characters mapped above 0x80. You're talking about ISO-8859-1, or possibly code page 1252 (the Windows encoding based on it).

'Ã'.decode('iso-8859-1')

Well that depends on what encoding you've used to save the character Ã in the source, doesn't it? It sounds like your text editor has saved it as UTF-8. (That's a good thing, because locale-specific encodings like ISO-8859-1 need to go away ASAP.)

Tell Python that the source file you've saved is in UTF-8 as per PEP 263:

# coding=utf-8

urllib.quote(u'Ã'.encode('iso-8859-1'))    # -> %C3

Or, if you don't want that hassle, use a backslash escape:

urllib.quote(u'\u00C3'.encode('iso-8859-1'))    # -> %C3

Although, either way, a modern webapp should be using UTF-8 for its input rather than ISO-8859-1/cp1252.

the entire webapp is already utf-8, but the external url-based webservice that I am trying to communicate doesn't recognize %C3%BC, only %C3. your solution works fine. — bluefoot, Jun 25 '10 at 17:49

score 2 · Answer 3 · answered Jun 24 '10 at 22:04

2

pretty well working asciification is this way:

import unicodedata
unicodedata.normalize('NFKD', 'Ã'.decode('UTF-8')).encode('ascii', 'ignore')

answered Jun 24 '10 at 22:04

mykhal

19,175
11
72
80

score 1 · Answer 4 · answered Jun 24 '10 at 22:02

1

If your input is actually UTF-8 and you want iso-8859-1 as output (which is not ASCII) what you need is:

'ñ'.decode('utf-8').encode('iso-8859-1')

answered Jun 24 '10 at 22:02

Vinko Vrsalovic

330,807
53
334
373

score 0 · Answer 5 · answered Jun 04 '15 at 17:05

Package unihandecode is

US-ASCII transliterations of Unicode text.
an improved version of Python unidecode, that is Python port of Text::Unidecode Perl module by Sean M. Burke .

pip install Unihandecode

then in python

import unihandecode
print(unihandecode.unidecode(u'Ã'))

prints A.

score 0 · Accepted Answer · answered Jun 25 '10 at 17:51

0

thanks to all solutions. all of you converge to the very same point. I made a mess changing the right code

.encode('iso-8859-1')

to

.decode('iso-8859-1')

turn back to .encode('iso-8859-1') and it works.

answered Jun 25 '10 at 17:51

bluefoot

10,220
11
43
56

Simple ascii url encoding with python

6 Answers6

Linked