urllib.parse.quote won't take utf8

Question

I am trying to use urllib.parse.quote as intended but cant get it to work. I even tried the example given in the documentation

Example: quote('/El Niño/') yields '/El%20Ni%C3%B1o/'.

If I try this following happens.

quote('/El Niño/')
  File "<stdin>", line 0

    ^
SyntaxError: 'utf-8' codec can't decode byte 0xf1 in position 13: invalid continuation byte

Anyone got a hint what is wrong? I am using Python 3.2.3

PS: Link to the docs http://docs.python.org/3.2/library/urllib.parse.html

What encoding was used for your source file? This error has nothing to do with `urllib`: Python is unable to parse your source file in the first place. — Thomas Orozco, Feb 16 '14 at 11:17
It happens when I just use the interpreter as well. So no source file — KrawallKurt, Feb 16 '14 at 11:19
Worked for me from the interpreter. What's probably wrong is that your interpreter is not sure about what encoding your console or source file are using. — Paulo Bu, Feb 16 '14 at 11:22
So how can I change that? Or if I called the function from a script how would it work? Btw for python 2 it works for me as well.. — KrawallKurt, Feb 16 '14 at 11:26

score 1 · Accepted Answer · answered Feb 16 '14 at 12:49

\xf1 is a latin-1 encoded ñ

>>> print(b'\xf1'.decode("latin-1"))
ñ

..not a utf-8 encoded character, like Python 3 assumes by default:

>>> print(b'\xf1'.decode("utf-8"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 0: unexpected end of data

Meaning, there is an encoding issue either with the .py file you have written, or the terminal in which you are running the Python shell - it is supplying latin-1 encoded data to Python, not utf-8

I got it. My Putty was set to Latin-1 and my files were ANSI. Changed all to UTF-8, now it works :) Thanks a lot — KrawallKurt, Feb 16 '14 at 16:09

score 0 · Answer 2 · edited May 23 '17 at 12:10

try adding the following line at the begining of your source code:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os, sys
…

per default, python assumes that your source code is written in ASCII, and thus it can't read unicode strings from your source file. Read PEP-0263 on the topic.

Though, if you switch to python3, you don't need to place that coding: utf-8 comment after the shebang line, because utf-8 is the default.

Edit: Just noticed that you are actually trying to do python3, which should be utf-8-safe. Though looking at the error, it looks to me that you're actually executing python2 code whereas you think you are executing python3.

Is the shebang line correctly set?

Are you calling the script with the right interpreter?

here's the right shebang line:

#/usr/bin/env python3

or

#/usr/bin/python3

and not just /usr/bin/python or /usr/bin/env python.

can you give your full failing script, and the way you're calling it in your question?

urllib.parse.quote won't take utf8

2 Answers2