To confirm the log and your suspicion, this is an encoding issue (bytes vs string most likely). Even if you set the encoding of your file to UTF-8 with # -*- coding: utf-8 -*-
, you can still have issues working with text which has been changed from one form to another.
Strings are not really strings, but representations of bytes in a particular order. UTF-8 provides for the encoding of many more characters than ASCII can handle, so if you try to convert a character which exists in a UTF-8 encoded string to an ASCII encoded string, then you will get an error because no such encoding exists.
I cannot give a much better answer without more information, such as code and / or data source.
Reading https://docs.python.org/2/howto/unicode.html#the-unicode-type we learn by studying the following example(s):
>>> unicode('abcdef')
u'abcdef'
>>> s = unicode('abcdef')
>>> type(s)
<type 'unicode'>
>>> unicode('abcdef' + chr(255))
Traceback (most recent call last):
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 6:
ordinal not in range(128)
The documentation also mentions that you can choose to handle these exceptions by either replacing or ignoring them, like so:
>>> unicode('\x80abc', errors='strict')
Traceback (most recent call last):
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
ordinal not in range(128)
>>> unicode('\x80abc', errors='replace')
u'\ufffdabc'
>>> unicode('\x80abc', errors='ignore')
u'abc'
NOTE 1: In Python 3 things have changed. for writing code which is compatible with Python 3, I recommend the following read:
https://docs.python.org/3.0/whatsnew/3.0.html#text-vs-data-instead-of-unicode-vs-8-bit
NOTE 2: It is also worth noting that if your encoding issue is encountered while trying to display the string on the console then, python has a -u
switch which can be used in certain situations, such as when you are serving a binary file through a CGI script, which will turn off buffering of strings, but that opens up another can of worms. But, none the less, mimicking this behavior without invoking -u
:
>>> print 'test'
'test'
>>> import os
>>> import sys
>>> sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
>>> print 'test'
test