Replacing non-ascii characters in an ascii encoded string

Question

I have this code fragment (Python 2.7):

from bs4 import BeautifulSoup

content = '  foo bar';
soup = BeautifulSoup(content, 'html.parser')
w = soup.get_text()

At this point w has a byte with value 160 in it, but it's encoding is ASCII.

How do replace all of the \xa0 bytes by another character?

I've tried:

w = w.replace(chr(160), ' ')
w = w.replace('\xa0', ' ')

but I am getting the error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 0: ordinal not in range(128)

And why does BS return an ASCII encoded string with an invalid character in it?

Is there a way to convert w to a 'latin1` encoded string?

Answer 1

At this point w has a byte with value 160 in it, but it's encoding is 'ascii'.

You have an unicode string:

>>> w
u'\xa0 foo bar'
>>> type(w)
<type 'unicode'>

How do replace all of the \xa0 bytes with another character?

>>> x = w.replace(u'\xa0', ' ')
>>> x
u'  foo bar'

And why does BS return an 'ascii' encoded string with an invalid character in it?

As mentioned above, it is not an ascii encoded string, but an Unicode string instance.

Is there a way to convert w to a 'latin1` encoded string?

Sure:

>>> w.encode('latin1')
'\xa0 foo bar'

(Note this last string is an encoded string, not an unicode object, and its representation is not prefixed by 'u' like the previous unicode objects).

Notes (edited):

• If you are typing strings into your source files, note that encoding of source files matters. Python will assume your source files are ASCII. The command line interpreter, on the other hand, will assume you are entering strings in your default system encoding. Of course you can override all this.
• Avoid latin1, use UTF-8 if possible: ie. w.encode('utf8')
• When encoding and decoding can tell Python to ignore errors, or replace characters that cannot be encoded with some marker character . I don't recommend to ignore encoding errors (at least without logging them), except for the hopefully rare cases when you know there are encoding errors or you need to encode text into a more reduced character set, requiring replacement of the code points that cannot be represented (ie if you need to encode 'España' into ASCII, you definitely should replace the 'ñ'). But for these cases there are imho better alternatives and you should look into the magical unicodedata module (see https://stackoverflow.com/a/1207479/401656).
• There is a Python Unicode HOWTO: https://docs.python.org/2/howto/unicode.html