Python Unicode hell: Decode and Encode not working

Question

I am trying to grab an article from the web and write it to the database.

If I do this

article = article.decode('utf-8')

I get this:

'ascii' codec can't decode byte 0xc3 in position 25729: ordinal not in range(128)

If I do this

article = article.encode('utf-8')

I get this:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 5409: ordinal not in range(128)

If I do this

article = article.encode('utf-8').decode()

or this

article = article.decode().encode('utf-8')

I still get this

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 5409: ordinal not in range(128)

Questions:

Any help will be greatly appreciated on solving this problem!

EDIT: Stackoverflow recommended an article that said do .encode('utf-8') as per the above, this doesn't work, the error persists.

Answer 1

Unicode is not a pain if you know what you do.

If we try a more systematic approach, and suppose we stay with Python 2.x, we have to understand that everything we get from the web etc. consists of bytes and thus is a str.

On a str, we can only call .decode(), on a unicode object, we can only call .encode(). (This is not completely true, but if we don't follow this, we lose control over the internal de-/encoding which happens to compensate for this mismatch.)

Example: If you do

article = article.encode('utf-8')

you get a UnicodeDecodeError which says that 'ascii' codec can't decode byte 0xc4 in position 5409: ordinal not in range(128)

We see that although we call .encode(), a decode error happens first. This is because there is an implicit call to .decode('ascii') which fails because there are non-ASCII bytes in the str.

However, I don't understand why

article = article.decode('utf-8')

gives a

'ascii' codec can't decode byte 0xc3 in position 25729: ordinal not in range(128)

because the ascii codec isn't used at all here. Maybe you could edit your question and add the output of print repr(article) before this .decode() call so that we can try to replicate this.

Answer 2

Reading between the lines of the question and the OP's own answer, it looks like the encoding of the original web page is not being handled.

The web page needs to be correctly decoded. This can be achieved by inspecting the Content-type: header or use a HTTP library which does it for you. Requests module does this for you and returns a decoded Unicode object. This object can then be passed to TextWrappers (via io.open())for writing to a file, to a database handler, or to BeautifulSoup for parsing. In fact, BeautifulSoup should only be passed Unicode strings.

Example using Requests:

response = requests.get(url)

# A decoded Unicode object
response_body_unicode = response.text

soup = BeautifulSoup(response_body_unicode)

Answer 3

It looks like the workaround is to add this to the code

html = unicode(html, errors='ignore')

so the full code to get the article looks like this

def getArticle(url):
    br = mechanize.Browser()
    br.set_handle_robots(False)
    br.addheaders = [("User-agent","Mozilla/5.0")] #our identity in the web
    html = br.open(url).read()

    html = unicode(html, errors='ignore')

    readable_article = Document(html).summary()
    readable_title = Document(html).short_title()

    soup = BeautifulSoup(readable_article)

    soup_title = BeautifulSoup(readable_title)

    final_article = soup.text
    final_title = soup_title.text

    links = soup.findAll('img', src=True)

    return html, final_article, final_title, links

Answer 4

Try using utf 16 instead. This tends to help solve my problems when the above occurs.

     .decode('utf-16')