19

I'm trying to decode HTML entries from here NYTimes.com and I cannot figure out what I am doing wrong.

Take for example: 

"U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’"

I've tried BeautifulSoup, decode('iso-8859-1'), and django.utils.encoding's smart_str without any success.

Kenan Banks
  • 207,056
  • 34
  • 155
  • 173
KeyboardInterrupt
  • 3,483
  • 8
  • 36
  • 41

4 Answers4

22
>>> from HTMLParser import HTMLParser
>>> print HTMLParser().unescape('U.S. Adviser’s Blunt Memo on Iraq: '
...                             'Time ‘to Go Home’')
U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’

The function is undocumented in Python 2. It is fixed in Python 3.4+: it is exposed as html.unescape() there.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • 4
    For future users, this answer appears to have so few upvotes simply because it came 4 years later than the existing answers. It seems to be at least as good an answer. This answer has the advantage that it is simple (unlike writing your own function to to interpret HTML standards using a regex) and uses a standard library (unlike BeautifulSoup). It has the disadvantage that is is using an undocumented function. – Daniel Koverman Oct 07 '14 at 19:03
20

Actually what you have are not HTML entities. There are THREE varieties of those &.....; thingies -- for example       all mean U+00A0 NO-BREAK SPACE.

  (the type you have) is a "numeric character reference" (decimal).
  is a "numeric character reference" (hexadecimal).
  is an entity.

Further reading: http://htmlhelp.com/reference/html40/entities/

Here you will find code for Python2.x that does all three in one scan through the input: http://effbot.org/zone/re-sub.htm#unescape-html

Florian
  • 2,562
  • 5
  • 25
  • 35
John Machin
  • 81,303
  • 11
  • 141
  • 189
18

This does work:

from BeautifulSoup import BeautifulStoneSoup
s = "U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’"
decoded = BeautifulStoneSoup(s, convertEntities=BeautifulStoneSoup.HTML_ENTITIES)

If you want a string instead of a Unicode object, you'll need to decode it to an encoding that supports the characters being used; ISO-8859-1 doesn't:

result = decoded.encode("UTF-8")

It's unfortunate that you need an external module for something like this; simple HTML/XML entity decoding should be in the standard library, and not require me to use a library with meaningless class names like "BeautifulStoneSoup". (Class and function names should not be "creative", they should be meaningful.)

Glenn Maynard
  • 55,829
  • 10
  • 121
  • 131
  • 2
    lxml, alas also not in the standard library, also provides a Beautiful Soup parser (and lots more) with somewhat less "creative" names. – Ned Deily Jul 30 '09 at 21:24
  • 1
    Support for entity decoding is in the standard library (module htmlentitydefs). What the OP has are (decimal) numeric character references, not entities. – John Machin Jul 30 '09 at 23:24
  • Works as well with BeautifulSoup instead of BeautifulStoneSoup - one step less "creative" :) – Beni Cherniavsky-Paskin Mar 27 '12 at 16:29
  • ' names should not be "creative" ' is that a stone cold rule, or just personal choice? – TankorSmash Jul 20 '12 at 18:51
  • @TankorSmash: There's no authority--beyond the compiler--forcing you to follow any coding standards at all, but this seems like common sense to me. – Glenn Maynard Jul 20 '12 at 20:52
6

Try this:

import re

def _callback(matches):
    id = matches.group(1)
    try:
        return unichr(int(id))
    except:
        return id

def decode_unicode_references(data):
    return re.sub("&#(\d+)(;|(?=\s))", _callback, data)

data = "U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’"
print decode_unicode_references(data)
Evan Fosmark
  • 98,895
  • 36
  • 105
  • 117
  • UnicodeEncodeError: 'charmap' codec can't encode character u'\u2019' in position 12: character maps to This seems to be the error I keep getting regardless of what I try. – KeyboardInterrupt Jul 30 '09 at 20:05
  • Could you provide more code, then? I just tried it with the function I wrote and the character 2019 works fine. It shows up as: ߣ – Evan Fosmark Jul 30 '09 at 20:08
  • A few questions on your regexp: (1) Shouldn't it be \d instead of \w? The regexp will match ` ` and ` ` but then it will crash in int() (2) Allowing the character reference (it's NOT an entity) to end in a whitespace instead of ';' seems very tolerant -- shouldn't you mention this? (3) Wouldn't the last part be better written as [;\s]? – John Machin Jul 30 '09 at 23:55
  • John, you were correct on point one *partially*. It won't match   since that doesn't start with ``, but yes it should have been `\d`. Regarding point two to allowing it to end with whitespace, it should be noted that even though it isn't pretty, it's still supported. I've updated the code in the following way: (1) Changed it to `\d`, (2) made the callback a bit stronger, and (3) used a lookahead assertion for ending whitespace instead of absorbing it like it was. – Evan Fosmark Jul 31 '09 at 06:00
  • Evan, thanks for the enlightenment, especially about the tolerance of whitespace, which I didn't know about. I got some more clues by looking in the HTML 4.01 and 2.0 specs. They referred to the SGML standard (ISO 8879). Cost = CHF 238(!) so I didn't read it, but HTML 2.0 commented that ';' is only needed when the character following the reference would otherwise be part of the name. Experiments with FF, IE and Opera using space - / X A and `&` instead of ; all gave the same result: they terminate the reference and are not swallowed. I'm looking forward to your updated solution ;-) – John Machin Aug 01 '09 at 07:24