2

I need to parse a HTML-page with windows-1251 charset (it's in russian).

The problem is that it is the web application and I have to use Python 2.4 without any opportunity to install modules on server. The only thing I tried to do was asking an administrator to install lxml module but nevertheless it wasn't built in the right way on 2.4 and attempt to import lxml.html fails.

Now I'm trying to select between BeautifulSoup and html5lib modules, but I didn't find any simple examples of using html5lib (I need just to extract some text from the certain div element with stripping all the other tags inside). In turn, BeautifulSoup returns an error 'junk characters in start tag: u'\u041f\u0440\u043e\u0434\u0430\u0436\u0430>' and any attempts to decode the source page from CP1251 to unicode or any other charset didn't make it.

What am i doing wrong? Or what parser I should use?

kennytm
  • 510,854
  • 105
  • 1,084
  • 1,005
Enchantner
  • 1,534
  • 3
  • 20
  • 40
  • What release of BeautifulSoup are you using? See http://www.crummy.com/software/BeautifulSoup/3.1-problems.html , avoid `3.1.*` (unless you're using Python 3) and stick with `3.0.x` (for `x >= 8`). – Alex Martelli Jun 30 '10 at 14:39
  • I'm using 3.0.8.1 because of Python 2.4. It's the main problem that Python version is so old. – Enchantner Jun 30 '10 at 14:46
  • 1
    Your error message looks like you already have Unicode objects. Are you sure you are really reading binary string data encoded in CP1251 and that it's not getting converted to unicode somewhere in the stack? – Peter Lyons Jun 30 '10 at 20:52
  • AFAIK BeautifulSoup can use `chardet` which can autodetect cp1251 quite happily even if the encoding is not supplied in the HTML. In any case as Peter Lyons as pointed out, `\u04xx` looks like Cyrillic already in unicode to me. – John Machin Aug 08 '10 at 09:00

1 Answers1

0

This question has a great answer on how to convert to UTF-8:

Python: how to convert from Windows 1251 to Unicode?

I love the BeautifulSoup library but I'm not familiar with odd character sets. If I were writing this I'd probably just try to sanitize the input to get me into unicode land where most libraries are able to play nicely.

Community
  • 1
  • 1
Chet
  • 21,375
  • 10
  • 40
  • 58