0

Looking to parse an HTML page for images (from http://www.z-img.com), and when I load the page into BeautifulSoup (bs4), Python crashes. The "problem details" shows that etree.pyd was the "Fault Module Name", which means its probably a parsing error, but so far, I can't quite nail down the cause of it.

Here's the simplest code I can boil it down to, on Python2.7:

import requests, bs4

url = r"http://z-img.com/search.php?&ssg=off&size=large&q=test"
r = requests.get(url)
html = r.content
#or 
#import urllib2
#html = urllib2.urlopen(url).read()
soup  = bs4.BeautifulSoup(html)

along with a sample output on PasteBin (http://pastebin.com/XYT9g4Lb), after I had passed it through JsBeautifier.com.

TankorSmash
  • 12,186
  • 6
  • 68
  • 106

2 Answers2

1

This is a bug that was fixed in lxml version 2.3.5. Upgrade to version 2.3.5 or later.

icktoofay
  • 126,289
  • 21
  • 250
  • 231
0

Oh, there you go, naturally the first thing I try after I submit the question is the solution: the <!DOCTYPE> tag seems to be at the root of it. I created a new HTML file, temp.html:

<!DOCTYPE>
<html>
</html>

and passed that to BeautifulSoup as an HTML string, and that was enough to crash Python again. So I just need to remove that tag before I pass the HTML to BeautifulSoup in the future:

import requests, bs4

url = r"http://z-img.com/search.php?&ssg=off&size=large&q=test"
r = requests.get(url)
html = r.content
#or 
#import urllib2
#html = urllib2.urlopen(url).read()

#replace the declaration with nothing, and my problems are solved
html = html.replace(r"<!DOCTYPE>", "")
soup  = bs4.BeautifulSoup(html)

Hope this saves someone else some time.

TankorSmash
  • 12,186
  • 6
  • 68
  • 106
  • That sounds like a serious bug. Perhaps you should report it. – icktoofay Apr 28 '13 at 04:29
  • Actually, it looks like it was already reported and [fixed in `lxml` version 2.3.5](https://github.com/lxml/lxml/commit/901c891b37e9bc54834d0ba2396d33612e08b85f). – icktoofay Apr 28 '13 at 04:34
  • @icktoofay Thanks for checking, I'm still on 2.3.4 (just found out about `pkg_resources` too, so bonus http://stackoverflow.com/a/4939465/541208) – TankorSmash Apr 28 '13 at 04:38