2

I am using Beautiful Soup 3.2 on python 2.7.1 here.

I have recently been trying to get something simple to work, but it seems rather tricky:

I do the following:

temp=BeautifulSoup(urllib2.urlopen(urlList[1], None,15))

However, I get the error:

File "/home/foo/k/kat/BeautifulSoup.py", line 1519, in __init__
BeautifulStoneSoup.__init__(self, *args, **kwargs)
File "/home/foo/k/kat/BeautifulSoup.py", line 1144, in __init__
self._feed(isHTML=isHTML)
File "/home/foo/k/kat/BeautifulSoup.py", line 1186, in _feed
SGMLParser.feed(self, markup)
File "/usr/lib/python2.7/sgmllib.py", line 104, in feed
self.goahead(0)
File "/usr/lib/python2.7/sgmllib.py", line 143, in goahead
k = self.parse_endtag(i)
File "/usr/lib/python2.7/sgmllib.py", line 320, in parse_endtag
self.finish_endtag(tag)
File "/usr/lib/python2.7/sgmllib.py", line 358, in finish_endtag
method = getattr(self, 'end_' + tag)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfa' in
position 4: ordinal not in range(128)

If I run the same loop another time, sometimes, I also get:

File "/home/foo/k/kat/BeautifulSoup.py", line 1519, in __init__
BeautifulStoneSoup.__init__(self, *args, **kwargs)
File "/home/foo/k/kat/BeautifulSoup.py", line 1144, in
__init__
self._feed(isHTML=isHTML)
File "/home/foo/k/kat/BeautifulSoup.py", line 1186, in _feed
SGMLParser.feed(self, markup)
File "/usr/lib/python2.7/sgmllib.py", line 104, in feed
self.goahead(0)
File "/usr/lib/python2.7/sgmllib.py", line 143, in goahead
k = self.parse_endtag(i)
File "/usr/lib/python2.7/sgmllib.py", line 320, in parse_endtag
self.finish_endtag(tag)
File "/usr/lib/python2.7/sgmllib.py", line 358, in finish_endtag
method = getattr(self, 'end_' + tag)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
4-5: ordinal not in range(128)

How do I avoid this errors? Obviously something is wrong with the sgmllib.py.

I tried some solutions from SOF:

*] Tried soup = BeautifulSoup(page, fromEncoding=<encoding of the page>) Result: Dosent work, same errors.

*] Tried upgrading my sgmllib.py from a 2.7.2 version onto my 2.7.1 verision Result: Dosent work, same errors.

*] Tried html = BeautifulSoup(page.encode('utf-8')) Result: Dosent work, same errors.

I would appreciate any suggestions as to how to solve this encode error.

JohnJ
  • 6,736
  • 13
  • 49
  • 82

1 Answers1

4

try this code in your module

if __name__ == "__main__":
    reload(sys)
    sys.setdefaultencoding("utf-8")
Eric Fortin
  • 7,533
  • 2
  • 25
  • 33
DrStrangeLove
  • 11,227
  • 16
  • 59
  • 72
  • Pardon my ignorance, but "inside of your if name == "main":" - is this a piece of code to be inserted too? if yes, where should i include this code? sys.setdefaultencoding does not work. sys.getdefaultencoding returns ascii. – JohnJ Jan 24 '12 at 15:32
  • Updated code to show the right syntax as it was obfuscated by formatting. – Eric Fortin Jan 24 '12 at 16:01
  • 1
    That is absolute genius. The above code elimates the UnicodeEncodeError. What I seem to fail to understand is how the above code works. what does "if __name__ == "__main__":" do? Can you please explain? – JohnJ Jan 24 '12 at 16:29
  • 1
    @user7699 Recently i stumbled upon the same problem, so i decided to post this simple trick. :) Defaultsetting needs to be set if you work with Unicode and version of your Python interpreter – DrStrangeLove Jan 24 '12 at 20:19
  • Thanks a lot again for your explanation. That has solved the problem I had been struggling with for about 2 days now! – JohnJ Jan 25 '12 at 12:24