Getting a Unicode error when printing prettified BeautifulSoup

Question

I am currently taking a course on Python and during our unit on Beautiful Soup the instructor uses the following code:

import requests, pprint
from bs4 import BeautifulSoup

url = 'https://www.epicurious.com/search/tofu%20chili'
response = requests.get(url)
page_soup = BeautifulSoup(response.content, 'lxml')
print(page_soup.prettify())

When I run this code, I get the following error:

Traceback (most recent call last):
  File "/Users/arocklin/Documents/Python/whiteboard2.py", line 11, in <module>
    print(page_soup)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 1479: ordinal not in range(128)

I was wondering why I got this since it worked for him and how I can fix it going forward. Thanks!

Update: When I change the last line to: print(results_page.prettify(encoding='utf-8')) I do not get the error but it also does not print neatly and it prints in bytestring. Also when I run this from the terminal it works fine but when I run it from Atom I run into the issue — anonym00se, Sep 23 '17 at 02:14
maybe just add `#!/usr/bin/env python # -*- coding: utf-8 -*-` as the first two lines in your python file as in [Working with utf-8 encoding in Python source](https://stackoverflow.com/q/6289474/1248974) — chickity china chinese chicken, Sep 23 '17 at 05:55
Thanks! I think it is an issue with Atom, though, since UTF-8 is the default encoding in Python 3. When I put that line at the top, the issue remains — anonym00se, Sep 23 '17 at 12:40

score 0 · Answer 1 · answered Sep 24 '17 at 15:06

Your problem is not related to BeautifulSoup or to parsing HTML. Your code up to and including BeautifulSoup.prettify gets you some unicode string defined by a webserver not under your control.

That more or less arbitrary unicode string you then try to print.

On a system where Python has determined that the terminal sys.stdout can only handle ascii encoded strings, and if the webserver has (for reasons entirely beyond your control) has decided to give you some Unicode characters outside the ASCII range, Python cannot encode that character and throws an exception.

I suggest you research how your version of Python determines the encodings/codecs to use on the platform you are running Python on.

Then put a test case into your program's test suite which actually verifies it can properly output Unicode strings. For that test, you can replace your entire program with

print(u"foo\xe9bar")

Thank you so much. It seems like this issue is with Atom for some reason. Not sure if you know any solutions to that or not. I am using Atom V 1.20.1 x64 on Mac OS Sierra 10.12. My terminal can print the output but atom has issues with it for some reason. — anonym00se, Sep 24 '17 at 21:35

Getting a Unicode error when printing prettified BeautifulSoup

1 Answers1