UnicodeEncodeError in BeautifulSoup webscraper

Question

I'm having a unicode encode error with the following code for a simple web scraper.

print 'JSON scraper initializing'

from bs4 import BeautifulSoup
import json
import requests
import geocoder


# Set page variable
page = 'https://www.bandsintown.com/?came_from=257&page='
urlBucket = []
for i in range (1,3):
    uniqueUrl = page + str(i)
    urlBucket.append(uniqueUrl)

# Build response container
responseBucket = []

for i in urlBucket:
    uniqueResponse = requests.get(i)
    responseBucket.append(uniqueResponse)


# Build soup container
soupBucket = []
for i in responseBucket:
    individualSoup = BeautifulSoup(i.text, 'html.parser')
    soupBucket.append(individualSoup)


# Build events container
allSanFranciscoEvents = []
for i in soupBucket:
    script = i.find_all("script")[4]

    eventsJSON = json.loads(script.text)

    allSanFranciscoEvents.append(eventsJSON)


with open("allSanFranciscoEvents.json", "w") as writeJSON:
   json.dump(allSanFranciscoEvents, writeJSON, ensure_ascii=False)
print ('end')

The odd thing is the sometimes, this code works, and doesn't give an error. It has to do with the for i in range line of the code. For example, if I put in (2,4) for the range, it works fine. If I change it to 1,3, it reads:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 12: ordinal not in range(128)

Can anyone tell me how to fix this issue within my code? If I print allSanFranciscoEvents, it is reading in all the data, so I believe the issue is happening in the final piece of code, with the JSON dump. Thanks so much.

Answer 1

Best Fix

Use Python 3! Python 2 is going EOL very soon. New code written in legacy python today will have a very short shelf life.

The only thing I had to change to make your code work in python 3 was to call the print() function instead of the print keyword. Your example code then worked without any error.

Persisting with Python 2

The odd thing is the sometimes, this code works, and doesn't give an error. It has to do with the for i in range line of the code. For example, if I put in (2,4) for the range, it works fine.

That is because you are requesting different pages with those different ranges, and not every page has a character that can't be converted to str using the ascii codec. I had to go to page 5 of the response to get the same error that you did. In my case, it was the artist name, u'Mø' that caused the issue. So here's a 1 liner that reproduces the issue:

>>> str(u'Mø')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf8' in position 0: ordinal not in range(128)

Your error explicitly singles out the character u'\xe9':

>>> str(u'\xe9')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0: ordinal not in range(128)

Same issue, just different character. The character is Latin small letter e with acute. Python is trying to use the default encoding, 'ascii', to convert the Unicode string to str, but 'ascii' doesn't know what the code point is.

I believe the issue is happening in the final piece of code, with the JSON dump.

Yes, it is:

>>> with open('tmp.json', 'w') as f:
...     json.dump(u'\xe9', f, ensure_ascii=False)
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python2.7/json/__init__.py", line 190, in dump
    fp.write(chunk)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1: ordinal not in range(128)

And from the traceback, you can see that it's actually coming from writing to the file (fp.write(chunk)).

file.write() writes a string to a file, but u'\xe9' is a unicode object. The error message: 'ascii' codec can't encode character... tells us that python is trying to encode that unicode object to turn it into a str type, so it can write it to the file. Calling encode on the unicode string uses the "default string encoding", which is defined here to be 'ascii'.

To fix, don't leave it up to python to use the default encoding:

>>> with open('tmp.json', 'w') as f:
...     json.dump(u'\xe9'.encode('utf-8'), f, ensure_ascii=False)
...
# No error :)

In your specific example, you can fix the intermittent error by changing this:

allSanFranciscoEvents.append(eventsJSON)

to this:

allSanFranciscoEvents.append(eventsJSON.encode('utf-8'))

That way, you are explicitly using the 'utf-8' codec to convert the Unicode strings to str, so that python doesn't try to apply the default encoding, 'ascii' when writing to the file.

Answer 2

eventsJSON is object it can't use eventsJSON.encode('utf-8'). For Python 2.7 to write the file in utf-8 or unicode you can use codecs or write it using binary or wb flag.

with open("allSanFranciscoEvents.json", "wb") as writeJSON:
   jsStr = json.dumps(allSanFranciscoEvents)
   # the decode() needed because we need to convert it to binary
   writeJSON.write(jsStr.decode('utf-8')) 
print ('end')

# and read it normally
with open("allSanFranciscoEvents.json", "r") as readJson:
    data = json.load(readJson)
    print(data[0][0]["startDate"])
    # 2019-02-04