UnicodeEncodeError: 'ascii' codec can't encode characters in position 26-27: ordinal not in range(128)

Question

I'm running a job that runs a python script collecting Facebook events data from Facebook Graph API and printing out information about the events using Heroku Scheduler. Since it is an automated job that runs daily at a specific time in the cloud, the printing to screen will be logged into Heroku logs when that job starts running in the cloud by Heroku Scheduler.

The weird thing is, I have no problem when I run the python script manually by executing a heroku run command, but when it is run by the Heroku Scheduler automatically in the cloud and logging the printing, there is an encoding error.

The relevant python script with the error is as follows:

def importFromPages():
    sources = sources_collection.find(
        projection= {
            'id'        : 1,
            'source'    : 1,
            'name'      : 1,
            '_id'       : 0
        }
    )

    headers = {
        'Content-Type'  : 'application/json'
    }

    params = {
        'fields'        : 'id, name, cover, start_time, end_time, place',
        'access_token'  : FB_ACCESS_TOKEN
    }

    for source in sources:
        print(source['name'])
        data = requests.get(
            'https://graph.facebook.com/v2.7/{}/events'.format(source['id']),
            params=params,
            headers=headers
        )

        if data.status_code == 200:
            data = json.loads(data.text)
            data = data['data']
            if len(data) == 0: continue

            for event in data:
                now     = arrow.now().timestamp
                end     = arrow.get(event.get('end_time', None)).timestamp

                if end != None:
                    if end <= now: continue

                print(event['id'])
                event['host'] = source
                events_collection.update(
                    { 'id'      : event['id'] },
                    { '$set'    : event },
                    upsert=True
                )
        else:
            print('data.status_code != 200')

The heroku logs that showed the encoding error when the job is scheduled to run is here:

2016-10-18T12:31:34.270891+00:00 app[scheduler.1368]: UNSW Bitfwd
2016-10-18T12:31:34.411757+00:00 app[scheduler.1368]: IGA UNSW
2016-10-18T12:31:34.531127+00:00 app[scheduler.1368]: UNSW Barbell Club
2016-10-18T12:31:34.721310+00:00 app[scheduler.1368]: 660525034113560
2016-10-18T12:31:34.724900+00:00 app[scheduler.1368]: Circusoc - The UNSW Circus Society
2016-10-18T12:31:34.947553+00:00 app[scheduler.1368]: UNSW Medical Music Society
2016-10-18T12:31:35.140233+00:00 app[scheduler.1368]: 1166082866820866
2016-10-18T12:31:35.145683+00:00 app[scheduler.1368]: MODsoc Ministry of Dance UNSW
2016-10-18T12:31:35.370460+00:00 app[scheduler.1368]: Arc at UNSW Art & Design
2016-10-18T12:31:35.616661+00:00 app[scheduler.1368]: 1145482732204348
2016-10-18T12:31:35.622455+00:00 app[scheduler.1368]: Accounting Society Of UNSW (AccSoc)
2016-10-18T12:31:35.815662+00:00 app[scheduler.1368]: 1375370362476567
2016-10-18T12:31:35.817958+00:00 app[scheduler.1368]: 1679765869019301
2016-10-18T12:31:35.821599+00:00 app[scheduler.1368]: Traceback (most recent call last):
2016-10-18T12:31:35.821601+00:00 app[scheduler.1368]:   File "data.py", line 127, in <module>
2016-10-18T12:31:35.821652+00:00 app[scheduler.1368]:     main()
2016-10-18T12:31:35.821655+00:00 app[scheduler.1368]:   File "data.py", line 124, in main
2016-10-18T12:31:35.821709+00:00 app[scheduler.1368]:     importFromPages()
2016-10-18T12:31:35.821711+00:00 app[scheduler.1368]:   File "data.py", line 92, in importFromPages
2016-10-18T12:31:35.821715+00:00 app[scheduler.1368]:     print(source['name'])
2016-10-18T12:31:35.821745+00:00 app[scheduler.1368]: UnicodeEncodeError: 'ascii' codec can't encode characters in position 26-27: ordinal not in range(128)
2016-10-18T12:31:36.333406+00:00 heroku[scheduler.1368]: Process exited with status 1
2016-10-18T12:31:36.337350+00:00 heroku[scheduler.1368]: State changed from up to complete

When I run the job manually using heroku run as follows, there is no encoding print error and the job is able to complete without any errors:

heroku run python data.py
Running python data.py on ⬢ eventobotdatacollector... up, run.4056 (Free)

Does anyone have an idea what is the problem here and why the different environments caused the error?

Python is trying to convert characters that fall outside of the scope of ascii encoding. You can force your encoding to UTF-8 instead of ascii. — nullByteMe, Oct 19 '16 at 00:23

score 3 · Accepted Answer · edited May 23 '17 at 12:17

I don't know about the influences of the different environments. But I was able to solve a similar problem by explicity unicode-encoding the data I wanted to print out (or write to a file, for that matter). Try

print(source['name'].encode('utf-8'))

or have a look here, for a more complete and knowledgeable explanation of unicode issues in Python: SQLite, python, unicode, and non-utf data

Hope this helps. Regards,

UnicodeEncodeError: 'ascii' codec can't encode characters in position 26-27: ordinal not in range(128)

1 Answers1