UnicodeEncodeError only when running as a cron job

Question

My program works right in the commandline, but when I run it as a cron job it crashes on the error:

UnicodeEncodeError: 'ascii' codec can't encode character
u'\xa7' in position 13: ordinal not in range(128)

It crashes on the statement

print title

Why this is happening only when the app runs as a cron job? How could this be fixed?

I tried (with no help):

print unicode(title)

Python is 2.7

See http://stackoverflow.com/questions/492483/setting-the-correct-encoding-when-piping-stdout-in-python — Mark Ransom, Mar 29 '12 at 19:59
I think it's the same issue; Python 2 uses Unicode when stdout is a terminal but not when it's a pipe (as it would be in a cron job). — Mechanical snail, May 31 '13 at 19:16

score 98 · Answer 1 · answered Aug 01 '12 at 15:08

98

Nah, you can have it simpler. Just define PYTHONIOENCODING before executing this script. Like this:

PATH=<your path>
MAILTO=someone@somewhere
PYTHONIOENCODING=utf8

* * * * * /run/your/script

answered Aug 01 '12 at 15:08

kworr

3,579
1
21
33

1

Indeed this is the correct answer. Cool! :) – mac Jan 09 '13 at 14:56
14

If I could I'd give you +2. This shit's been messing with my head! – TTT Jul 04 '13 at 09:17
1

What he said... here I am, two years later, and this saved lots of keyboards from being smashed. – mudda Sep 24 '15 at 12:35
This should be accepted answer. @xralf Could you update it? – chhantyal Dec 09 '15 at 09:28
1

Holy crap thanks so much! I've been going nuts for a couple of hours trying to figure out what the heck. – xv70 Mar 24 '16 at 19:29
don't forget "export PYTHONIOENCODING" – Thierry Oct 11 '17 at 06:48
@Thierry that's crontab, not a bash script. – kworr Oct 12 '17 at 08:26

score 4 · Accepted Answer · answered Mar 29 '12 at 19:50

4

Aside: This is a common problem; as such this is probably a duplicate question.

The default encoding on 2.7 is ascii.
You need to provide an encoding for your program's output.
A common encoding to use is 'utf8'.

So you'd do instead:

print title.encode('utf8')

Here's one way to check the default encoding:

import sys

sys.getdefaultencoding()
# -> 'ascii'

answered Mar 29 '12 at 19:50

mechanical_meat

163,903
24
228
223

Thank you for answer, but why this doesn't work only when the app runs as a cron job? – xralf Mar 29 '12 at 19:55
...difficult say based on provided information. You may be passing different input, for one. Whether or not it works isn't the issue. As a responsible programmer you should care about your code working with non-ascii characters. – mechanical_meat Mar 29 '12 at 19:57
Actually the string has been taken from the SQlite database. It was of type TEXT, so encoded as UTF-8 http://sqlite.org/datatype3.html. Maybe Python changed it when it was combined with another strings via "%s %s" % (s1, s2) – xralf Mar 29 '12 at 20:09
1

Ok, so *decode* it to Unicode when you get it from db; e.g. `'mystring'.decode('utf8')`. Then encode on output as mentioned above. And, yes, Python 2.7 does automatic conversion to Unicode when strings are combined. So you don't need to decode in that case, but it would be clearer if you did. – mechanical_meat Mar 29 '12 at 20:11
@bernie: It does automatic conversion *with the `ascii` codec*. It's not just clearer to decode it, it's really the only sane way to avoid bugs. – Daenyth Mar 29 '12 at 20:21
1

@xralf: it is different with cron job probably because it is run as a different user or in a non-interactive shell, where the `LANG` variable is not set. Check the value of `os.environ.get('LANG')` and the value for `sys.stdout.encoding`. – piro Mar 29 '12 at 22:31
`>>> import os, sys >>> os.environ.get("LANG") 'en_US.UTF-8' >>> sys.stdout.encoding 'UTF-8'` – xralf Mar 30 '12 at 06:34

UnicodeEncodeError only when running as a cron job

2 Answers2

Linked