Errors when trying to save command line output to a file

Question

I was running a python tool and trying to save its output to a file. If I don't save the output to a file, the tool runs perfectly fine. But when I try to save the output to the file, it throws following error and interrupt the program:

  File "./androdiff.py", line 118, in <module>
main(options, arguments)
  File "./androdiff.py", line 94, in main
ddm.show()
  File "./elsim/elsim/elsim_dalvik.py", line 772, in show
self.eld.show()
  File "./elsim/elsim/elsim.py", line 435, in show
i.show()
  File "./elsim/elsim/elsim_dalvik.py", line 688, in show
  print hex(self.bb.bb.start + self.offset), self.pos_instruction, self.ins.get_name(), self.ins.show_buff( self.bb.bb.start + self.offset )
  UnicodeEncodeError: 'ascii' codec can't encode character u'\u0111' in position 35: ordinal not in range(128)

I've tried command |less , command > output and command | tee output, all of them will throw such error.

Please help to resolve the issue.

Thanks!

When you "save to file" are you redirecting `stdout`? Like so `python app.py > file` ? — Martin Konecny, Jun 19 '14 at 02:28

score 3 · Answer 1 · answered Jul 21 '14 at 17:03

3

Set PYTHONIOENCODING environment variable explicitly if stdout character encoding can't be determined automatically e.g., when the output is redirected to a file:

$ PYTHONIOENCODING=utf-8 python app.py > file

Don't hardcode the character encoding in your scripts if the output may go to a terminal; print Unicode strings instead and let users to configure their environment.

answered Jul 21 '14 at 17:03

jfs

399,953
195
994
1,670

This is what I needed - terminal took the output OK, and for small outputs I could copy-paste it correctly. The variable name is not very handy, but it could save the day. I need to check, if this works with `os.environ`. OK - it does. – Tomasz Gandor Sep 09 '14 at 07:41

Martin Konecny · Accepted Answer · 2014-06-19T18:39:12.490

You will want to specify the encoding of your string before you print it:

print unicode(hex(self.bb.bb.start + self.offset)).encode('utf-8')
print unicode(self.pos_instruction, self.ins.get_name()).encode('utf-8')
print unicode(self.ins.show_buff( self.bb.bb.start + self.offset )).encode('utf-8')

The reason this works is because python automatically encodes your string correctly (in your case utf-8) when printing to the terminal (it detects that the terminal uses utf-8).

When you are redirecting your output to a file instead, Python has no information about what encoding it should use and it defaults to ascii instead (which is causing your error).

As a general rule of thumb, make sure you always encode your string before printing to make print work in all environments.

The best method may be to define your own print method for this:

def myprint(unicodestr): 
    print unicodestr.encode('utf-8')

If you want to avoid the above and make printing with utf-8 encoding the default you can do

import sys
import codecs
sys.stdout=codecs.getwriter('utf-8')(sys.stdout)

Beware of this approach! Some third-party libraries may depend on the default encoding being ascii and break. Note that this whole mess has been resolved in Python 3 (which defaults to UTF-8 encoding)

Thanks! Martin. Can I change the defaults from ascii to utf-8 instead of change every invocations of `print`? There are tons of places using `print` in the tool, and I don't bother to refactor every one of them. — Wei Yang, Jun 19 '14 at 18:36
you shouldn't hardcode user's terminal character encoding in your script. You could use [`PYTHONIOENCODING` instead](http://stackoverflow.com/a/24870668/4279). — jfs, Jul 31 '14 at 13:42

Errors when trying to save command line output to a file

2 Answers2

Linked