How to handle UnicodeEncodeError in python script?

Question

I have a python script from which I am generating list of softwares installed on my machine. Let the name of this script be 'install.py' - it looks as below:

import wmi
w = wmi.WMI()
for p in w.Win32_Product():
    if (p.Version is not None) and (p.Caption is not None):
        print  p.Caption + " & "+ p.Version + "\\\\"
        print "\hline"

Now I am actually writting the output of this script into a output.tex file by executing it from some another script say "output_file.py", looks as below:

with open("D:/output.tex", "w+") as output:
    process = sp.call(["python", "D:/install.py"], stdout=output)

So when the above piece is executed I do get output in "output.tex" but along with error as :

UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 43:
ordinal not in range(128)

So, actually don't get details of all software on my system. So what shall I do to remove this error in my script. Kindly help.

You could try to use `from __future__ import unicode_literals` at the beginning of your python script. Or you could try to open the file with unicode encoding. See https://docs.python.org/2/howto/unicode.html — Randrian, Dec 21 '15 at 09:43

score 0 · Answer 1 · edited May 23 '17 at 11:52

The immediate issue is that Python 2 uses ascii encoding (sys.getdefaultencoding()) when sys.stdout is redirected. You could override it with PYTHONIOENCODING envvar:

call([sys.executable, os.path.join(script_dir, 'install.py')], stdout=file,
     env=dict(os.environ, PYTHONIOENCODING='utf-8'))

It would be enough on *nix system but Windows may interfere with passing bytes between install.py and the file (e.g., the pipe | is broken for binary content in PowerShell).

To workaround it, you could pass the filename as the command-line parameter to install.py and write to the file instead of printing to sys.stdout there.

The right solution is to put the necessary functionality into a function and import the module instead of running it as a subprocess. You can use multiprocessing if you want to run the code in a different process as shown in the link.

How to handle UnicodeEncodeError in python script?

1 Answers1