2

I use a Python script written in 2.7 (seafile-cli, from Seafile, a file synchronization solution).

I know that unicode is problematic in Python 2 but the filenames with diacritic signs are thankfully handled correctly when starting the script:

$ # seaf-cli status
# Name  Status  Progress
photos  downloading     0/0, 0.0KB/s
Ma bibliothèque downloading     566/1770, 1745.7KB/s
videos  downloading     28/1203, 5088.0KB/s
dev-perso       downloading     0/0, 0.0KB/s
dev-pro downloading     0/0, 0.0KB/s

To my surprise, when piping this output, the Python script crashes with UnicodeEncodeError:

$ seaf-cli status | cat -
# Name  Status  Progress
photos  downloading     0/0, 0.0KB/s
Traceback (most recent call last):
  File "/usr/bin/seaf-cli", line 845, in <module>
    main()
  File "/usr/bin/seaf-cli", line 841, in main
    args.func(args)
  File "/usr/bin/seaf-cli", line 649, in seaf_status
    tx_task.rate/1024.0)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 11: ordinal not in range(128)

While I understand that it might have had issues with Ma bibliothèque in the first place (which it has not), why piping it triggers a Traceback?

Shouldn't that be the problem of the shell? - the output has "left" the script at that point.

EDIT: the answer is in another question. Marking as duplicate.

WoJ
  • 27,165
  • 48
  • 180
  • 345
  • Possible duplicate of [Setting the correct encoding when piping stdout in Python](https://stackoverflow.com/questions/492483/setting-the-correct-encoding-when-piping-stdout-in-python) – WoJ Jun 20 '18 at 14:50

1 Answers1

2

Python knows how to handle encoding inside your program because it uses whatever encoding your terminal application is using.

When you are sending (piping) your output out, it needs to be encoded. This is because using pipe actually sends streams of bytes between the applications. Every pipe is a unidirectional channel, where one side writes data and the other side reads it.

Using pipes or redirections, you are sending out data to a fd, which is read by the another application.

So you need to make sure Python correctly encode the data before it sends it out, and then the input program needs to decode it before processing.

You also might find this question useful

Update: I'll try to elaborate more about encoding. What I mean by the first line of my answer is, because your Python interpreter uses specific encoding, it knows how to transform the hexa values (actual bytes) to symbols.

My interpreter doesn't; if I try to create a string from your text - I get an error:

>>> s = 'bibliothèque'
Unsupported characters in input

This is because I use different encoding on my interpreter.

Your shell uses different encoding than the Python interpreter. When Python sends data out of your program, it uses default encoding: ASCII. It can't translate your special character (which displayed by the hexa value \xe8) using ASCII. So, you have to specify which encoding to use in order for Python to send it.

You might be able to overcome this if you change your shell encoding - check this question on SO.

PS - There's a great video by Ned Batchelder about Unicode on youtube - Maybe this will shed some more light on the subject.

Chen A.
  • 10,140
  • 3
  • 42
  • 61
  • Yes, I understand this - having gone through the pains of handling unicode in Python 2. When starting the script on its own (without a pipe) the output is correct. It means that the script has correctly encoded it. When piping it is up to the receiving program to handle what it gets (`cat` in my case). But is the Python script which crashes. – WoJ Jun 20 '18 at 14:02
  • And by the way it does not matter whether this is a pipe, a redirection `>` crashes it as well. – WoJ Jun 20 '18 at 14:03
  • Your code works when run in an script because Python **encodes** the output to whatever encoding your terminal application is using. If you are piping or redirecting it somewhere else - then you must encode it yourself. – Chen A. Jun 20 '18 at 14:04
  • Sorry, but I do not understand. The string is output using a `print`. If there was an encoding problem, it would have emerged at that point (a single line `print 'école'` in a python 2 script crashes - because of a lack of encoding outputting the string). Since it has already issued this line without problems when ran standalone, anything that happens on stdout (where the stream is sent) is up the shell to manage. How can redirecting or not this stdout stream can impact the script? – WoJ Jun 20 '18 at 14:13
  • https://stackoverflow.com/a/4027726/903011 gives the exact solution. I will accept yours as it led me to understand the difference in the descriptors in `print` vs piping o redirecting. Thanks. – WoJ Jun 20 '18 at 14:49