Setting Python to stdout to match encoding of terminal

Question

I have a python project where I have to display data piped to me from another machine with an unknown encoding. I'm running python 3 on an Ubuntu VM. What I get is a stream of bytes (perhaps command output or a cat'd file or similar). I just need to display the data, regardless of source, as best as I can.

As a test, I'm trying to cat /dev/urandom and have it be displayed the same way via python as it would be if I typed the command myself. To make it reproducible, I've used head -n 2 /dev/urandom instead of the endless stream that you get with cat.

In bash, when I cat, the file, I get the standard random gunk. I have LANG=en_US.ETF-8. Lots of character don't really render (sort of diamond with a question mark in it) or just blanks (because its obviously not UTF-8, its just random bytes of raw data)

eJ̘��}��jf��)���N�n��t��8=����X-�L�^t�M����Z���g�8#K T��c��z�ZO+�ϩD1{|EX
��)'���ei۝{W�r��畴��Ii�Y���
                        �}���+��;-�i-
                                     S��Az
                                          uV�1XBxFZ3+4��G�*��Q�+!  

However, if I read the file in python and print to standard out, I get encoding errors unless I use 'latin-1'. I've even tried to use the encoding from the default stream, assuming it inherited from the terminal. This clearly isn't right as I would need the encoding from the stream on the remote end of the pipe(which I don't have).

>>> f = open("foo.txt", "rb")
>>> data = f.read(530)
>>> import sys
>>> sys.stdout.write(data.decode(sys.stdout.encoding))

As expected, the error produced is based on random data not following the UTF8 encoding standard :

>>> sys.stdout.write(data.decode(sys.stdout.encoding))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 0: invalid start byte

So, its picking up the UTF-8 from LANG(?) and apply it, but it simply has an encoding issue and fails. If I explicitly use "latin-1" I get something similar, but it isn't the same as the terminal either.

sys.stdout.write(data.decode('latin-1'))

Yields (notional example -- not actual text) :

©7yIº*ø^Mÿ*Ig«áEIt±.Q   ÈyT?æsÎ_%v1DÎú¹×,sÛÐûóÜun¢$&6YuApÁ¼­pnòàJð

So, the question is... how do I read the terminal settings so I can use them to decode and reproduce exactly what would have appeared on the terminal?

I've checked out these other questions : Why does Python print unicode characters when the default encoding is ASCII? and Convert bytes to a Python string and they cover some parts, but I'm not understanding the interaction with the shell/bash/etc.

EDIT (Answer Evaluation):

Using errors="replace" gets me close. os.write duplicates the output of cat. The difference seems to be based on how error characters are clumped together, so the difference isn't really important.

EDIT EDIT (Actual Solution) : In the end, I apparently need to read the locale / remote encoding from the original machine so I can send it across, then interpret the bytes as that encoding, then transcode to the local machine encoding, then display. The project wasn't well thought through in this sense as I am not sent the remote information and I clearly need it. I ended up using this (for the current build as it gets rid of the errors) :

sys.stdout.write(data.decode('utf-8', errors='replace'))
sys.stdout.flush()

EDITs ( cleared up text of question to focus on actual topic )

Answer 1

Addressing the problem and not the question given (https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem)

I have a python project which is getting sent to various users on various linux flavors (vms, real machines, etc) all with python 3. As part of this program, the python may display one of their local files to the terminal screen. I'm looking for a way to read the settings of the terminal to use them for displaying the data. In this way, it will be up to user to set their terminal to handle extended character sets or whatever if that is what they want.

You have already discovered that Python uses the locale to resolve the likely encoding of the terminal - If you write a Python 3 str to stdout, using write() or print(), you will find that it's automatically encoded for the terminal. That means that your code does not need (nor shouldn't have to) detect its environment.

I said "likely encoding", as the decoding of characters is not the responsibility of the shell, but the responsibility of terminal (Terminal/iTerm/Putty etc), which may be running on the user's desktop remotely. You will have to hope that most people will have left the locale and terminal encoding as default, which is now thankfully usually UTF-8.

If you're to open a text file on their local machine, you should also apply a encoding when reading that file (You could just write it straight out with decoding/encoding but you won't be able to manipulate it as cleanly). Thankfully, open() will also use locale.getpreferredencoding(), sourced from the user's locale, to set the encoding to decode the file.

That means, if a user's LANG='en_GB.UTF-8', then locale.getpreferredencoding() == "UTF-8". If you open a file with open('file.txt', 'r'), that file will be decoded as UTF-8 into a Python str.

You will be able to write/print this data to the console, confident that you've done what you can to work around the encoding legacy we have to live with.

Answer 2

If you want to reproduce what cat would do, then just write the data to stdout without decoding it:

import os
f = open("/dev/urandom", "rb")
data = f.read(1000)
os.write(1, data)

Answer 3

sys.stdout (and sys.stdin and sys.stderr) are text files. They have an encoding associated with them, which you can read from their aptly named encoding attributes, and they expect to work with strings, which they will encode or decode automatically, depending on I/O direction. On Linux, you should expect the encoding to be chosen based on environment variables of the Python process's initial environment. I am unaware of any mechanism to change the encoding of an open file, but in this case the encoding should indeed be the same as the one the terminal expects.

As thatotherguy wrote, you can perform the equivalent of a cat command by reading from the source file in binary mode, and using a low-level os.write() to send the bytes to the file descriptor for stdout. Note, however, that the underlying system function does not necessarily always write the full number of bytes specified, so that you must generally call it in a loop to ensure that all the desired bytes are pushed out. The Python docs don't specify, but inasmuch as the method has an equivalent interface to that of the underlying system call, one might best assume that it has the same semantics, too.

But really, that's all going about things the wrong way. If you force a raw dump of the file's bytes, then you don't force users just to configure their environment to support extended character sets, you force them to configure their environment to support the exact character set in which the file is encoded (or else to endure gibberish).

A better solution would be to open the file as a text file, specifying the correct encoding (which presumably you know), to read in the data via the file object, so that Python decodes it correctly, and to write the resulting string(s) to stdout, whereby they will be correctly encoded for the terminal, at least inasmuch as that is possible. That way you can accommodate any character encoding at the terminal that supports all the characters in the file -- it doesn't have to actually be the same as the file's.

Answer 4

You are almost there, just pass errors='replace' to the decoding function.

with open('/dev/urandom', 'rb') as f:
    x = f.read(100)

print(x.decode(sys.stdout.encoding, errors='replace'))

�{�ʛf$��s���<�w'`�i6�/��Z�ʫ;����ek|%�-+����V�U��;w>פ���TV��
�}���639

sys.stdout.write(x.decode(sys.stdout.encoding, errors='replace'))

�{�ʛf$��s���<�w'`�i6�/��Z�ʫ;����ek|%�-+����V�U��;w>פ���TV��
�}���639

os.write(1, x)

�{�ʛf$��s���<�w'`�i6�/���Z�ʫ;����ek|%�-+����V�U��;w>פ����TV��
�}���639