Logging Python stdout to File... with active stdout (backspacing/updating)

Question

Ok, so using this example, I'm logging my stdout to a file as well as sending it to the terminal.

But when I look at the log file, the backspaces are not processed, but printed along with the output.

Any way I could log the "final" state of stdout of a python script?

Can you use a regex to strip the unprintable characters from the output you send to the file? — Some programmer dude, Mar 30 '12 at 16:39
but then I still see every update iteration, good idea though. — jondavidjohn, Mar 30 '12 at 16:40
As it turns out, terminals are fundamentally different things than flat file logs. One is designed to be temporary (and thus invisibly updatable); the other is designed to be a *log*. — Amber, Mar 30 '12 at 16:56

score 1 · Answer 1 · answered Mar 30 '12 at 18:02

Here is a solution that takes the basic class from the answer you linked and adds some regex handling for \r and \b:

import sys
import re

class Logger(object):
    def __init__(self, filename="Default.log"):
        self.terminal = sys.stdout
        self.log = open(filename, "a")
        self.cr_pattern = re.compile("^.*\r", re.M)
        self.bs_pattern = re.compile(".\b")

    def write(self, message):
        self.terminal.write(message)
        message = self.bs_pattern.sub('', self.cr_pattern.sub('', message))
        self.log.write(message)

sys.stdout = Logger("yourlogfilename.txt")
print "Hello\rGoodbyee\b world!"

Example run:

$ python test.py
Goodbye world!
$ cat yourlogfilename.txt
Goodbye world!

Note that normally you should use raw string literals for your regular expressions, this is one of the rare cases where you shouldn't.

score 0 · Answer 2 · edited Oct 11 '12 at 01:48

I'm using the logging facility of Python and had the same problem that it transforms the correct end-of-line sequence "\x0D\x00\x0A\x00" to "\x0D\x0A\x00".

The program output which I want to log via Python is UTF-16 as you can see.

To avoid that Python does anything different from just writing the Bytes it receives from stdout to the logfile, I tried to add encoding="UTF-8" to logging.FileHandler().

The result is that '\x0D' isn't printed anymore and I get "\x0A\x00\x0A\x00" instead. That's a little bit better I guess.

On Linux it seems that there is no problem with that. I'm using the same script there but my program prints UTF-8 characters instead.

(There are two questions left here: Why does Python add another newline? And is it possible to log additional Information to the logfile? Putting an 'U' before strings or using "...".encode(UTF-16) doesn't work. But otherwise the logfile will contain strings of different encodings and becomes useless...)

Logging Python stdout to File... with active stdout (backspacing/updating)

2 Answers2

Linked