Replication of the Bash tee command behavior in python

Question

I have a collection of Bash scripts which I want to recreate in python. One of the key feature of these scripts is when i execute them it will save the contents of the terminal into a logfile. In Bash simply i used the tee command.

2>&1 | tee "logfile.txt";

the problem is to find equal solution for python.

I found two half of this "puzzle" so far (solution A and B), one of the expected behaviour works in one of the scripts but not in the other and vice versa.

solution A)

#!/usr/bin/env python3

import sys
from subprocess import Popen, PIPE, STDOUT


with Popen(['ffmpeg','-i','1.webm','-y','1.mp3'], stdout=PIPE, stderr=STDOUT, bufsize=1) as p, \
    open('logfile.txt', 'ab') as file:
    for line in p.stdout:
        sys.stdout.buffer.write(line)
        file.write(line)

solution B)

#!/usr/bin/env python3

import sys
from subprocess import Popen, PIPE


with Popen(['ffmpeg','-i','1.webm','-y','1.mp3'], stdout=PIPE, bufsize=1, universal_newlines=True) as p:
    logfile = open('logfile.txt', 'w')
    for line in p.stdout:
        print(line, end='')

i tried to "merge" the features of these 2 code snippets, but i cant figure it out, how to put it together.

What i looking for is the EXACT behavior replication of the tee command in a python script file. which means...

the contents of the terminal appear in the terminal window AND saved into a log file (just like solution A)
when i start the python script file, i want to follow the progress of the process in the terminal, to check how far is from completion (just like solution B). I dont want to stare at a blank screen until the process completes (solution A).

I would appreciate the help.

for testing i use a webm format file (downloaded with youtube-dl) and convert it to mp3 with ffmpeg in cygwin. you can download the ffmpeg binary from here if you want experimenting with it https://www.ffmpeg.org/download.html

Thank you!

Could it be just a matter of flushing stdout after each line in solution A? And why use `sys.stdout.buffer.write(line)` in solution A instead of `print` like in solution B? — joanis, Mar 17 '20 at 21:51
I think all you need is indeed `sys.stdout.flush()` inside your inner loop. Relevant (but not duplicate) question: https://stackoverflow.com/a/10019605/3216427 — joanis, Mar 17 '20 at 22:00

score 1 · Answer 1 · answered Mar 17 '20 at 22:50

I did some testing and no, sys.stdout.flush() doesn't solve it. The problem appears to be within the Popen/PIPE implementation itself -- the way it sets up pipes between the subprocess and your process introduces buffering.

What does seem to fix this is to:

$ export PYTHONUNBUFFERED=1

in the environment you run your Python script from. (The variable can be set to anything.)

To solve this within your Python script itself, there may be a more elegant way, but this rather odd approach seemed to work for me. I'm rerunning the script with that environment variable set:

import os
from subprocess import run

if not "PYTHONUNBUFFERED" in os.environ:
    os.environ["PYTHONUNBUFFERED"] = "1"
    completed = run(sys.argv)
    sys.exit(completed.returncode)

Found some pointers to this solution here in question 230751.

score 1 · Answer 2 · answered Mar 17 '20 at 23:08

You are reading line by line, but ffmpeg does not output distinct lines.

You should do what tee does and read buffer by buffer, ignoring linefeeds:

#!/usr/bin/env python3.8

import sys
from subprocess import Popen, PIPE, STDOUT

with Popen(['ffmpeg','-i','1.webm','-y','1.mp3'], stdout=PIPE, stderr=STDOUT, bufsize=0) as p, \
    open('logfile.txt', 'ab') as file:

    while buf := p.stdout.read(4096):
        sys.stdout.buffer.write(buf);
        sys.stdout.buffer.flush()
        file.write(buf)

CuriousOne · Answer 3 · 2022-09-04T07:02:01.237

I decided to re-open the case, and with some tinkering i was able to come up with the solution, but i thank you all for your efforts!

#! /bin/python3

from subprocess import Popen, PIPE, STDOUT
 
with Popen(['ffmpeg','-i','test.wav','-y','1.mp3'], stdout=PIPE, stderr=STDOUT, universal_newlines=True) as process, \
    open('logfile.txt', 'w') as logfile:
    for line in process.stdout:
        
        print(line) 
        logfile.write(line)

    logfile.close()

Replication of the Bash tee command behavior in python

3 Answers3