3

(Windows)

I wrote some Python code that calls the program SoX (subprocess module), which outputs the progress on STDERR, if you specify it to do so. I want to get the percentage status from the output. If I call it not from the Python script, it starts immediately and has a smooth progression till 100%.

If I call it from the Python script, it lasts a few seconds till it starts and then it alternates between slow output and fast output. Although I read char by char sometimes there RUSHES out a large block. So I don't understand why at other times I can watch the characters getting more one by one. (It generates 15KiB of data in my test, by the way.)

I have tested the same with mkvmerge and mkvextract. They output percentages, too. Reading STDOUT there is smooth.

This is so unreliable! How can I make the reading of sox's stderr stream smoother, and perhaps prevent the delay at the beginning?

How I call and read:

process = subprocess.Popen('sox_call_dummy.bat', stderr = subprocess.PIPE, stdout = subprocess.PIPE)
while True:
    char = process.stderr.read(1).encode('string-escape')
    sys.stdout.write(char)
rynd
  • 1,865
  • 5
  • 22
  • 24

1 Answers1

1

As per this closely related thread: Unbuffered read from process using subprocess in Python

process = subprocess.Popen('sox_call_dummy.bat', 
                stderr = subprocess.PIPE, bufsize=0)
while True:
    line = process.stderr.readline()
    if not line: 
        break
    print line

Since you aren't reading stdout, I don't think you need a pipe for it.

If you want to try reading char by char as in your original example, try adding a flush each time:

sys.stdout.write(char)
sys.stdout.flush()

Flushing the stdout every time you write is the manual equivalent of disabling buffering for the python process: python.exe -u <script> or setting the env variable PYTHONUNBUFFERED=1

Community
  • 1
  • 1
jdi
  • 90,542
  • 19
  • 167
  • 203
  • I create a pipe for stdout because of a bug when I try to run the script if there's no console window but an IDLE GUI window. – rynd Apr 14 '12 at 02:23
  • @rynd: Ok well then add it back by all means, if something is reading it. – jdi Apr 14 '12 at 02:24
  • The problem remains. If I look at the CPU load of the subprocess, I see sox is working with 90% load while my script still has a delay. Why does read() not work at this point? Then the first output (information not related to the progress) is output slowly. – rynd Apr 14 '12 at 02:37
  • @rynd: Did the flush not help with your write() ? – jdi Apr 14 '12 at 02:58
  • I think flush() on sys.stdout did a little bit, but there are still points where it twiddles its thumbs although sox already has generated output. "Oops, sox already has 25%, I should hurry." – rynd Apr 14 '12 at 03:06
  • Not sure what to tell ya beyond this point because you would be unbuffered. My experience is that it should be live. – jdi Apr 14 '12 at 03:08
  • OK, thanks! I'll try Python 3, suspending the process shortly and some other things. – rynd Apr 14 '12 at 03:20
  • With Python 3 it's the same and suspending is an unclear approach. I've just tested adding "2> stderr" to the called command, i.e. redirecting stderr to a file called "stderr", and then reading the file. It's the same: alternating slow and fast output. Unbelievable! Seems that Windows doesn't like SoX. – rynd Apr 14 '12 at 15:48
  • 1
    @rynd: Maybe windows specific. You might try doing some searching on subprocess buffered output and windows. – jdi Apr 14 '12 at 15:57