0

Hi I have spent a few days reading at stackoverflow for an answer why I get the output listed below instead of seeing the loop in hello.py count forever. Still unclear for me and nothing works. How to start a few scripts in POpen subprocesses and get their output continously. Not when they have finished, because they will run forever. In case this is considered already answered, I'd be glad if someone can link to something functional with Python3.

I run this .py code in the hope to see hello.py execute:

import subprocess
import sys

command = [sys.executable, 'hello.py']
process_handle = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

while process_handle.poll() is None:
    for line in iter(process_handle.stdout.readline, ""):
        output = line.decode('utf-8')
        if output != '':
            print('Last number is: ' + output)

Here is what I get. No running of hello.py:

Python 3.7.3 (default, Apr 3 2019, 05:39:12) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

hello.py contains:

i = 0
while True:
    print(i)
    i=i+1
    sleep(1)

I tried the first variant from the second duplicate suggestion from live output from subprocess command

It does not compile/execute:

import subprocess
import sys
with open('test.log', 'w') as f:  # replace 'w' with 'wb' for Python 3
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    for c in iter(lambda: process.stdout.read(1), ''):  # replace '' with b'' for Python 3
        sys.stdout.write(c)
        f.write(c)

Result:

Traceback (most recent call last): File "hello.py", line 11, in Traceback (most recent call last): File "hello5.py", line 7, in sys.stdout.flush() NameError: name 'sys' is not defined sys.stdout.write(c) TypeError: write() argument must be str, not bytes

Now I tried the second variant from above, modified like this:

import subprocess
import sys

command = [sys.executable, 'hello.py']

process = subprocess.Popen(command, stdout=subprocess.PIPE)
for line in iter(process.stdout.readline, b''):  # replace '' with b'' for Python 3
    sys.stdout.write(line)

Result:

Traceback (most recent call last): File "hello.py", line 11, in Traceback (most recent call last): File "hello5.py", line 8, in sys.stdout.flush() NameError: name 'sys' is not defined sys.stdout.write(line) TypeError: write() argument must be str, not bytes

After advice from ShadowRanger I added b'' for binary, sys was already imported however:

import subprocess
import sys

command = [sys.executable, 'hello.py']

process = subprocess.Popen(command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), b''):  # replace '' with b'' for Python 3
sys.stdout.write(c)

Result, despite adding import sys, sys is not defined? Plus now it complains about binary and wants string.

Traceback (most recent call last): File "hello.py", line 11, in sys.stdout.flush() NameError: name 'sys' is not defined Traceback (most recent call last): File "hello5.py", line 8, in sys.stdout.write(c) TypeError: write() argument must be str, not bytes

OK, I went with ShadowRanger's info and converted the binary output to text again. hello.py is now printing text. 

sys.stdout.write(c.decode('ASCII'))

BUT its coming all at once and thats not what I need. Every printed line in hello.py must show up in realtime and not just when the script finishes.

Rimfire
  • 378
  • 1
  • 3
  • 12
  • Possible duplicate of [real time subprocess.Popen via stdout and PIPE](https://stackoverflow.com/questions/2082850/real-time-subprocess-popen-via-stdout-and-pipe) – Chris Aug 13 '19 at 02:13
  • Possible duplicate of [live output from subprocess command](https://stackoverflow.com/questions/18421757/live-output-from-subprocess-command) – Hans Musgrave Aug 13 '19 at 02:18
  • @Chris Thanks for the suggestion. The accepted answer in that thread was to add sys.stdout.flush() to prints. I added that to all my print()'s but unfortunately there is still no result. – Rimfire Aug 13 '19 at 02:21
  • 1
    FYI, the second argument to all of your two-arg `iter` calls is wrong; you're passing `""`, the empty `str`, but your `Popen` calls are all defaulting binary mode, so you should be using `b""`. Your final traceback is just because you used `sys.stdout` without an `import sys` in your `hello5.py` script, and/or you failed to use `sys.stdout.buffer.write(c)` (because again, you have `bytes`, and you were trying to write to text-based `sys.stdout`). – ShadowRanger Aug 13 '19 at 02:36
  • @ShadowRanger sys is among imports. Why does it still complain? I added b"" but then it complains about "TypeError: write() argument must be str, not bytes" – Rimfire Aug 13 '19 at 03:02
  • 1
    @Rimfire: It's in the imports of one of your scripts, but not the one being run by `Popen`. And the error on `write` is because `read`/`readline` still *returns* `bytes`. You need to learn how `bytes` and `str` work in Python 3 first; if you don't understand that, and just copy and paste code written for Python 2 using `subprocess`, you're going to have errors on top of errors. There are multiple possible fixes, but if you don't have the background to understand the problem, I'm not helping by pointing things out piecemeal. – ShadowRanger Aug 13 '19 at 03:05
  • @ShadowRanger You were right there, sys was imported to hello5, but not the hello.py. I have been programming C/C++/Delphi since early 90's and sometimes I just want to whip something together in other languages. Usually works, after some pain. :-) But seriously, how to get the text out if its binary now? – Rimfire Aug 13 '19 at 03:15
  • I was not supposed to do this, but use Python 4 Delphi component in a Lazarus environment. Alas a bug in the component made me go this path. – Rimfire Aug 13 '19 at 03:22

1 Answers1

1

Got it working, thanks for all help. In order to get output in realtime instead of all of it at the end of script, flushing of stdout must be done in the subprocess script (hello.py in this case) Also, like ShadowRanger said, if b'' for binary is used there will be a complaint from stdout.write. That can be handled by decoding to text.

hello.py

import sys
from time import sleep

i = 0
while True:
    print(i)
    sys.stdout.flush()
    i=i+1
    sleep(0.1)

mainscript.py

import sys
import subprocess
command = [sys.executable, 'hello.py']
process = subprocess.Popen(command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), b''):  # replace '' with b'' for Python 3
    sys.stdout.write(c.decode('utf-8'))  # decode b'' back to text
Rimfire
  • 378
  • 1
  • 3
  • 12