1

I have some python code which I would like to open as a subprocess. I have tried threading the process through subprocess.Popen() and also through asyncio from another Python script. I know that the addition of time.sleep() is preventing the process output from being read as the program produces output, but a time consuming large loop has the same effect. Interestingly enough, anything I try works from Eclipse with PyDev, but if I run my program from the shell it only outputs when the program is finally finished running. I need to be able to read output as the program outputs to stdout as it progresses. Any help would be greatly appreciated. 

# would like to call this python script as a subprocess from another program.

import time
import sys

for j in range( 50000000000 ):
    print("blah") # if this is removed asynchronous reading occurs just fine
    time.sleep(1)
    #for k in range( 10000000 ): # this would also cause output to wait until program is complete
    #    b=2
    print("\n"*2)
Udi
  • 29,222
  • 9
  • 96
  • 129
csandy83
  • 31
  • 9
  • Does this help: https://stackoverflow.com/q/107705/57952 ? – Udi Jun 24 '17 at 15:54
  • Yes this works. I used the "unbuff" class and I had complications when using Popen from the caller and using stdout= subp.PIPE as an arg to Popen when I tried to do proc.stdout.readline(), so I went with os.environ['PYTHONUNBUFFERED] = 1. I don't know which to prefer. I kind of liked using the Unburffered class and changing the stream because it did not affect the entire Python environment. Any idea how to get this to work by setting stdout to PIPE and proc.stdout.readline() still working? Is there a drawback to setting the OS var, like messing up the way stuff gets normally printed? – csandy83 Jun 24 '17 at 19:49

1 Answers1

1

Try disabling python's output buffering, for example by running python with the -u flag.

For more info see: Disable output buffering

Udi
  • 29,222
  • 9
  • 96
  • 129
  • I can use the "-u" flag and it works. I will probably do this. The only issue is that I have to over-write the shebang of the external process I'm doing to accomplish this. In other works if it's #!/usr/bin/anaconda/python, I would be over-writing that. Ideally, I would be able to do it programmatically. Ideally as the client of generic scripts I don't want to ask the authors to include the option -u. I know I'm probably being nitpicky. – csandy83 Jun 24 '17 at 20:50