2

I am using Python to call a Shell script with 

def run_command(cmd):
    print "Start to run: " + cmd
    run = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    while True:
        line = run.stdout.readline().decode()[:-1]
        if line == '' and run.poll() is not None:
            break
        print line # print the log from shell
    recode = run.returncode
    if recode != 0:
        raise Exception("Error occurs!")

    print "End to run: " + cmd

Then I run

run_command("sh /home/tome/a.sh")

I notice the console output from a.sh is not in real time, looks like that there is a buffer for the stdout and when the stdout buffer is full, then the output is printed out.

I would ask how to disable the shell stdout buffer in my script a.sh

Thanks!

Tom
  • 5,848
  • 12
  • 44
  • 104

1 Answers1

3

The buffering in question would largely be a problem on the script's side, not the Python side; while Python would buffer the reads, it wouldn't block unless the buffer was emptied and there was nothing available to read.

So really, you need to disable buffering in the script itself. Adding stdbuf -oL (or -o0 for completely unbuffered, but line buffering should cover you since you read by line as well) to your commands should help in some cases (where the programs don't adjust their own buffering internally).

If you're seeing this behavior only by looking at Python's output, be aware that Python itself can buffer output as well. You can disable this by passing -u when running Python, or setting the environment variable PYTHONUNBUFFERED=1 before running it, or from within a script, you can manually call sys.stdout.flush() after any writes (direct, or implicit via print) to stdout. On modern Python, print takes an argument to force a flush after printing, but since you're on Python 2.x, that's not an option.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • Thanks @shadowranger for the helpful answer, let me try. – Tom Sep 07 '19 at 01:06
  • That's too bad.. When I set the command as `stdbuf -o0 sh abc.sh`, the output is still buffered. Not sure where the problem is. – Tom Sep 07 '19 at 03:01
  • @Tom: Whether `stdbuf` works depends on the actual programs being run; if they adjust their own buffering internally, at runtime, then `stdbuf`'s changes get undone. If they don't use standard streams, the changes don't affect their I/O at all. The `stdbuf` man page has a note on this, mentioning `tee`, `dd` and `cat` as being specific offenders, but those are just limited examples. – ShadowRanger Sep 07 '19 at 03:18
  • Thanks @hadowranger, When I run the python(which is calling shell) directy, it will output in real time, but when I run it in our scheduler system, then the output is buffered, I will investigite more to find out how to make it work. – Tom Sep 07 '19 at 03:21
  • When I am running python in interactive mode, then the buffer is disabled, but when I run python in non-interactive mode, then the buffer is not disabled, Even I have used `unbuffer sh run.sh a b c`,it also doesn't take effect. Not sure how to fix now. – Tom Sep 07 '19 at 05:04
  • @Tom: When you run with `shell=True`, there is going to be another layer of shell around you that won't be unbuffered. Perhaps remove `shell=True`, and split your command manually (e.g. [with shlex.split](https://docs.python.org/3/library/shlex.html#shlex.split)), so you can run `Popen(['stdbuf', '-o0'] + shlex.split(cmd), ...)` and avoid additional shell wrapping that might interfere with buffering. You might also want to test setting the environment variable `export PYTHONUNBUFFERED=1` just to ensure Python's own buffering isn't screwing with you; you might be seeing delayed Python output. – ShadowRanger Sep 11 '19 at 15:36
  • @Tom: I realized it's likely you're seeing this by looking at Python's own output, in which case you may need to ensure Python either isn't buffering or explicitly flushes as needed. I've edited the answer to include that info. – ShadowRanger Sep 11 '19 at 15:41
  • Thanks @shadowranger for the answer, it works, thanks very much! – Tom Sep 12 '19 at 01:10