2

Running a long and time consuming number crunching process in the shell with a Python script. In the script, to indicate progress, I have inserted occassional print commands like 

#!/usr/bin/env python3
#encoding:utf-8
print('Stage 1 completed')

Triggering the script in the shell by 

user@hostname:~/WorkingDirectory$chmod 744 myscript.py && nohup ./myscript.py&

It redirects the output to nohup.out, but I cannot see the output until the entire script is done, probably because of stdout buffering. So in this scenario, how do I somehow adjust the buffering parameters to check the progress periodically? Basically, I want zero buffering, so that as soon a print command is issued in the python script, it will appear on nohup.out. Is that possible?

I know it is a rookie question and in addition to the exact solution, any easy to follow reference to the relevant material (which will help me master the buffering aspects of shell without getting into deeper Kernel or hardware level) will be greatly appreciated too.

If it is important, I am using #54~16.04.1-Ubuntu on x86_64

Della
  • 1,264
  • 2
  • 15
  • 32
  • 1
    Try force `stdout`,`stderr` to be totally unbuffered: `nohup python -u ./myscript.py &` – ERemarque Sep 17 '19 at 08:14
  • Please let me know, if it will helpfull. – ERemarque Sep 17 '19 at 08:31
  • Thanks. It worked. So the buffering is done by the interpreter (python in this case) rather than shell, which invokes the script? What if I have an command line executable compiled from a C++ code? – Della Sep 17 '19 at 09:26
  • 1) Yes,buffering is done by the interpreter, so we use key `-u` to override it. See `man python` and `python --help` for more information. Also I think you can just use `#!/usr/bin/python -u` in you script and run in shell `nohup ./myscript.py &` 2) Unfortunately, I don't know about `C++` code. – ERemarque Sep 17 '19 at 10:36

1 Answers1

1

Python is optimised for reading in and printing out lots of data.
So standard input and output of the Python interpreter are buffered by default.

We can override this behavior some ways:

  1. use interpretator python with option -u.

From man python:

       -u     Force stdin, stdout and stderr to be totally unbuffered.  On systems where it matters, also put stdin,  stdout  and  stderr  in
              binary  mode.   Note  that  there  is  internal  buffering in xreadlines(), readlines() and file-object iterators ("for line in
              sys.stdin") which is not influenced by this option.  To work around this, you will want to use "sys.stdin.readline()" inside  a
              "while 1:" loop.

Run script in shell:

nohup python -u ./myscript.py&

Or modify shebang line of script to #!/usr/bin/python -u and then run: 

nohup ./myscript.py&
  1. use shell command stdbuf for turn off buffering stream

See man stdbuf.

Set unbuffered stream for output:

stdbuf --output=0 nohup ./myscript.py&

Set unbuffered stream for output and errors:

stdbuf -o0 -e0 nohup ./myscript.py&
ERemarque
  • 497
  • 3
  • 16
  • Tried the shebang `#!/usr/bin/env python3 -u`, but it is throwing an error saying no such file or directory. – Della Sep 18 '19 at 05:57
  • 1
    @Della There is no simple way to pass arguments to `python` in shebang line where used `env` . Please, see https://stackoverflow.com/questions/3306518/cannot-pass-an-argument-to-python-with-usr-bin-env-python But I would to suggest additional way. Use shell utility `stdbuf` for turn off bufferisation, in this case you don't require modify your scripts or use interpretator python in command: `stdbuf --output=0 nohup ./myscript.py&`. See `man stdbuf`. It works for me. – ERemarque Sep 18 '19 at 08:23
  • 2
    or without error stream buffering : `stdbuf -o0 -e0 nohup ./myscript.py&` – ERemarque Sep 18 '19 at 08:48