3

Without pointing out ten different libraries to do this a different way, can anyone tell me why python restarts my script when STDOUT is redirected on the command line? (running on CentOS 7)

#!/usr/bin/python
import os
print("Top of the script")
pid = os.fork()
print("Pid: " + str(os.getpid()))

Regular output is:

$ ./forktst2.py
Top of the script
Pid: 1919
Pid: 1920

But when I pipe it to anything it becomes:

$ ./forktst2.py | cat
Top of the script
Pid: 2151
Top of the script
Pid: 2153

Which is doubly confusing because if it's rerunning the first print() then why isn't it also rerunning os.fork()?

Thanks!

Frank
  • 31
  • 3
  • FYI: You can "fix" this (not that it is broken) by changing the first `print` to `print("Top of the script", flush=True)` (at least on Mac OS X, that is). – Warren Weckesser Feb 26 '18 at 23:59
  • And based on that, I guess that the reason for the repeated output in the second case is that the output is buffered, and the buffer is duplicated by the call to `fork()` before the output has actually been written to the output file. – Warren Weckesser Feb 27 '18 at 00:07
  • Also relevant: [Is stdout line buffered, unbuffered or indeterminate by default?](https://stackoverflow.com/q/3723795/364696) – ShadowRanger Feb 27 '18 at 00:08
  • The original script was meant to grab 20 web pages with a max of 5 parallel processes. I wrote the initial test with having the children sleep for a randomly chosen 1, 2, or 3 seconds. Everything was fine until I tried to grep the output. At that point the parent started printing as if it was launching 2, then 3, then 4, then 5, then 6, then 7.... 19 (124 total) but it doesn't actually launch the extras. I'm more confused than ever. I made Perl and PHP versions of the above script and they do what I would expect. The C version does what Python does tho! – Frank Feb 28 '18 at 19:14
  • Now in my 10th test script I'm forcing `sys.stdout.flush()` after every print and it's finally behaving. So Python is buffering and re-`write()`ing ALL of its output despite `write()` returning. I rewrote my C version with a `fflush(stdout);` after every `printf()`. Absolutely maddening. – Frank Feb 28 '18 at 19:41
  • I've dealt with output being _delayed_ due to buffering but this is the first time in 25 years of programming that I've had output _repeated_ due to buffering. – Frank Feb 28 '18 at 20:38

0 Answers0