1

I would like to write a program that can take input piped to it, manipulate it, and output it immediately. It seems that no matter what various things I try (using stdout.write rather than print, explicitly flushing stdout, setting stdin to be unbuffered), the output seems to be delayed until the input program is closed.

Example input script:

import sys
import time

sys.stdout.write('1\n')
sys.stdout.flush()

time.sleep(3)

sys.stdout.write('2\n')
sys.stdout.flush()

Example output script:

import sys
import time

for line in sys.stdin:

    print time.time()

    sys.stdout.write(line)
    sys.stdout.flush()

If I run this:

python input.py | python output.py

I get something like this:

1428023794.99
1
1428023794.99
2

Clearly the lines are making it to the second program at the same time. I've also tried running the second python script with the -u flag to unbuffer stdin, but get the same output.

How can I make my program pulling the text from stdin output the text immediately?

Community
  • 1
  • 1
Dan
  • 1,925
  • 3
  • 22
  • 28

1 Answers1

0

I'm piping out crontab errors to Slack channels using the following code:

import os
import sys

def get_stdin():
  #
  # This will get stdin piped input *if exists*
  #

  with os.fdopen(sys.stdin.fileno(), 'rb', buffering=0) as stdin:
    if not stdin.seekable():
      return [_.decode('utf-8') for _ in stdin.readlines()]

stdin = get_stdin()
print("Found content:", stdin)

The biggest challenge I had here, was:

  • If there's stdin piped content - code worked just fine
  • If there's nothing passed to the script - script hang and wait for input + (CTRL+D)

That was solved by checking stdin.seekable() if there's any content pending to be read or not ....

Tested on both:

$ cat file.txt | slack.py   # stdin=True
$ ./slack.py --help         # stdin=False
Ricky Levi
  • 7,298
  • 1
  • 57
  • 65
  • "That was solved by checking stdin.seekable() if there's any content pending to be read or not" - `seekable` has nothing to do with whether there's data available to read. – user2357112 Apr 20 '23 at 10:17
  • Check and see ... there is. lowering the only answer in 8 years - did you even tried it ? I can guess it's because that when you have data pending - it initialize stdin with `IOTextWrapper` and if you don't have content then it waits for input and hangs until CTRL+D - it initialize it with `IOBufferWrapper` .. this is why `seekable` works on one but not the other if you have different suggestion - please specify, i couldn't find such, including the comments in the original question. – Ricky Levi Apr 20 '23 at 11:21
  • There is no such thing as IOBufferWrapper or IOTextWrapper. `io.TextIOWrapper` and `io.BufferedReader` exist, but the way `sys.stdin` is initialized has nothing to do with whether data is available. Also, I *did* try this, and it completely failed - `./slack.py --help` hangs, because `sys.stdin` is reading from terminal input, which is unseekable. (Also you forgot the shebang line, which I had to fix.) – user2357112 Apr 20 '23 at 19:15
  • If you check the [docs](https://docs.python.org/3/library/io.html#io.IOBase.seekable), you'll find that `seekable` is documented as "Return True if the stream supports random access. If False, seek(), tell() and truncate() will raise OSError." It's a check for whether you're reading from something you can freely jump ahead or back in, like a file, or something you can't do that with, like a pipe or a tty. – user2357112 Apr 20 '23 at 19:16
  • @user2357112 I got the function - from the docs. run every function i saw there. and this the pipe is converted to a "file"-like object (`os.fdopen` ) this is why `seekable` works. if you can't jump - there's nothing there. for me, piping works just fine in both cases ( when we have content and when we don't (use just args) ) and i've setup my cronjob to pass all unexpected content to my Slack channel this way - if someone else wants to try it out - great. – Ricky Levi Apr 22 '23 at 13:21
  • @user2357112 In regards to `There is no such thing as IOBufferWrapper or IOTextWrapper` your explanation that comes afterwards is exactly what I explained in the answer. the `readlines()` that we run cause the script to hang. so how in the code you can *check* if you have pending content or not ? this is what i was able to overcome - do you have another option or not ?! that was my first question. and I didn't forgot the "shabang" - I provided a code snippet, i had no intention of adding it as it's not important, so there's nothing to "fix" – Ricky Levi Apr 22 '23 at 13:43
  • 1
    Some people ... god, how I hate SO sometimes. Thanks for the answer @RickyLevi, this gave me a decent pointer – Fynn May 08 '23 at 17:25