shell: pipe file to python stdin and also display to screen

Question

Hello! this is my first question on StackOverflow please guide me if I do something incorrectly.

so I have a python script and I want to make a shell script to pipe text file to the python when run.

the problem is the text from file piping to python is not display on the screen.

here is the code that I tried:

python3 index.py < input.txt

index.py

while True:
    x = input("Input: ")
    print("Input ->", x)

input.txt:

Test1
Test2
Test3

and after I try to run the command the result is:

Input: Input -> Test1
Input: Input -> Test2
Input: Input -> Test3
Input: Input ->
Input: Traceback (most recent call last):
  File "index.py", line 2, in <module>
    x = input("Input: ")
EOFError: EOF when reading a line

(ignore the error)

the text from piping not display. more than that it does not even display the new line.

sadly, that is not the result that I want :(

here is the result that I expected:

Input: Test1
Input -> Test1
Input: Test2
Input -> Test2
Input: Test3
Input -> Test3
...

I already try the following commands:

cat input.txt | python3 index.py

the result just like above.

here is another way that I tried:

cat input.txt | tee /dev/tty | python3 index.py

the result is still not what I expected:

Test1
Test2
Test3

Input: Input -> Test1
Input: Input -> Test2
Input: Input -> Test3

I also take a look at the screen command by start a detached screen session and send the text via -X stuff "Test1^M" but the problem is I don't want to make a shell script something like looping text line and using screen command to send it. (I want python to read stdin as fast as possible)

is this possible with pure shell script?

thank you for your further answer.

Answer 1

The way you're calling the Python-Script by piping the file-output to it won't work because you're not handling any parsed arguments in the script.

Take a look at:

https://docs.python.org/3/library/argparse.html

What's the best way to parse command line arguments?

Answer 2

here is the result that I expected:

It is generally inherently not possible to do generically. Generally, to do that, a mean of 1-way synchronization is required between processes, to detect when the underlying process has stopped processing the current chunk of data and communicate the fact to the process feeding the data. No such synchronization in | piping is there.

In this case, you could potentially assume, that the underlying process stops processing the data after writing a newline. In that case, you could send one line of data to the process and receive one line of data from the process in a synchronized manner.

Or use the worst possible synchronization manner - sleep. Send one of data and sleep for 1 second, giving the process ridiculously more then enough time to process the current packet.

is this possible with pure shell script?

Yes. You might want to interest yourself in bash coproc-ceses, interprocess synchronization, pipes and file descriptors in shell, and buffering modes (not-buffered, line-buffered and full-buffered). For example:

coproc python3 index.py
exec 10<&${COPROC[0]}
exec 11>&${COPROC[1]}
while IFS= read -r input; do
   printf "Sending: %s\n" "$input"
   # sending excatly one line of input
   printf "%s\n" "$input" >&11
   # synchronize input with ouput by waiting for one line
   if ! IFS= read -t 1 -r output <&10; then
       echo "The process failed to answer in one second"
   fi
   printf "Process answered: %s\n" "$output"
done <<<$'Test1\nTest2\nTest3'
exec 11>&-  # close input
cat <&10    # flush whatever else there is
exec 10<&-

Sends one by line and expectes exactly one line of output from the process before sending the next line. Overall, "asynchronous" is a very hard concept to grasp.

Note that full buffering of output on the python3 side will break the synchronization.