Read line from shell pipe, pass to exec, and keep to variable

Question

I'm trying to pass the data in the pipe through the python script and write this data into a separate variable for further processing

The main purpose of the script is to be a layer between pipes, to record parsing errors and what leads to them.

Application:

echo '{"user": "Basic dGVzdDp0ZXN0"}' | script.py | rotatelogs {....}

I made a script:

cmd = ["/usr/bin/jq -c \'.user |= if test(\"^[Bb]asic \") then .[6:] | @base64d | gsub (\":.*$\"; \"\")  else . end \'"]

with open('/dev/stdin') as f: 
        try:
            subprocess.run(cmd, check = True, shell=True)
        except subprocess.CalledProcessError:
            with open('/path/to/parseerror.log', 'w') as pfile:
                pfile.write(f.read())        

Command in subprocess.run executes successfully and produces its output, but f.read() become empty If I move the reading of the variable f.read() to execution subprocess.run , then I will get the value of the variable, but the command subprocess.run in will not be executed (get null pipe input).

with open('/dev/stdin') as f: 
        line=(f.read())
        try:
            subprocess.run(cmd, check = True, shell=True)
        except subprocess.CalledProcessError:
                ....

How can I combine the execution of the command with the parameters from the pipe and record the incoming pipe itself? The main goal is to pass the command execution through the script and write the received pipe parameters to a separate file

Answer 1

Your approach reads until EOF, which is what read() does, leaving no input left for the subprocess to read. Try readline() instead.

Opening a second stdin also seems weird. I'd imagine you want something like

import sys
...
print(sys.stdin.readline(), end='')
subprocess.run(cmd, stdin=sys.stdin, shell=True, check=True)

Tangentially, you can probably get rid of shell=True if you only have a single command you want to run.

import shlex
...
subprocess.run(shlex.split(cmd), stdin=sys.stdin, check=True)

The stdin=sys.stdin is not strictly speaking crucial (the subprocess will inherit the standard input of its parent process, and consume as much or as little of it as it likes to), but it documents your intent.

Weirdly, even with a single readline() I get the same symptoms you describe. I can work around it with an explicit read():

subprocess.run(shlex.split(cmd), input=sys.stdin.read(), text=True, check=True)

... but this is obviously unattractive if you need to handle potentially large amounts of input.

The opposite case does not work; in the general case, the command can read all the available input up to EOF (though some commands, like head, will not) and then there will be nothing left for your Python script to read after the subprocess finishes.

Answer 2

You are making this more difficult than necessary. In order to read from a pipe such as echo "some data with spaces" | script.py, you can just read from stdin with any standard input method.

Similarly, to pass data to another process, just write to stdout with any standard output method. The OS will take care of the redirection for you.

For example:

# script.py
for line in input():
    print(line)

Now do

echo "my first line\nmy second line" | python script.py

And you will get this output:

my first line
my second line