Correct way in Python to use subprocess.run when PIPE is in the command

Question

I am trying to write a python script that executes the following terminal command:

echo -n | openssl s_client -connect {host}:{port} | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > {host}_{port}.cert

If I try to break up the command into arguments to pass to the subprocess.run it does not work (something is run but it does not store the certificate as I would like it to.

Using the below sytax correctly executes the command, however I fear it is not best practice and wanted to understand the correct way for how this should be done:

store_certificate_command = f"echo -n | openssl s_client -connect {host}:{port} | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > {host}_{port}.cert"

subprocess.run(store_certificate_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

Is there some reason you need to use `echo` and `sed`? Your `echo` is equivalent to passing `subprocess.DEVNULL` as the `stdin`. Your `sed` command is trivially implementable in Python, which can write the results to the file itself. All you really need to run is the `openssl` command. — ShadowRanger, Apr 07 '21 at 01:16
The pipes are handled by the shell therefore you must either do it as shown by you or you would have to do the "plumbing" yourself, call the different commands in command line separately and feed the output of one command as input into the next. — Michael Butscher, Apr 07 '21 at 01:17
@ShadowRanger I am actually just following someone else's solution for how to download a certificate. I agree that I can use python instead for parts of the command that require pipe (honestly can achieve the entire result with just python) but I would still like to understand the best practice for running commands that require pipe. — AnotherCourier, Apr 07 '21 at 01:22
@MichaelButscher I did try the second solutio in this link: https://thomas-cokelaer.info/blog/2020/11/how-to-use-subprocess-with-pipes/ (feeding p1 into p2, p2 into p3, etc) but got the error at the bottom. — AnotherCourier, Apr 07 '21 at 01:24
Don't use `run` if you are doing your own plumbing. The object it returns has an attribute `stdout` which is however not a filehandle. — tripleee, Apr 07 '21 at 01:32
@AnotherCourier: That link tells you why the error happens; only one command in the pipeline can be run with `.run`, all the rest (that feed data to that command or read from it) must be run with `Popen` so they run in parallel with the synchronous `.run` command (which must come *after* all the other commands have been launched with `Popen`). — ShadowRanger, Apr 07 '21 at 01:33

score 0 · Accepted Answer · answered Apr 07 '21 at 01:31

The correct way to do this is to limit yourself to the minimum amount of non-Python executables. In this case, echo isn't necessary, Python can do the work sed is doing, and can write the resulting file as well. A clean solution would be something like:

import subprocess

with open(f'{host}_{port}.cert', 'wb') as outf,\
     subprocess.Popen(['openssl', 's_client', '-connect', f'{host}:{port}'], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as proc:
    for line in proc.stdout:
        if b'-BEGIN CERTIFICATE-' in line:
            outf.write(line)
            break
    else:
        raise ValueError("BEGIN CERTIFICATE not found in output")

    for line in proc.stdout:
        outf.write(line)
        if b'-END CERTIFICATE-' in line:
            break
    else:
        raise ValueError("END CERTIFICATE not found in output")

I switched to subprocess.Popen instead since it allows you to process the output in a streaming fashion (the same way the shell pipes would work), but given the relatively small output, subprocess.run would likely work just fine too. The ValueErrors aren't strictly necessary, but I like having them there so you fail hard when the output isn't what you expect.

Correct way in Python to use subprocess.run when PIPE is in the command

1 Answers1