0

I'm struggling to understand why this fails with a wget: missing URL error:

import shlex
import subprocess

copy_command = "wget -O - 'http://example.com/somepath/somefile.txt?someparam=test' | sshpass -p pass ssh user@localhost -p 2222 \"cat - > /upload/somefile.txt\""

cmd = shlex.split(copy_command, posix=False)

with subprocess.Popen(
    cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True
) as proc:
    output, error = proc.communicate()

What am I missing here? If I just give subprocess the copy_command string directly, then it works without issues.

L42
  • 3,052
  • 4
  • 28
  • 49
  • Wait, you can't simulate a whole pipeline with redirection with a single call to Popen. You need multiple calls, one for each process. – Keith Mar 12 '21 at 07:51
  • Thanks for commenting! Note that this works as I expect when I don't use shlex. – L42 Mar 12 '21 at 07:54
  • Because `shell=True` passes it as a string to a shell with `sh -c '...'`. So the subshell then interprets the pipeline. – Keith Mar 12 '21 at 07:55
  • BTW, I'm not quite sure what you are trying to do there, but I'm sure there's a better way. – Keith Mar 12 '21 at 07:56
  • assuming you're on Linux. – Keith Mar 12 '21 at 07:57
  • Thanks Keith! The X of my Y is that I'm trying to pipe data from an S3 bucket to a server that I can connect to using ssh. I can't afford to temporarily store the file on the machine that will run this script. The code in my question works (except the shlex part) exactly as intended, but yes: Perhaps there's a better way? – L42 Mar 12 '21 at 07:59
  • With a bit more Python you can use something like httplib and [ssh2-python3](https://github.com/pycopia/ssh2-python3). Actually, with S3 you should use boto3 API. – Keith Mar 12 '21 at 08:03
  • Thanks for the tip! I'll consider that. Could you answer this question explaining why the use of piping won't work with a single subprocess? Then I can mark that as an answer. I found https://stackoverflow.com/a/13332300/1539208 now also. – L42 Mar 12 '21 at 08:07

1 Answers1

1

To set up a pipeline requires the parent process to spawn all the programs involved and the connect (pipe) the stdio of one to another.

The Python documentation for subprocess explains how to do this.

It works with string argument andshell=True because then it just hands off the command line to a sub shell, and that shell handles all those details.

Keith
  • 42,110
  • 11
  • 57
  • 76