Why does passing variables to subprocess.Popen not work despite passing a list of arguments?

Question

I have a script which calls another Python script by subprocess.Popen. But since I have arguments stored in variable(s)

servers[server]['address']
servers[server]['port']
servers[server]['pass']

I am unable to perform the command

p = subprocess.Popen(
    ["python mytool.py -a ", servers[server]['address'], "-x", servers[server]['port'], "-p", servers[server]['pass'], "some additional command"],
    shell=True,
    stdout=subprocess.PIPE
)

Use the variables to build a string that is the command, or pass them as a list of arguments. — beroe, Nov 22 '13 at 08:25
Calling Python as a subprocess of Python is an antipattern unto itself; a better solution is usually to refactor the script from the subprocess so you can `import` it and call it directly from the parent script. There _are_ situations where you genuinely want a subprocess (for example, if the script uses signals which you need to handle differently in the parent) but more often than not, you probably shouldn't. — tripleee, Oct 12 '21 at 10:29
As a further aside, as already pointed out in the `subprocess` documentation, you should avoid `Popen` if your use case is already handled by one of the higher-level functions `subprocess.run()` and friends. Basically, unless you require parallel processing or interaction with the running process, don't use `Popen` (and if you do, take care of waiting for the child process etc which `Popen` does not do for you). — tripleee, Oct 12 '21 at 10:45

score 14 · Accepted Answer · edited May 23 '17 at 12:24

Drop shell=True. The arguments to Popen() are treated differently on Unix if shell=True:

import sys
from subprocess import Popen, PIPE

# populate list of arguments
args = ["mytool.py"]
for opt, optname in zip("-a -x -p".split(), "address port pass".split()):
    args.extend([opt, str(servers[server][optname])])
args.extend("some additional command".split())

# run script
p = Popen([sys.executable or 'python'] + args, stdout=PIPE)
# use p.stdout here...
p.stdout.close()
p.wait()

Note that passing shell=True for commands with external input is a security hazard, as described by a warning in the docs.

babbageclunk · Answer 2 · 2013-11-22T08:48:07.900

12

When you call subprocess.Popen you can pass either a string or a list for the command to be run. If you pass a list, the items should be split in a particular way.

In your case, you need to split it something like this:

command = ["python",  "mytool.py", "-a", servers[server]['address'], 
           "-x", servers[server]['port'], 
           "-p", servers[server]['pass'], 
           "some",  "additional", "command"]
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

This is because if you pass in a list, Popen assumes you have already split the command line into words (the values that would end up in sys.argv), so it doesn't need to.

The way you're calling it, it will try to run a binary called "python mytool.py -a", which isn't what you meant.

The other way to fix it is to join all of the words into a string (which Popen will then split up - see subprocess.list2cmdline). But you're better off using the list version if possible - it gives simpler control of how the commandline is split up (if arguments have spaces or quotes in them, for example) without having to mess around with quoting quote characters.

edited Nov 22 '13 at 08:48

answered Nov 22 '13 at 08:29

babbageclunk

8,523
1
33
37

The variable I use to store the output, but on print it's empty (which I guess hasn't been executed). – GaNi Nov 22 '13 at 08:49
I think you mean that when you read from p.stdout, there's no output? That will be because the command is not being run. – babbageclunk Nov 22 '13 at 09:36
Actually, the shell=True is probably muddying the water here - unless you're using globbing (to expand out a list of files, say), it's best to turn it off. – babbageclunk Nov 22 '13 at 13:32
without shell=true, the script refuses to execute, it shows "sh: mytool.sh command not found" – GaNi Nov 23 '13 at 05:28
Put in the full path to mytool.sh - whatever "which mytool.sh" gives when you type it on the commandline. – babbageclunk Nov 26 '13 at 09:49

Michael Kazarian · Answer 3 · 2019-01-30T13:50:38.300

5

Your problem in type str for first Popen argument. Replace it to list. Below code can work:

address = servers[server]['address']
port = servers[server]['port']
pass = servers[server]['pass']

command = "python mytool.py -a %s -x %d -p %s some additional command" % (address, port, pass)
p = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
#        it is a list^^^^^^^^^^^^^^^  shell=False

If command arguments get from a trusted source you can construct command and use it with shell=True to such manner:

import pipes as p
command = "python mytool.py -a {} -x {} -p {} some additional command".format(p.quote(address), p.quote(port), p.quote(pass))
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

Note 1: constructed command with shell=Trueis potentially insecure. Use pipes.quote() for reduce injection possibility.
Note 2: pipes.quote() deprecated since python2; for python3 use shlex module.

edited Jan 30 '19 at 13:50

answered Nov 22 '13 at 08:50

Michael Kazarian

4,376
1
21
25

Had the similar issue as before, but I resolved it now. Thanks :) – GaNi Nov 22 '13 at 10:05
1

If you pass a list with `shell=True`, the first argument is treated as a shell script, and subsequent arguments as parameters for that script (its `$0`, `$1`, etc). The results of this `shlex.split()` operation isn't a list intended to be parsed that way; in practice, it will run `python` with no arguments. – Charles Duffy Jan 25 '19 at 15:27
1

Much improved as to the first half. Re: the second half, better to demonstrate use of `pipes.quote()` (or in Python 3 `shlex.quote()`) for creating content that *can* be substituted into a shell command safely, rather than just putting in unsafe code with a bunch of caveats. – Charles Duffy Jan 30 '19 at 12:54
@CharlesDuffy, Updated Now I can't check the last update. Please review it, thanks. – Michael Kazarian Jan 30 '19 at 13:47

score 1 · Answer 4 · answered Nov 22 '13 at 08:30

1

You should concatenate the command to a whole string:

p = subprocess.Popen("python mytool.py -a " + servers[server]['address'] + " -x " + servers[server]['port'] + " -p " + servers[server]['pass'] + " some additional command", shell=True, stdout=subprocess.PIPE)

answered Nov 22 '13 at 08:30

ciphor

8,018
11
53
70

I have the output saved to variable but there's no output or trace of the script being executed. may be direct error output to check it? – GaNi Nov 22 '13 at 08:48
Add "stderr=subprocess.PIPE" also – ciphor Nov 22 '13 at 08:50
If someone feeds you a list of addresses that contains `$(rm -rf ~)`, you'd better hope nobody is using this concatenation-based approach. This is massively insecure; even if you *do* want to use `shell=True`, the (only) safe way to do it is to pass variables out-of-band from script code: `p = subprocess.Popen(['python mytool.py -a "$1" -x "$2" -p "$3" some additional command', '_', str(servers[server]['address']), str(servers[server]['port']), str(servers[server]['pass'])], shell=True, stdout=subprocess.PIPE)` – Charles Duffy Jan 25 '19 at 05:11

score 0 · Answer 5 · answered Mar 15 '23 at 20:56

0

You can set the environment by passing

env={whatever_you_want: values} | os.environ

in your subprocess call.

answered Mar 15 '23 at 20:56

Max Bileschi

2,103
2
21
19

score -1 · Answer 6 · answered Dec 20 '21 at 08:16

Usecase

To run n amount of shell script or python script

Create a run.py file that responsible to pass the argument(integer) to the shell file.

Let say you like to run n(4) qty of shell script or python script that accept 1 argument.

Create a file run.python with the code below.

Below code illustrate

instanceQty = Amount of shell script to run
os.getcwd() = path to your current file
mockScript.sh = shell script that I put on same directory with run.py

To run shell script

# this is python file with name run.py
import subprocess,os
instanceQty = 4
for i in range(0, instanceQty):
    print(os.getcwd())
    subprocess.Popen(f"{os.getcwd()}/mockScript.sh {i}",shell=True,executable='/bin/bash')

To run python script

import subprocess,os,sys
instanceQty = 4
for i in range(0, instanceQty):
    print(os.getcwd())
    subprocess.Popen([sys.executable,f"{os.getcwd()}/mockScript.py",str(i)])

Run this file with

python run.py

permission problem on MacOS

sudo chmod ug+x mockScript.sh

sudo chmod ug+x run.py

All code tested on Python 3.8.1 and MacOs 12.0.1 environment.

The call to `os.getcwd()` is completely unnecessary here; see [What exactly is current working directory?](https://stackoverflow.com/questions/45591428/what-exactly-is-current-working-directory) — tripleee, Mar 14 '22 at 07:33
The "permission problem" also seems like a beginner misunderstanding. If you explicitly run `python script.py` you only need read access to `script.py` (though for convenience in interactive use, you probably want the script to have a shebang and be marked executable too). — tripleee, Mar 14 '22 at 07:35

Why does passing variables to subprocess.Popen not work despite passing a list of arguments?

6 Answers6

Linked

Related