subprocess.call() arguments ignored when using shell=True w/ list

Question

I am trying to get python's subprocess.call method to accept some args commands through a list (consisting of a sequence of strings) as advised in the python documentation. To explore this behavior before putting it into my actual script, I opened up IPython, ran some commands involving different combinations of shell settings and args commands and got the following behavior:

In [3]: subprocess.call(['ls', '-%sl' %'a'])
total 320
drwxr-xr-x  20 Kohaugustine  staff   680 Oct 15 16:55 .
drwxr-xr-x   5 Kohaugustine  staff   170 Sep 12 17:16 ..
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 a.out
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct  3 10:28 ex1-6
-rw-r--r--@  1 Kohaugustine  staff   204 Oct  3 10:28 ex1-6.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Oct  3 10:15 ex1-7
-rw-r--r--@  1 Kohaugustine  staff    71 Oct  3 10:15 ex1-7.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:22 hello
-rw-r--r--@  1 Kohaugustine  staff    58 Sep 12 16:27 hello.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@  1 Kohaugustine  staff   185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 21 10:06 temperature.o
-rw-r--r--@  1 Kohaugustine  staff   406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@  1 Kohaugustine  staff   582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@  1 Kohaugustine  staff   178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 23 17:21 temperature_ex1-5.o
Out[3]: 0

In [4]: subprocess.call(['ls', '-%sl' %'a'], shell=True)
a.out           ex1-7           hello.c         hello_2.o       lesson_1-5.c            temperature_ex1-4.c
ex1-6           ex1-7.c         hello.o         hello_3.o       temperature.o           temperature_ex1-5.c
ex1-6.c         hello           hello_1.o       lesson_1-5      temperature_ex1-3.c     temperature_ex1-5.o
Out[4]: 0

In [6]: subprocess.call(['ls', '-al'])    
total 320
drwxr-xr-x  20 Kohaugustine  staff   680 Oct 15 16:55 .
drwxr-xr-x   5 Kohaugustine  staff   170 Sep 12 17:16 ..
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 a.out
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct  3 10:28 ex1-6
-rw-r--r--@  1 Kohaugustine  staff   204 Oct  3 10:28 ex1-6.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Oct  3 10:15 ex1-7
-rw-r--r--@  1 Kohaugustine  staff    71 Oct  3 10:15 ex1-7.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:22 hello
-rw-r--r--@  1 Kohaugustine  staff    58 Sep 12 16:27 hello.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@  1 Kohaugustine  staff   185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 21 10:06 temperature.o
-rw-r--r--@  1 Kohaugustine  staff   406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@  1 Kohaugustine  staff   582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@  1 Kohaugustine  staff   178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 23 17:21 temperature_ex1-5.o
Out[6]: 0

In [7]: subprocess.call(['ls', '-al'], shell = True)
a.out           ex1-7           hello.c         hello_2.o       lesson_1-5.c            temperature_ex1-4.c
ex1-6           ex1-7.c         hello.o         hello_3.o       temperature.o           temperature_ex1-5.c
ex1-6.c         hello           hello_1.o       lesson_1-5      temperature_ex1-3.c     temperature_ex1-5.o
Out[7]: 0

It seems like whenever shell=True, the output seems to be the same as:

In [9]: subprocess.call(['ls'])
a.out           ex1-7           hello.c         hello_2.o       lesson_1-5.c            temperature_ex1-4.c
ex1-6           ex1-7.c         hello.o         hello_3.o       temperature.o           temperature_ex1-5.c
ex1-6.c         hello           hello_1.o       lesson_1-5      temperature_ex1-3.c     temperature_ex1-5.o
Out[9]: 0

I'm puzzled; what happened to the '-a' option when I set shell=True? Didn't the shell read it? I've read the Docs and that it says that when shell=True, it means that my specified command will be executed through the shell, so it should mean that ls -a was fed to the shell and acted upon by the shell. Then why the behavior in [4] and [7] ? Also the pydocs doesn't explain it directly (although it does say what subpprocess will NOT DO when we set shell=False); what does it mean when we let shell=False? Is a new process spawned in the OS without having the shell actually control it?

Also, in case it might seem really awkward that I'm using a format string in [3] and [4], its because in my actual script where I'll be using subprocess.call, I will have to rely on these format strings to substitute in the appropriate command options. I cannot hardcode some of the command line options. Using a pure string for args is out of the question too because in my script there will be a method that has to do list operations on the commands. I don't know if there might be a better way to go about this though, so if will really help if anyone can suggest something different.

Thank you very much!

You could have chosen a directory with less files in it to make your example more compact and readable here :) — GreenAsJade, Oct 17 '14 at 04:03
related Python issue: [Don't use a list argument together with shell=True in subprocess' docs](http://bugs.python.org/issue21347) — jfs, Oct 17 '14 at 21:55
@jfs, ...I rather strongly disagree that this should be discouraged. Using a list with `shell=True` provides the best of both worlds: You can use functionality/syntax that's built into your shell, and you can pass literal strings as arguments without needing to escape them. Certainly, one doesn't need it if one is running a completely hardcoded shell command that isn't parameterizable, or a command that could be run with `shell=False`, but not every situation falls into one of those categories. — Charles Duffy, Sep 06 '22 at 21:56
@CharlesDuffy could you provide a code example (`shell=True` + list) that you find useful? — jfs, Sep 07 '22 at 03:24
@jfs, sure. I just edited such an example into the community-wiki answer to [bash process substitution in python with popen](https://stackoverflow.com/questions/26248137), allowing that code to be parameterized without either injection vulnerabilities or need of `shlex.quote()`. — Charles Duffy, Sep 07 '22 at 13:51
@CharlesDuffy: thanks. The example looks complicated. Perhaps, a separate script + `shell=False` would be more readable (if there is no ready script, I would implement it in Python (without process substition, using `Path.iterdir()` or similar). — jfs, Sep 07 '22 at 18:50
@jfs, _nod_, most things can and should be done with `shell=False`, so this is only relevant in the subset of use cases where `shell=True` is valuable, _and_ when you need a way to pass data to that shell script you're running in-band. It's not a commonly needed practice, but on the other hand, it's far better to encourage it than to have people be writing shell injection vulnerabilities when they inject variables into their Python-embedded shell scripts badly. — Charles Duffy, Sep 07 '22 at 19:18
@jfs, another example would be a program that only supports receiving a filename for input where you'd rather not create a temporary file. `subprocess.Popen(['''input=$1; shift; "$1" <(printf '%s\n' "$input") "${@:2}"''', 'bash', input_string_here, program_name, program_arg1, program_arg2, ...], shell=True, executable='/bin/bash')` -- you get your program run with `input_string_here` replaced with a filename from which `input_string_here` can be read. — Charles Duffy, Sep 07 '22 at 19:24
...it's _possible_ to do that with native Python, but unless you use named FIFOs (which means you're touching the filesystem) it's a whole lot wordier (bash determines at compile time if the operating system supports anything equivalent to `/dev/fd` and uses that to back the implementation of `<()` and `>()` if possible, and named FIFOs if not). — Charles Duffy, Sep 07 '22 at 19:25

chepner · Accepted Answer · 2014-10-17T04:16:10.070

15

When shell is True, the first argument is appended to ["/bin/sh", "-c"]. If that argument is a list, the resulting list is

["/bin/sh", "-c", "ls", "-al"]

That is, only ls, not ls -al is used as the argument to the -c option. -al is used as the first argument the shell itself, not ls.

When using shell=True, you generally just want to pass a single string and let the shell split it according the shell's normal word-splitting rules.

# Produces ["/bin/sh", "-c", "ls -al"]
subprocess.call("ls -al", shell=True)

In your case, it doesn't see like you need to use shell=True at all.

edited Oct 17 '14 at 04:16

answered Oct 17 '14 at 04:07

chepner

497,756
71
530
681

4

Thank you very much! This makes a lot more sense now! This should be included in the documentation. – AKKO Oct 23 '14 at 03:17

Charles Duffy · Answer 2 · 2022-09-06T21:44:52.667

When you use shell=True with a list, extra arguments are passed to the shell itself, not to the command running in the shell. They can then be referred to from within the shell script (passed as argv[0]) as $0, $1, etc.

The easiest answer is "don't do that": If you want to pass a list, don't use shell=True; if you want to pass a string, always use shell=True.

That said, it is possible to form your command in such a way as to read those arguments. The below is an example that violates my above rule -- a command you couldn't implement_[*1] without shell=True (and executable='/bin/bash' to avoid having a dependency on your operating system using bash for /bin/sh), because it relies on bash's built-in version of printf (which supports %q as an extension):

subprocess.call([
    "printf '%q\\n' \"$0\" \"$@\"",
    'these strings are\r\n',
    '"shell escaped" in the output from this command',
    "so that the output can *safely* be run through eval",
    "observe that no /tmp/owned file is created",
    "including when the output of this script is run by bash as code:"
    "$(touch /tmp/owned) \"$(touch /tmp/owned)\"",
    '$(touch /tmp/owned) \'$(touch /tmp/owned)\'',
], shell=True, executable='/bin/bash')

_{[*1] - If one ignores that one could use /bin/bash as argv[0] with shell=False.}

Thanks a lot for your help! I think I'll just keep things simple without using shell=True but it is interesting to see that it is possible to still use that and have the shell interpret all the arguments in a list. Only issue is that it prints out all the '\' backslash characters and this won't be good as I need the commands to be in one continuous line separated by spaces. — AKKO, Oct 23 '14 at 03:21

subprocess.call() arguments ignored when using shell=True w/ list

2 Answers2

Linked

Related