subprocess wildcard usage

Question

import os

import subprocess

proc = subprocess.Popen(['ls','*.bc'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

out,err = proc.communicate()

print out

This script should print all the files with .bc suffix however it returns an empty list. If I do ls *.bc manually in the command line it works. Doing ['ls','test.bc'] inside the script works as well but for some reason the star symbol doesnt work.. Any ideas ?

related: [Wildcard not working in subprocess call using shlex](http://stackoverflow.com/q/7156892/4279) — jfs, May 16 '14 at 03:42

score 101 · Accepted Answer · answered Apr 03 '12 at 16:01

101

You need to supply shell=True to execute the command through a shell interpreter. If you do that however, you can no longer supply a list as the first argument, because the arguments will get quoted then. Instead, specify the raw commandline as you want it to be passed to the shell:

 proc = subprocess.Popen('ls *.bc', shell=True,
                                    stdout=subprocess.PIPE,
                                    stderr=subprocess.PIPE)

answered Apr 03 '12 at 16:01

Niklas B.

92,950
18
194
224

1

Thanks this worked just fine. Some of the examples that I found on the internet had a list as their first argument for some reason – Cemre Mengü Apr 03 '12 at 16:06
17

@Cemre: That's usually advisable because you *don't* want the shell to interpret the arguments. Imagine you pass user input to a command like in `'ls ' + user_supplied_path`. The user could just input the path `; shutdown -s` and the system would halt! If you use `['ls', user_supplied_path]`, you prevent this kind of injection. – Niklas B. Apr 03 '12 at 16:09
2

In dynamic case it can be very dangerous. ``Popen("ls " + filename, shell=True)``, where filename equals ``blahblah; rm -rf /``. Here is [my variant](https://stackoverflow.com/a/50682740/2039471) – Alexander C Jun 04 '18 at 14:21
1

You probably want to avoid `Popen()` in favor of `subprocess.run()` and friends. See also https://stackoverflow.com/questions/4256107/running-bash-commands-in-python – tripleee Oct 06 '19 at 12:12

score 72 · Answer 2 · edited Jul 28 '16 at 01:05

72

Expanding the * glob is part of the shell, but by default subprocess does not send your commands via a shell, so the command (first argument, ls) is executed, then a literal * is used as an argument.

This is a good thing, see the warning block in the "Frequently Used Arguments" section, of the subprocess docs. It mainly discusses security implications, but can also helps avoid silly programming errors (as there are no magic shell characters to worry about)

My main complaint with shell=True is it usually implies there is a better way to go about the problem - with your example, you should use the glob module:

import glob
files = glob.glob("*.bc")
print files # ['file1.bc', 'file2.bc']

This will be quicker (no process startup overhead), more reliable and cross platform (not dependent on the platform having an ls command)

edited Jul 28 '16 at 01:05

Christian Long

10,385
6
60
58

answered Apr 03 '12 at 16:17

dbr

165,801
69
278
343

`subprocess does not send your commands via a shell` Why? Can you provide any reference? Thanks – Alston Nov 22 '15 at 15:03
1

Plus it gives a python object to work with and manipulate. This should be the selected answer. Much more `python` friendly. – Cyan May 21 '16 at 13:52
1

@Alston The `subprocess` documentation very clearly documents the meaning of `shell=True` and how the default is `shell=False`. See also https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess – tripleee Oct 06 '19 at 12:13

score 2 · Answer 3 · answered Mar 18 '20 at 19:49

2

Besides doing shell=True, also make sure that your path is not quoted. Otherwise it will not be expanded by shell.

If your path may have special characters, you will have to escape them manually.

answered Mar 18 '20 at 19:49

Tamaki Sakura

482
5
22

subprocess wildcard usage

3 Answers3

Linked

Related