subprocess popen argument with wildcard

Question

I wrote a method that is defined as below and works

def cmd_exec(cmd_tokens = []):
    p = subprocess.Popen(cmd_tokens, 
    stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    out, err = p.communicate()
    return (out, err)

I have a constant as LOAD_IMAGES=['docker', 'load', '-i', 'my_img_file_101']

When I execute the above method with LOAD_IMAGES as arguments, it works fine. However, the filename number might change for me and when I try to use a wildcard, I get the error. Say when I have LOAD_IMAGES=['docker', 'load', '-i', 'my_img_file*'], I get an error from the Py/Bash as open my_img_file*: no such file or directory

How do I make the wild card work. Executing the command directly on bash works.I mean when I say this on bash, it works docker load -i my_img_file*

score 3 · Accepted Answer · edited Jun 20 '20 at 09:12

Wildcard expansion is something bash takes care of while you're in the shell. It's not something built into Linux/Unix to be able to expand wildcards or any of that syntax. So you need to be explicit about it and do the expansion by hand.

There is an alternative, which is actually letting the shell do all the work, via shell=True. It has its drawbacks, as documented in the question. Quoting:

This is a good thing, see the warning block in the "Frequently Used Arguments" section, of the subprocess docs. It mainly discusses security implications, but can also helps avoid silly programming errors (as there are no magic shell characters to worry about)

My main complaint with shell=True is it usually implies there is a better way to go about the problem - with your example, you should use the glob module...

`glob` is a better alternative. – darksky Jul 16 '17 at 06:37 — darksky, Jul 16 '17 at 06:37

score 0 · Answer 2 · answered Mar 30 '23 at 12:54

In addition to using shell=True as an argument to subprocess.Popen, you must also use a string as argument, not a list. Its actually the literal evaluation of the list that is causing the issue. With the move away from command and os.system usage in Python 3, its important to get this right. If you pass a list it will evaluation only index 0 of the list.

def cmd_exec(cmd_string_not_list):
    p = subprocess.Popen(cmd_string_not_list, 
    stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
    out, err = p.communicate()
    return (out, err)

subprocess popen argument with wildcard

2 Answers2