Error with subprocess Popen and shell=False for the tar command

Question

I have following simple python code that aims to minic /usr/bin/tar -czvf x002.tgz *.py

The code is:

import subprocess

base_dir = "/home/workspace/x002"
tar_file = "x002.tgz"
py_file = "*.py"
p = subprocess.Popen(["/usr/bin/tar", "-czvf", tar_file, py_file], cwd=base_dir, stdout=subprocess.PIPE,
                     stderr=subprocess.STDOUT,
                     shell=False)
results = []
while True:
    line = p.stdout.readline().decode('utf8').strip()
    if line == '' and p.poll() is not None:
        break
    else:
        results.append(line)

print(results)

I confirm that there exists py files under the directory of base_dir, but it complains as follows, I don't know where the problem is ,could someone help take a look? Thanks!

['tar: *.py: Cannot stat: No such file or directory', 'tar: Exiting with failure status due to previous errors', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']

Does this answer your question? [How to tar all files with wildcard character (\*) using subprocess](https://stackoverflow.com/questions/52245068/how-to-tar-all-files-with-wildcard-character-using-subprocess) — Nishant, Jun 29 '20 at 11:16

Nishant · Accepted Answer · 2020-06-30T11:10:22.477

This simple code will reproduce the issue you are facing:

import subprocess

p = subprocess.Popen(
    ["ls", "*"],
    shell=False
)

The problem is that wildcard * expansion is a shell (bash, zsh etc.) feature; commands like tar, ls etc. don't understand it. It is evaluated by the shell prior to the command execution (using exec system call). You can fix this by supplying shell=True. It will cause Python to use a shell to execute the command. Read more about it here: Actual meaning of 'shell=True' in subprocess.

However, it is better to use shell=False in Python scripts since that avoids spawning an extra shell. Python has an equivalent feature called glob that does the same thing. This will change the coding style a bit i.e. instead of thinking about executing commands in an interactive shell from Python, you think about doing the same thing using Python's features like variable, functions etc.

Thanks @Nishant for the great help! – Tom Jun 30 '20 at 09:28 — Tom, Jun 30 '20 at 09:28

Error with subprocess Popen and shell=False for the tar command

1 Answers1