Using back-ticks in Python subprocess

Question

I want to run this git command through a Python script and get the output of it:

git diff --name-only mybranch `git merge-base mybranch develop`

The purpose of the command is to see what changes have been made on mybranch since the last merge with develop.

To achieve this I'm using subprocess.Popen:

output = subprocess.Popen(["git", "diff", "--name-only", "mybranch", "`git merge-base mybranch develop`"], stdout=subprocess.PIPE, shell=True)

However, this does not work. The variable output.communicate()[0] simply gives me a printout of git usage -- essentially telling me the input command is wrong.

I saw that a similar question exists here, but it only told me to use shell=True which didn't solve my problem.

I also attempted to run the two commands in succession, but that gave me the same output as before. It is possible that I am missing something in this step, though.

Any help or tips are appreciated.

why are you trying to use shell=True and passing a list of args? Also why is git in a separate list?` — Padraic Cunningham, Jul 20 '15 at 12:32
The answer in the linked question said that using back-ticks would only work with shell=True. They are in separate lists for no particular reason, I'll edit to avoid confusion. — Jokab, Jul 20 '15 at 12:36
You should either use a string with shell=True or remove shell=True and pass a list of individual args — Padraic Cunningham, Jul 20 '15 at 12:38
Thank you, this was the answer. I must have missed this in the docs. — Jokab, Jul 20 '15 at 12:41
Note that backticks are deprecated in Bash: the `$(cmd)` syntax is less prone to being mis-read, and you can nest it if necessary. See [BashFAQ/082](http://mywiki.wooledge.org/BashFAQ/082) "Why is $(...) preferred over (backticks)?" in the BashGuide for further details. Also, backticks are a pain to write in SO comments. :) — PM 2Ring, Jul 20 '15 at 12:57
You might consider having a second `subprocess.Popen()` call in your Python to invoke `git merge-base` and collect its output for use in the later call; that's what your other command is doing, just telling the shell to create the subprocess that runs that second call. More efficient to do it yourself than create a shell to do it for you. (And if this is a web service and `mybranch` is coming from an untrusted source, you should also do that for security reasons; shell injection is no joke). — Charles Duffy, Jul 20 '15 at 16:38

bakkal · Accepted Answer · 2015-07-20T12:54:13.727

2

Backticks and subprocess

The backtick being a shell feature, you may not have a choice but to use shell=True, however pass in a shell command string, not a list of args

So for your particular command (assuming it works in the first place)

process = subprocess.Popen("git diff --name-only mybranch `git merge-base mybranch develop`", stdout=subprocess.PIPE, shell=True)

Notice when you call Popen() you get a process, shouldn't be called output IMO

Here's a simple example that works with backticks

>>> process = subprocess.Popen('echo `pwd`', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'

Or you can use the $(cmd) syntax

>>> process = subprocess.Popen('echo $(pwd)', stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'/Users/bakkal\n'

Here's what did NOT work (for backticks)

>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=True)
>>> out, err = process.communicate()
>>> out
'\n'
>>> process = subprocess.Popen(['echo', '`pwd`'], stdout=subprocess.PIPE, shell=False)
>>> out, err = process.communicate()
>>> out
'`pwd`\n'

edited Jul 20 '15 at 12:54

answered Jul 20 '15 at 12:40

bakkal

54,350
12
131
107

@PM2Ring Added a sample using `$(cmd)` with `subprocess` – bakkal Jul 20 '15 at 12:57
Thanks. I decided to move my comment to the question, since it's probably more appropriate there. – PM 2Ring Jul 20 '15 at 12:58
1

you *have* a choice, you can [drop `shell=True`](http://stackoverflow.com/a/31521714/4279) – jfs Jul 20 '15 at 16:33
You'd have to do the backtick expansion manually, because from my trials it doesn't seem you can expand backticks or `$(cmd)` without `shell=True`, backticks being the crux of the question – bakkal Jul 20 '15 at 16:46

score 2 · Answer 2 · answered Jul 20 '15 at 16:32

On POSIX, the argument list is passed to /bin/sh -c i.e., only the first argument is recognized as a shell command i.e., the shell runs git without any arguments that is why you see the usage info. You should pass the command as a string if you want to use shell=True. From the subprocess docs:

On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])

You don't need shell=True in this case.

#!/usr/bin/env python
from subprocess import check_output

merge_base_output = check_output('git merge-base mybranch develop'.split(), 
                                 universal_newlines=True).strip()
diff_output = check_output('git diff --name-only mybranch'.split() +
                           [merge_base_output])

Using back-ticks in Python subprocess

2 Answers2

Backticks and subprocess