python3 subprocess error capturing output from shell commands

Question

I've followed a number of questions on SO about capturing subprocess output. using git within subprocess seems to output to error instead of stdout or stderr

This is what I want to replicate from shell.

git add .
git commit -m 'commit message blah'
git push -u origin master

with tracking of output from each stage. In this example, I have no updates to commit.

$git add .
$ git commit -m 'wip'
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

as per code below, I am using

output = run("git add .", check=True, capture_output=True, shell=True)

and checking output.returncode, but experiencing an CalledProcessError instead of values to output. any ideas? I can fudge functionality through error trapping, but it's far from optimal.

I can't capture the "nothing to commit, working tree clean" output from 'git push', this seems weird. as are the values for '' stderr & stdout. returncode is a 0 after 'git commit'.

nb: 'git add .' does not produce any output in shell.

environment : ubuntu app on windows 10. Python3, anaconda 64 bit.

from subprocess import *
import sys
try:
  print("start git add.")
  #output = run("git add .", check=True, stdout=PIPE, stderr=PIPE, shell=True)
  output = run("git add .", check=True, capture_output=True, shell=True)
  print("git add completed.")
  print("output.stderr:", "'"+output.stderr.decode('utf-8')+"'")
  print("output.stdout:", "'"+output.stdout.decode('utf-8')+"'")
  print("output.stderr==None:", output.stderr==None)
  print("output.stdout==None:", output.stdout==None)
  print("output.returncode:", output.returncode)
  if output.returncode==0:
    output=None
    print("start git commit.")
    #output = run(f"git commit -m 'commit message three'", check=True, stdout=PIPE, stderr=PIPE, shell=True)
    output = run("git commit -m 'commit message three'", check=True, capture_output=True, shell=True)
    #exits to exception here when error occurs dueing git commit.
    print("git commit completed.")
    print("output.stderr:", output.stderr.decode('utf-8'))
    print("output.stdout:", output.stdout.decode('utf-8'))
    print("output.stderr==None:", output.stderr==None)
    print("output.stdout==None:", output.stdout==None)
    print("output.returncode:", output.returncode)
    if output.returncode==0:
      output=None
      print("start git push.")
      #output = run("git push -u origin master -f", check=True, stdout=PIPE, stderr=PIPE, shell=True)
      output = run("git push -u origin master -f", capture_output=True)
      print("git push completed.")
      print("output.stderr:", output.stderr)
      print("output.stdout:", output.stdout)
      print("output.stderr==None:", output.stderr==None)
      print("output.stdout==None:", output.stdout==None)
      print("output.returncode:", output.returncode)
      if output.returncode==0:
        print("success @ git push, output.returncode==0.")
except CalledProcessError as error:
  print("CalledProcessError error caught.")
  print('CalledProcessError:An exception occurred: {}'.format(error))
  print("CalledProcessError:sys.exc_info()[0]:", sys.exc_info()[0])
  print("CalledProcessError:output:", output)
  print("CalledProcessError:output.stderr:", output.stderr)
  print("CalledProcessError:output.stdout:", output.stdout)

output from this code block

start git add.
git add completed.
output.stderr: ''
output.stdout: ''
output.stderr==None: False
output.stdout==None: False
output.returncode: 0
start git commit.
CalledProcessError error caught.
CalledProcessError:An exception occurred: Command 'git commit -m 'commit message three'' returned non-zero exit status 1.
CalledProcessError:sys.exc_info()[0]: <class 'subprocess.CalledProcessError'>
CalledProcessError:output: None
Traceback (most recent call last):
  File "<stdin>", line 15, in <module>
  File "/home/bmt/anaconda3/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'git commit -m 'commit message three'' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 41, in <module>
AttributeError: 'NoneType' object has no attribute 'stderr'

edit: my main error was using 'check=True' and 'shell=True' also changing the shell command from one string to a list of strings.

output = run("git add .", check=True, capture_output=True, shell=True)

to

output = run(["git", "add", "."], capture_output=True)

Answer 1

I've added some general comments about using Git from other programming languages as comments. For your more specific case of calling git commit here, though, note that your error CalledProcessError occurs because git commit found nothing to commit. When you did this from the shell:

$ git commit -m 'wip'
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

the git commit command returned a failure exit status, value 1. Your shell did not complain about this.

When you ran the same thing with:

output = run("git commit -m 'commit message three'", check=True,
    capture_output=True, shell=True)

the git commit command again returned a failure exit status. It was Python, not Git—specifically, the subprocess.run function—that raised a CalledProcessError, and it did so because you included check=True.

This exception jumped out of the normal code flow and went into your except CalledProcessError section, where you tried to use the value found via the variable named output. This variable was last bound to a value here:

    output=None
    print("start git commit.")

The variable still contains None at this point, which is why you saw:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 41, in <module>
AttributeError: 'NoneType' object has no attribute 'stderr'

As a general rule,¹ if you want to capture everything—including the exit code and both stdout and stderr streams—with subprocess.run, don't set check=True, because that means jump out of normal processing, raising a CalledProcessError, if the exit code is anything other than zero. That is, had you written:

output = run("git commit -m 'commit message three'", capture_output=True,
    shell=True)

the whole thing would have behaved more in the way you expected, though you should consult the output.returncode value to see if git commit succeeded.

Note, too, that shell=True is usually a bad idea: it tends to introduce security holes. The Python documentation calls this out. In this case you could use:

output = run(["git", "commit", "-m", "commit message three"], capture_output=True)

which avoids the need for shell=True. (The particular example you have here is safe as the entire string is a constant, but this is a bad habit to get into.)

Last, while git add and git commit can be pretty safely run in such a way as to be non-interactive—you can just inspect their return code value to determine success or failure—git push is trickier. The problem here is that git push requires that your (local) Git call up some other (remote) Git. Git will often do this using https/ssl or ssh. To do that, Git invokes other third-party programs, and those sometimes demand input passwords or passphrases.

Because Git can use multiple different third-party programs for this, each of which has its own different options, there is no one right way to avoid it. See Git keeps prompting me for a password and Git push requires username and password for various details.

---

¹Since Python 3.5, however, you can use the error value raised here to access everything:

try:
    result = subprocess.run(cmd, check=True)
    ... use result.stdout etc ...
except subprocess.CalledProcessError as err:
    ... use err.stdout etc ...

but this will often lead to more duplicated code, so it's probably better to just leave out the check=True and inspect result.returncode.