Read git clone's output in real time

Question

I need to read the status ( Receiving objects XXX%) of a git clone process, but can't figure out how.

I am using subprocess.Popen but I cant' figure out how to grab the line I need:

proc = subprocess.Popen([GIT, "-C",IDIR+"/"+REPO,"clone",URL],shell=False,bufsize=0,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)

This is the typical output:

Cloning into 'xxx'...
remote: Reusing existing pack: 5250, done.
remote: Counting objects: 1652, done.
remote: Compressing objects: 100% (1428/1428), done.
remote: Total 6902 (delta 389), reused 0 (delta 0)
Receiving objects: XXX% (6902/6902), 27.66 MiB | 1.51 MiB/s, done.
Resolving deltas: 100% (2010/2010), done.
Checking connectivity... done.
Checking out files: 100% (3266/3266), done.

Edit

I have tried all of the suggestions both in this thread and in the one suggested as a duplicate, but none appear to work.

This suggestion results in "Cloning into 'test'... done" but none of the other output:

popen = subprocess.Popen(["git", "clone", "https://github.com/xxx/test.git"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT,shell=False )
for line in popen.stdout:
  print "out newline:"
  print line
print "done"

This is the output which does not contain any of the stats:

out newline:
Cloning into 'test'...

done

You correctly set up your subprocess with `stdout=subprocess.PIPE` so now you can grab the process's output by `proc.stdout.readline()` or `proc.communicate()`. After that, it's just a game of string matching or using regexes. — Cory Kramer, Apr 18 '14 at 13:57
possible duplicate of [Getting realtime output using subprocess](http://stackoverflow.com/questions/803265/getting-realtime-output-using-subprocess) — m.wasowski, Apr 18 '14 at 13:58
I have tried all of the suggestions in that thread, as well as the ones in this thread, none appear to be ble to capture the output. — crankshaft, Apr 19 '14 at 00:49
You're still scanning `popen.stdout` instead of `popen.stderr`. To avoid any further issues, I'd also suggest combining stdout and stderr i.e. call Popen with `stderr=subprocess.STDOUT`. — Cristian Ciupitu, Apr 19 '14 at 00:50
Hi Christian, thanks so much for your patience. I have modified the code as suggested and update the main thread with the code I tested. Now I get the output "Cloning into 'test'..." followed by "done" but none of the other status output. — crankshaft, Apr 19 '14 at 01:04

evading · Answer 1 · 2014-04-19T06:03:02.590

3

[edited as per the comments] Calling git clone with the --progress flag redirects the output ok.

This works for me.

import subprocess

popen = subprocess.Popen(["git", "clone", "--progress", "git@bitbucket.org:xxx/yyy.git"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in popen.stdout:
    print line,

print "done"

Gives the output

$ python test.py 
Cloning into 'yyy'...
remote: Counting objects: 1076, done.
remote: Compressing objects: 100% (761/761), done.
remote: Total 1076 (delta 488), reused 576 (delta 227)
Receiving objects: 100% (1076/1076), 6.24 MiB | 260.00 KiB/s, done.
Resolving deltas: 100% (488/488), done.
Checking connectivity... done.
done

[edit] Like Cristian Ciupitu points out you don't need the iter() just for line in popen.stdout: works just as well (or not depending on version).

edited Apr 19 '14 at 06:03

answered Apr 18 '14 at 14:01

evading

3,032
6
37
57

This doesn't look like it will work. How does it know when to stop iterating? – ebarr Apr 18 '14 at 14:03
@ebarr: Doesn't the input end with EOF character? – Danstahr Apr 18 '14 at 14:06
You can iterate directly over `popen.stdout` (at least in Python 2.7). – Cristian Ciupitu Apr 18 '14 at 14:13
Thanks, I tried this and does not appear to work, the output appears to originate from the subprocess.Popen() line and not from the "for line in..print". even if I comment out the for-loop completely, the output is the same. – crankshaft Apr 18 '14 at 23:30
That's because some of git's output (e.g. `Cloning into...`) goes to **stderr**, not **stdout**. So **stderr** needs to be redirected, too. – Cristian Ciupitu Apr 18 '14 at 23:48
Thanks, I tried adding the stderr redirect but now get no output at all. I have added your suggested code to the main thread above, I guess I misunderstand the implementation. – crankshaft Apr 19 '14 at 00:01

score 1 · Answer 2 · answered Apr 18 '14 at 14:02

1

I assume that you want the real-time communication to display some sort of progress while the process is still running.

The problem is that the normal stdout stream is buffered. What you need is the unbuffered stream. You can obtain it using the os module, for example:

  fd = proc.stdout.fileno()
  while proc.returncode is None:
      l = os.read(fd, 1000)   # Read a bit of data
      print l

answered Apr 18 '14 at 14:02

EvertW

1,160
9
18

Thanks, yes, that's exactly what I want to do, I tried your suggestion, but unfortunately the output was basically thousands of empty lines. – crankshaft Apr 19 '14 at 00:03
Hmm. Sounds like you are using a non-blocking read. It should not do that, perhaps you have modified the stream timeout and or blocking behaviour? – EvertW Apr 20 '14 at 12:57
@crankshaft try `l.decode('utf-8')` and then `if l != '': print l` – Justapigeon Jan 25 '20 at 06:39

score 0 · Answer 3 · answered Apr 18 '14 at 14:02

0

I think a subprocess.communicate() should do the trick.

I usually do it like this:

process=subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdoutput,stderror=process.communicate()
for line in stdoutput:
    if line.startswith('Receiving objects'):
        print line

answered Apr 18 '14 at 14:02

Andy S. C.

368
1
4
17

I just tried your suggestion, but there was no output at all generated. Even removing the "if line contains.." and printing all lines still showed no output. – crankshaft Apr 19 '14 at 00:10

score 0 · Answer 4 · edited May 23 '17 at 10:28

First of all, I suggest combining stdout and stderr to have everything in one place i.e. call Popen with stderr=subprocess.STDOUT.

Second, you need to execute git with --progress because the git-clone man page says:

--progress

Progress status is reported on the standard error stream by default when
it is attached to a terminal, unless -q is specified.  This flag forces
progress status even if the standard error stream is not directed to a
terminal.

and your git is not attached to a terminal.

So the code should look like this:

popen = subprocess.Popen(["git", "clone", "--progress", url],
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
for line in popen.stdout:
  print "out newline:"
  print line
print "done"

I suggest reading the answers to the "Getting realtime output using subprocess" question to improve your program.

score 0 · Answer 5 · answered Jan 25 '20 at 06:34

Tagging on to other responses that have described stderr and --progress, here is a complete minimal working example that will print the git clone command in real time:

import os
import re
import subprocess

def test():

    url = 'http://github.com/octocat/Hello-World/'

    output = subprocess.Popen(['git', 'clone', '--progress', url], stderr=subprocess.PIPE, stdout=subprocess.PIPE)

    fd = output.stderr.fileno()
    while True:
        lines = os.read(fd,1000).decode('utf-8')
        lines = re.split('\n|\r', lines)
        for l in lines:
            if l != '':
                print(l)
        if len(lines) == 1:
                break

    print('Press enter to continue.')

Read git clone's output in real time

Edit

5 Answers5

Linked