Python Curses Git Subprocess Output with unexpected spacing

Question

I am observing a weird spacing issue when I call git within a python-curses screen. What am I doing in my minimal working example below that causes the spacing to be ajar and not flush with the side of the screen?

Minimal Working Example:

import curses, subprocess

class MyApp(object):

    def __init__(self, stdscreen):
        self.screen = stdscreen
        self.screen.addstr("Loading..." + '\n')
        self.screen.refresh()

        url = 'http://github.com/octocat/Hello-World/'

        process = subprocess.Popen(['git', 'clone', url], stdout=subprocess.PIPE)

        self.screen.addstr("Press any key to continue.")

        self.screen.getch()

if __name__ == '__main__':
    curses.wrapper(MyApp)

Output:

Loading...
Press any key to continue.Cloning into 'Hello-World'...
                                                       warning: redirecting to https://github.com/octocat/Hello-World/
              remote: Enumerating objects: 13, done.
                                                    remote: Total 13 (delta 0), reused 0 (delta 0), pack-reused 13
Unpacking objects: 100% (13/13), done.3)

Expected Output:

Loading...
Cloning into 'Hello-World'...
warning: redirecting to https://github.com/octocat/Hello-World/
remote: Enumerating objects: 13, done.
remote: Total 13 (delta 0), reused 0 (delta 0), pack-reused 13
Unpacking objects: 100% (13/13), done.3)
Press any key to continue.

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

git clone writes its informative messages to stderr, not stdout.

From the documentation (emphasis added):

--verbose

Run verbosely. Does not affect the reporting of progress status to the standard error stream.

(the code below is by Justapigeon)

Updated MWE with this consideration:

import curses, subprocess
import os, re

class MyApp(object):

    def __init__(self, stdscreen):
        self.screen = stdscreen
        self.screen.addstr("Loading..." + '\n')
        self.screen.refresh()

        url = 'http://github.com/octocat/Hello-World/'

        #process = subprocess.Popen(['git', 'clone', url], stdout=subprocess.PIPE)

        # additional code from: https://stackoverflow.com/a/39564562/6924364

        dloutput = subprocess.Popen(['git', 'clone', '--progress', url], stderr=subprocess.PIPE, stdout=subprocess.PIPE)

        fd = dloutput.stderr.fileno()
        gitmsg = [] 
        while True:
            lines = os.read(fd,1000).decode('utf-8')
            lines = re.split('\n|\r', lines)
            for l in lines:
                self.screen.erase()
                for g in gitmsg:
                    self.screen.addstr(g + '\n')
                if l != '':
                    self.screen.addstr(str(l) + '\n')
                    self.screen.refresh()
                if 'Cloning' in l:
                    gitmsg.append(l)
                if 'done' in l:
                    gitmsg.append(l)
            if len(lines) == 1:
                break

        self.screen.addstr("Press any key to continue.")

        self.screen.getch()

if __name__ == '__main__':
    curses.wrapper(MyApp)

My comment on your second edit: you can keep the `universal_newlines=True` (or in 3.7 or later, `text=True`) setting if you like. It's probably wise to capture stdout in case `git clone` feels like writing to stdout. There's a bug of sorts with fetch progress strings in underlying Git: they're not flushed to the stream when they should be. This makes it difficult to read progress values, if you try to get fancy. — torek, Jan 19 '20 at 07:22
Note that the stderr stream will contain carriage-returns without newlines: these are intended to overwrite previous output lines. That's how Git prints its progress indicators as it runs the fetch step of cloning, or an actual fetch when fetching. — torek, Jan 19 '20 at 07:23
Think I solved how to stream the data in real time in the most recent edit. — Justapigeon, Jan 25 '20 at 06:38
You should probably move your code to your own answer (you can answer your own questions!). I will note that in your latest edit you call `os.read` with a size of 1000, and then decode that using a UTF-8 decoder. If the actual string has UTF-8 in it, you could hit an edge case where the UTF-8 encoded Unicode character is split across multiple `read` calls. Whether it's worth trying to handle this is up to you. — torek, Jan 25 '20 at 06:56

Python Curses Git Subprocess Output with unexpected spacing

Minimal Working Example:

Output:

Expected Output:

1 Answers1