Handling pagers with python's subprocess

Question

I'm writing some scripts that call git commands, but having trouble when git tries to output using a pager, which blocks on user input. For example the following drops me in less (afaik) and I have to q to continue:

subprocess.Popen(['git', 'diff', '--stat', '--cached', 'origin/master']).wait()

In this specific case I can disable that by telling git not to use a pager:

subprocess.Popen(['git', '--no-pager', 'diff', '--stat', '--cached', 'origin/master']).wait()

Is there some general solution (Edit: i.e. that works for everything, not just git), where I can capture the output and not block on the pager, or is there some mechanism that could be informing the sub-process (git in this case) not to use a pager automatically?

I've tried closing subprocess's stdin and experimenting with the shell=True/False argument, but neither helped.

Update:
For example, git diff --stat --cached origin/master will land you in a pager if the output is longer than a page. git diff --stat --cached origin/master | cat will not. How does git know? How can I apply this same effect using subprocess?

'--no-pager' should not use 'less'. Are you using an up-to-date version of git ? Your git command works fine for me on 2.9.0 (without less) — Maurice Meyer, Jan 05 '17 at 20:28
@MauriceMeyer Yes, it does work for git. However I'm after a general solution for any programs that may attempt to automatically use a pager. — jozxyqk, Jan 05 '17 at 21:32

score 2 · Answer 1 · answered Jan 05 '17 at 20:26

2

git is sensible to the PAGER environment variable, the following works nice:

subprocess.Popen(['git', 'diff', '--stat', '--cached', 'origin/master'], env={'PAGER':'cat'}).wait()

answered Jan 05 '17 at 20:26

serge-sans-paille

2,109
11
9

Thanks! However I'm also after a solution to the root cause rather than just fixing git in the case. – jozxyqk Jan 05 '17 at 21:36

score 0 · Answer 2 · answered Jan 05 '17 at 21:49

My suggestion would be to pipe escape sequences required to exit the pager you expect to be opened. You can use the stdin argument to subprocess.Popen() to create a pipe, where you can send escape characters when you expect this issue to arise:

p = subprocess.Popen(
        ['git', 'diff', '--stat', '--cached', 'origin/master'], 
        stdin=subprocess.PIPE)

p.stdin.write('q') # q should quit most pagers
p.wait()

This could be easily wrapped up in functions or classes to handle different pagers in a simple way. It doesn't handle all of them as-is, but simply changing what is being written to the pipe allows others to be handled.

score 0 · Answer 3 · edited May 23 '17 at 10:30

Two changes seem to fix this:

Provide an stdout and stderr pipe.
```
subprocess.Popen(['git', '--no-pager', 'diff', '--stat', '--cached', 'origin/master'], stdout=subprocess.PIPE, stderr=subprocess.PIPE).wait()
```
However, after making this change DO NOT USE WAIT:

Warning This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.^{from the docs}

Instead...

Use communicate():

p = subprocess.Popen(['git', '--no-pager', 'diff', '--stat', '--cached', 'origin/master'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
print p.returncode

(can also use stderr=subprocess.STDOUT to merge stderr with stdout)

I guess git is doing something like the following to decide whether to use a pager or not: How do I detect whether sys.stdout is attached to terminal or not?

Handling pagers with python's subprocess

3 Answers3