I can't see the output of a subprocess in Python

Question

I'm making a program that interacts with the Sphinx-Quickstart. So what I want to do is that my program recognises the word "path" and then enter a specific value. The same case with the others, and when I don't have a specific word, simply enter a enter ('\n'). I do this because Sphinx sometimes changes the order of the questions and if I use a communicate they can fail.

I think about something like this:

 import subprocess
 from subprocess import PIPE
p = subprocess.Popen('sphinx-quickstart', stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=PIPE, shell=True, bufsize=0)
p.stdin.write('doc\n')
a=p.stdout.readline()
print a.read()
while out:
   line = out
   line = line.rstrip("\n")

   if "autodoc" in line:
    pr = 1
    p.stdin.write('y\n')
    out = p.stdout.readline()
    continue

   if "source and build" in line:
    pr = 2
    p.stdin.write('y\n')
    out = p.stdout.readline()
    continue

out = p.stdout.readline()
p.stdin.close()
p.wait()

My program hangs when I try to read the output.

Thanks for your questions

Answer 1

You don't see the output because you've redirected both stdout/stderr of the subprocess.

Your program hangs because sphinx-quickstart waits for you to provide some input e.g., pass a newline to accept a default value or input something if there is no default value e.g., for a project name, author values.

Another reason is that there is no newline after : and sphinx-quickstart won't flush the prompt in time when the stdout is redirected.

To fix it: read one character at a time instead of line by line and run python with -u option (or use PYTHONUNBUFFERED envvar) to disable buffering.

Make sure that your script provides a valid input each time you see a prompt line (ends with ':' char) in the output:

#!/usr/bin/env python
from __future__ import print_function
import os
from subprocess import Popen, PIPE, CalledProcessError

answers = {
    'Root path': 'doc',
    'source and build': 'y',
    'autodoc': 'y',
    'Project name': '<Project name>',
    'Author name': '<author>',
    'Project version': '<version>',
}

def iter_chunks(pipe, terminator_char):
    """Yield chunks from *pipe* that end with *terminator_char*."""
    buf = []
    for char in iter(lambda: pipe.read(1), ''):
        buf.append(char)
        if char == terminator_char:
            yield ''.join(buf)
            del buf[:]
    if buf: # last chunk
        yield ''.join(buf)

cmd = ['sphinx-quickstart']
p = Popen(cmd, stdin=PIPE, stdout=PIPE, universal_newlines=True, bufsize=0,
          env=dict(os.environ, PYTHONUNBUFFERED='1'))
with p.stdin, p.stdout: # close pipes at the end
    for chunk in iter_chunks(p.stdout, ':'):
        line = chunk.rpartition('\n')[-1] # get last line
        if line.lstrip().startswith('>') and line.endswith(':'): # found prompt
            answer = next((a for q, a in answers.items() if q in line), '')
            print(answer, file=p.stdin) #NOTE: short write is possible
if p.wait() != 0: # raise on non-zero exit status
    raise CalledProcessError(p.returncode, cmd)

Note: stderr is not redirected

You could also use pexpect module for a conversation-like interaction with an external command, example.

Answer 2

Not sure exactly what you want but this will run and look for the correct substrings:

from subprocess import PIPE,Popen,STDOUT

p = Popen('sphinx-quickstart', stdout=PIPE, stderr=STDOUT, stdin=PIPE, universal_newlines=True)
p.stdin.write('doc\n')
for line in iter(p.stdout.readline, ""):
    print(line.rstrip())
    if "autodoc" in line:
        pr = 1
        p.stdin.write('y\n')
    elif '"source" and "build"' in line:
        pr = 2
        p.stdin.write('y\n')

When you run the code you will see the output, source and build are wrapped in quotes so your if was never going to work.

for line in iter(p.stdout.readline, "") will read the output in real time replacing your while loop.

Output:

Welcome to the Sphinx 1.2.2 quickstart utility.

Please enter values for the following settings (just press Enter to
accept a default value, if one is given in brackets).

Enter the root path for documentation.
> Root path for the documentation [.]:
You have two options for placing the build directory for Sphinx output.
Either, you use a directory "_build" within the root path, or you separate
"source" and "build" directories within the root path.
> Separate source and build directories (y/n) [n]:
Inside the root directory, two more directories will be created; "_templates"
for custom HTML templates and "_static" for custom stylesheets and other static
files. You can enter another prefix (such as ".") to replace the underscore.