Interaction between Python script and linux shell

Question

I have a Python script that needs to interact with the user via the command line, while logging whatever is output.

I currently have this:

# lots of code

popen = subprocess.Popen(
    args,
    shell=True,
    stdin=sys.stdin,
    stdout=sys.stdout,
    stderr=sys.stdout,
    executable='/bin/bash')

popen.communicate()

# more code

This executes a shell command (e.g. adduser newuser02) just as it would when typing it into a terminal, including interactive behavior. This is good.

Now, I want to log, from within the Python script, everything that appears on the screen. But I can't seem to make that part work.

I've tried various ways of using subprocess.PIPE, but this usually messes up the interactivity, like not outputting prompt strings.

I've also tried various ways to directly change the behavior of sys.stdout, but as subprocess writes to sys.stdout.fileno() directly, this was all to no avail.

As a side note, is there a reason you're using `shell=True`? Especially when the thing you're launching is itself `/bin/bash`? You're basically telling Python to start a copy of `bash` to tell it to start another copy of `bash` to (presumably, since you didn't show the whole code) tell it to start a copy of `adduser`. You probably don't need even one shell involved, but I can't imagine why you'd need two. — abarnert, Jul 24 '14 at 17:36
@abarnert: `executable` doesn't start a new `bash` process: it tells `subprocess` module to use its value instead of `/bin/sh` if `shell=True`. `args` is presumably some shell command with bash-isms such as `'use_tty &>/dev/null'`. As I understand the intent is: `script -c '/bin/bash -c "$args"' log`. Though as [my answer shows](http://stackoverflow.com/a/22265426/4279), you don't necessarily need even one additional bash process to run a command here. — jfs, Apr 25 '15 at 22:35

score 3 · Answer 1 · edited May 23 '17 at 11:51

Popen might not be very suitable for interactive programs due to buffering issues and due to the fact that some programs write/read directly from a terminal e.g., to retrieve a password. See Q: Why not just use a pipe (popen())?.

If you want to emulate script utility then you could use pty.spawn(), see the code example in Duplicating terminal output from a Python subprocess or in log syntax errors and uncaught exceptions for a python subprocess and print them to the terminal:

#!/usr/bin/env python
import os
import pty
import sys

with open('log', 'ab') as file:
    def read(fd):
        data = os.read(fd, 1024)
        file.write(data)
        file.flush()
        return data

    pty.spawn([sys.executable, "test.py"], read)

Or you could use pexpect for more flexibility:

import sys
import pexpect # $ pip install pexpect

with open('log', 'ab') as fout:
    p = pexpect.spawn("python test.py")
    p.logfile = fout # or .logfile_read
    p.interact()

If your child process doesn't buffer its output (or it doesn't interfere with the interactivity) and it prints its output to its stdout or stderr then you could try subprocess:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE, STDOUT

with open('log','ab') as file:
    p = Popen([sys.executable, '-u', 'test.py'],
              stdout=PIPE, stderr=STDOUT,
              close_fds=True,
              bufsize=0)
    for c in iter(lambda: p.stdout.read(1), ''):
        for f in [sys.stdout, file]:
            f.write(c)
            f.flush()
    p.stdout.close()
    rc = p.wait()

To read both stdout/stderr separately, you could use teed_call() from Python subprocess get children's output to file and terminal?

score 0 · Answer 2 · answered Mar 05 '14 at 10:23

0

This should work

import subprocess
f = open('file.txt','w')
cmd = ['echo','hello','world']
subprocess.call(cmd, stdout=f)

answered Mar 05 '14 at 10:23

rogue-one

11,259
7
53
75

1

This doesn't show the output in the console. Nor does this work with commands that require user input, such as 'adduser' – HetMes Mar 05 '14 at 10:46
Your question suggested that you want the output to be logged into a file. Do you want it to be logged in both file and in the console? – rogue-one Mar 05 '14 at 10:53
Yes, the idea is that at the end of the python script, I have a complete log of everything that happened, including any logging explicitly generated by the script itself. – HetMes Mar 05 '14 at 12:53
1

@user480441: `adduser` might expect the input directly from terminal. Otherwise the child process started by `call()` should inherit its parent stdin i.e., you should be able to provide input e.g., try `subprocess.call('python')`. – jfs Mar 08 '14 at 05:33

Interaction between Python script and linux shell

2 Answers2

Linked