Use python subprocess module like a command line simulator

Question

I am writing a test framework in Python for a command line application. The application will create directories, call other shell scripts in the current directory and will output on the Stdout.

I am trying to treat {Python-SubProcess, CommandLine} combo as equivalent to {Selenium, Browser}. The first component plays something on the second and checks if the output is expected. I am facing the following problems

1. The Popen construct takes a command and returns back after that command is completed. What I want is a live handle to the process so I can run further commands + verifications and finally close the shell once done
2. I am okay with writing some infrastructure code for achieveing this since we have a lot of command line applications that need testing like this.

Here is a sample code that I am running

 p = subprocess.Popen("/bin/bash", cwd = test_dir)
 p.communicate(input = "hostname") --> I expect the hostname to be printed out
 p.communicate(input = "time") --> I expect current time to be printed out 

but the process hangs or may be I am doing something wrong. Also how do I "grab" the output of that sub process so I can assert that something exists?

Answer 1

subprocess.Popen allows you to continue execution after starting a process. The Popen objects expose wait(), poll() and many other methods to communicate with a child process when it is running. Isn't it what you need?

See Popen constructor and Popen objects description for details.

Here is a small example that runs Bash on Unix systems and executes a command:

from subprocess import Popen, PIPE
p = Popen (['/bin/sh'], stdout=PIPE, stderr=PIPE, stdin=PIPE)
sout, serr = p.communicate('ls\n')
print 'OUT:'
print sout
print 'ERR:'
print serr

UPD: communicate() waits for process termination. If you do not need that, you may use the appropriate pipes directly, though that usually gives you rather ugly code.

UPD2: You updated the question. Yes, you cannot call communicate twice for a single process. You may either give all commands you need to execute in a single call to communicate and check the whole output, or work with pipes (Popen.stdin, Popen.stdout, Popen.stderr). If possible, I strongly recommend the first solution (using communicate).

Otherwise you will have to put a command to input and wait for some time for desired output. What you need is non-blocking read to avoid hanging when there is nothing to read. Here is a recipe how to emulate a non-blocking mode on pipes using threads. The code is ugly and strangely complicated for such a trivial purpose, but that's how it's done.

Another option could be using p.stdout.fileno() for select.select() call, but that won't work on Windows (on Windows select operates only on objects originating from WinSock). You may consider it if you are not on Windows.

Answer 2

Instead of using plain subprocess you might find Python sh library very useful:

http://amoffat.github.com/sh/

Here is an example how to build in an asynchronous interaction loop with sh:

http://amoffat.github.com/sh/tutorials/2-interacting_with_processes.html

Another (old) library for solving this problem is pexpect:

http://www.noah.org/wiki/pexpect