Handling interactive shells with Python subprocess

Question

I am trying to run multiple instances of a console-based game (dungeon crawl stone soup -- for research purposes naturally) using a multiprocessing pool to evaluate each run.

In the past when I've used a pool to evaluate similar code (genetic algorithms), I've used subprocess.call to split off each process. However, with dcss being quite interactive having a shared subshell seems to be problematic.

I have the code I normally use for this kind of thing, with crawl replacing other applications I've thrown a GA at. Is there a better way to handle highly-interactive shells than this? I'd considered kicking off a screen for each instance, but thought there was a cleaner way. My understanding was that shell=True should be spawning a sub-shell, but I guess I it is spawning one in a way that is shared between each call.

I should mention I have a bot running the game, so I don't want any actual interaction from the user's end to occur.

# Kick off the GA execution
pool_args = zip(trial_ids,run_types,self.__population)
pool.map(self._GAExecute, pool_args)

---

# called by pool.map 
def _GAExecute(self,pool_args):
  trial_id       = pool_args[0]
  run_type       = pool_args[1]
  genome         = pool_args[2]
  self._RunSimulation(trial_id)

# Call the actual binary
def _RunSimulation(self, trial_id):
  command = "./%s" % self.__crawl_binary
  name    = "-name %s" % trial_id
  rc      = "-rc %s" % os.path.join(self.__output_dir,'qw-%s'%trial_id,"qw -%s.rc"%trial_id)
  seed    = "-seed %d" % self.__seed
  cdir    = "-dir %s" % os.path.join(self.__output_dir,'qw-%s'%trial_id)

  shell_command = "%s %s %s %s %s" % (command,name,rc,seed,cdir)
  call(shell_command, shell=True)

score 3 · Accepted Answer · edited May 23 '17 at 12:26

You can indeed associate stdin and stdout to files, as in the answer from @napuzba:

fout = open('stdout.txt','w')
ferr = open('stderr.txt','w')
subprocess.call(cmd, stdout=fout, stderr=ferr)

Another option would be to use Popen instead of call. The difference is that call waits for completion (is blocking) while Popen is not, see What's the difference between subprocess Popen and call (how can I use them)?

Using Popen, you can then keep stdout and stderr inside your object, and then use them later, without having to rely on a file:

p = subprocess.Popen(cmd,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.wait()
stderr = p.stderr.read()
stdout = p.stdout.read()

Another potential advantage of this method is that you could run multiple instances of Popen without waiting for completion instead of having a thread pool:

processes=[
  subprocess.Popen(cmd1,stdout=subprocess.PIPE, stderr=subprocess.PIPE),
  subprocess.Popen(cmd2,stdout=subprocess.PIPE, stderr=subprocess.PIPE),
  subprocess.Popen(cmd3,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
]

for p in processes:
  if p.poll():
     # process completed
  else:
     # no completion yet

On a side note, you should avoid shell=True if you can, and if you do not use it Popen expects a list as a command instead of a string. Do not generate this list manually, but use shlex which will take care of all corner cases for you, eg.:

Popen(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)

That is helpful, but I'm pretty sure I need the thread pool to be blocking. It is running in a GA, so evaluation of all population members must complete before the next iteration of the controlling loop is called. — erik, Apr 25 '17 at 12:15
It is/can be blocking thanks to the `poll()` in the for loop. The interest of this solution is that you do not actually need to manage threads/threadpool yourself to start runs of subprocesses. — Guillaume, Apr 25 '17 at 13:03

napuzba · Answer 2 · 2017-04-25T05:46:00.713

1

Specify the standard input, standard output and standard error with unique file handles for each call:

import subprocess
cmd  = ""
fout = open('stdout.txt','w')
fin  = open('stdin.txt','r')
ferr = open('stderr.txt','w')
subprocess.call(cmd, stdout=fout , stdin = fin , stderr=ferr )

edited Apr 25 '17 at 05:46

answered Apr 24 '17 at 19:09

napuzba

6,033
3
21
32

Would I actually be able to use that to capture a ttyrec? – erik Apr 25 '17 at 06:09
Well, I am not familiar with ttyrec. However, You can supply the input from `fin` which can be file object or any file-like object. – napuzba Apr 25 '17 at 06:18
If you work in linux - You can try using /dev/tty* files – napuzba Apr 25 '17 at 06:32
Do I need to close the handles or will subprocess handle that? – erik Apr 25 '17 at 12:48

Handling interactive shells with Python subprocess

2 Answers2