3

I have a python script that opens a .exe program using the subprocess module. This .exe program is an infinitely iterative script, in that it will continue to print the results of each iteration until the user closes the window. Every so often, it prints the results of the iteration into a file, replacing the previous data in the file.

My aims here are to:

  1. Run the .exe program, and test for the existence of the file it outputs.
  2. Once the file has been shown to exist, I need to run a test on the file to see if the iteration has converged to within a given tolerance. Once the iteration has converged, I need to kill the .exe subprocess.

This is my current code. It is designed to kill the subprocess once the iterate file has been created:

import subprocess
from subprocess import Popen, PIPE

fileexists = False

iteratecomms = Popen('iterate.exe', stdout=PIPE, stdin=PIPE, stderr=PIPE)

# Begin the iteration. Need to select options 1 and then 1 again at program menu
out, err = iteratecomms.communicate("1\n1\n".encode())

while (fileexists == False):
    fileexists = os.path.exists(filelocation)
else:
    Popen.kill(iteratecomms)

I know that this is incorrect; the issue is that as soon as I start the out, err = iteratecomms.communicate("1\n1\n".encode()) line, the program begins iterating, and does not move on to the next set of python code. Essentially, I need to start the .exe program, and at the same time test to see if the file has been created. I can't do this, however, because the program runs indefinitely.

How could I get around this? I have assumed that moving on to step 2 (testing the file and killing the subprocess under certain conditions) would not take too much work on top of this; if this is not true, how would I go about completing all of my aims?

Thank you very much for the help!

Edit: Clarified that the external file is overwritten.

Bixx
  • 139
  • 2
  • 8
  • 1
    Avoid getting hung up on the `communicate("1\n1\n".encode())`, by calling `Popen.stdin.write("1\n1\n".encode())` instead. – martineau Jul 15 '13 at 20:23
  • That works (when I replace `Popen` with `iteratecomms`); thank you! Would you have any idea how to continuously print the output of the .exe program using something like `iteratecomms.stdout.read()`? – Bixx Jul 15 '13 at 20:31
  • Off the top of my head, I'd set up a separate thread that continuously read lines via `iteratecomms.stdout.read()` and put them into a [`Queue`](http://docs.python.org/2/library/queue.html?highlight=queue#Queue) from which the main thread checked and retrieved them. This separate thread could easily be killed when no longer needed. – martineau Jul 15 '13 at 21:09
  • Also, the answers to the question [Non-blocking read on a subprocess.PIPE in python](http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python) might be helpful to you. (or at least give you an idea of what sort of thing to look for). – martineau Jul 16 '13 at 16:09

2 Answers2

1

Assuming that you're trying to continuously trying to read this file I would suggest running a tail on the file in question. This can be done from a separate terminal in any *nix family OS, but otherwise I would check out this article for a Python implementation:

http://code.activestate.com/recipes/157035-tail-f-in-python/

After that if you want to kill the program running you should just be able to call terminate on the process running:

import subprocess
sub = subprocess.popen(#Whatever)

#Do something

sub.terminate()
Slater Victoroff
  • 21,376
  • 21
  • 85
  • 144
  • Apologies; I am a beginner with Python and the subprocess module. I don't quite understand how this would practically apply to my problem! This `tail` uses `file.tell()`, which seems to find the current pointer position in a file; the file I use is replaced after each iteration; i.e. once the file is created, the current iteration data replaces the last set. The problem I am having is establishing when the file is created. Really sorry if I have missed your point! – Bixx Jul 15 '13 at 20:39
  • @FreeBixi wait, I may have misunderstood. Is your iteration continuously deleting and recreating a single file? – Slater Victoroff Jul 15 '13 at 20:49
  • @FreeBixi Omigoodness. If you have control over that code I would **highly** suggest appending instead of rewriting for a billion different reasons. Other than that I would suggest scrolling down on the page for the "last n" version of the tail function that will just let you look at the last line or few lines and figure out whether or not it meets your criteria. If you need to see everything then just change last n to some really big number. – Slater Victoroff Jul 15 '13 at 21:02
  • Unfortunately, I have no control over that code, otherwise I would not be putting myself through this. I also need to consider that I must compare values from the entire output file, not just the last line. Thanks very much for your suggestions; after spending the entire day on this problem, I will try it tomorrow with a fresh pair of eyes! – Bixx Jul 15 '13 at 21:13
  • @FreeBixi If you're going to read the whole thing anyway you should just write to stdout instead of a file. – Slater Victoroff Jul 15 '13 at 21:14
  • I have been having trouble with that too; as the program is iterative and runs continuously, I am unable to print the stdout using something like `print(iteratecomms.stdout.read())`. – Bixx Jul 15 '13 at 21:26
  • Oh, that's not how you should write to stdout. Basically you can just specify `-` as the location and it will go to a stdout. Run it in a different process, then just assign that to a file-like object (StringIO) and use that. – Slater Victoroff Jul 15 '13 at 22:36
1

I would use the multiprocessing module.

pool = multiprocessing.Pool()
def start_iteration():
    return Popen('iterate.exe', stdout=PIPE, stdin=PIPE, stderr=PIPE)
pool.apply_async(start_iteration)
while (fileexists == False):
    fileexists = os.path.exists(filelocation)
Popen.kill(???)

The only problem now is that you'll have to somehow find the PID of the process without waiting for Popen to return (because Popen should never return.)

Chris Barker
  • 2,279
  • 14
  • 15
  • `Popen()` returns an instance of the [`subprocess.Popen`](http://docs.python.org/2/library/subprocess.html?highlight=popen#popen-constructor) class. – martineau Jul 15 '13 at 20:14