How to get asynchronous input AND output using subprocess.POPEN

Question

I've been working on this for a few hours and haven't been able to come up with a good solution. A little background, I'm running a password cracking program that's closed source from the command line but have to constantly pause it when my gpu temperature gets too hot.

I do other manipulations in python with this program so that's the language I'd prefer. Anyways, the password program gives periodic updates on how well it's doing, the gpu temperature, etc. and allows me to pause it at any time.

I'm getting the temperature fine but because of blocking issues I'm guessing I can't send the pause command. It's not doing anything at least. I've seen several examples of threading the output, but haven't seen something that that uses threading input and output without causing any issues.

I mean for all I know this could be impossible under current POPEN constraints but would appreciate some direction.

popen = Popen(command, stdout=PIPE, stdin=PIPE, shell=True)
lines_iterator = iter(popen.stdout.readline, b"")
while 1:
    for line in lines_iterator:
        cleanLine = line.replace("\n", "")
        p = re.compile('[0-9][0-9]c Temp')
        m = p.search(cleanLine)
        print cleanLine
        if m:
            temperature = m.group(0)
            if int(temperature[:2]) > 80:
                overheating = True
                print "overheating"
        if overheating:
            if "[s]tatus [p]ause [r]esume [b]ypass [q]uit" in line:

                #It's not doing anything right here, it just continues
                print popen.communicate("p")[0]

This is the gist of my code. It's still kind of through the hacky phase so I know that it might not be following best coding practices.

It's OCLhashcat. It's the best gpu cracker I've used to date but my machine will get over 80 degrees every 15 minutes or so and I have to consistently monitor it which is a pain. I just want to automate more of this. — themuffinman, Oct 20 '15 at 09:46

score 1 · Answer 1 · answered Oct 20 '15 at 09:32

1

EDIT: Sorry, I was confused in the initial answer about the scope of overheating. I deleted the first part of my answer since it's not relevant anymore.

communicate will wait for the process to exit so it might not be what you're looking for in this case. If you want the process to keep going you can use something like popen.stdin.write("p"). You might also need to send a "\n" along if that's required by your process.

Also, if you're OK with an extra dependency you might be interested in the pexpect module that was designed to control interactive processes.

answered Oct 20 '15 at 09:32

gbs

1,305
13
12

I actually have no problem with pexpect, I've found it doesn't really work well with Windows though. communicate does look like it blocks, but the program seems to blow over stdin.write and continue running. – themuffinman Oct 20 '15 at 09:45
Try calling `popen.stdin.flush()` right after `write`. Or add `bufsize=0` to the `Popen` call to avoid buffering issues. – gbs Oct 20 '15 at 09:49
Damn, I tried both. Individually and then together. This is driving me crazy. – themuffinman Oct 20 '15 at 10:13
Popen is a mess really, especially on Windows. Try adding `universal_newlines=True` and send "p\n" instead of "p". If that doesn't work, it's possible that you're running into some kind of deadlock with the way you're reading the lines. Try adapting the code from this [answer](http://stackoverflow.com/questions/33088339/input-command-doesnt-seem-to-work-when-used-with-popen-python) and see if it works better for you. – gbs Oct 20 '15 at 10:26
Unfortunately this is the exact kind of stuff I'm finding. In the answer you showed, it actually "finishes" printing, and expects some sort of input. But with this password cracker it doesn't wait for input. – themuffinman Oct 20 '15 at 10:34
I see. Then I think you will save yourself some time by using the actual [Expect](http://docs.activestate.com/activetcl/8.5/expect4win) for Windows. You'll have to write the script in Tcl but it's really not that complicated if you just follow an example like [this](http://docs.activestate.com/activetcl/8.5/expect4win/ex_usage.html#ftp) – gbs Oct 20 '15 at 10:40

score 1 · Answer 2 · edited May 23 '17 at 11:58

1

A simple portable solution is to use threads here. It is enough if there are no block buffering issues.

To read output and stop input if overheating is detected (not tested):

#!/usr/bin/env python
from subprocess import Popen, PIPE, CalledProcessError
from threading import Event, Thread

def detect_overheating(pipe, overheating):
    with pipe: # read output here
        for line in iter(pipe.readline, ''): 
            if detected_overheating(line.rstrip('\n')):
                overheating.set()    # overheating
            elif paused: #XXX global
                overheating.clear()  # no longer overheating

process = Popen(args, stdout=PIPE, stdin=PIPE, bufsize=1,
                universal_newlines=True) # enable text mode
overheating = Event() 
t = Thread(target=detect_overheating, args=[process.stdout, overheating])
t.daemon = True # close pipe if the process dies
t.start()
paused = False
with process.stdin: # write input here
    while process.poll() is None:
        if overheating.wait(1): # Python 2.7+
            # overheating
            if not paused:
                process.stdin.write('p\n') # pause
                process.stdin.flush()
                paused = True
        elif paused: # no longer overheating
            pass #XXX unpause here
            paused = False
if process.wait() != 0: # non-zero exit status may indicate failure
    raise CalledProcessError(process.returncode, args)

edited May 23 '17 at 11:58

Community

1
1

answered Oct 20 '15 at 15:29

jfs

399,953
195
994
1,670

Unfortunately this is similar to other solutions I've found. It prints everything out fine but the stdin.write still isn't working. No errors, runs right over it. – themuffinman Oct 20 '15 at 19:57
@themuffinman: what does it mean *"runs right over it"* -- does the child receive the pause command? Are you expecting that the loop in the parent stops? Does the child accept *anything* on its stdin (via the pipe)? Can you pass any command to the child (regardless synchronously or not)? What is your OS? Are you sure the child does not read/write directly to terminal instead of the pipes? (Update your question with the info) – jfs Oct 20 '15 at 20:03
It looks like it's not accepting anything on its stdin. I've been testing using the quit command for awhile and nothing. It doesn't throw any errors and continues running however. Because the program can continue running without user input, as in it doesn't wait for user input, I can only pass it to the stdin when a specific line is output. But still nothing. – themuffinman Oct 20 '15 at 20:09
@themuffinman: are you on Windows? Can you read anything from `process.stdout`? Do you see any output on the screen if your Python script doesn't print captured output explicitly? Does the child process accept manual keyboard input despite `stdin=PIPE`? Can you pass *any* command using [`SendKeys` library](http://stackoverflow.com/a/12606327/4279)? – jfs Oct 20 '15 at 20:21
Using Windows, everything prints fine, asynchronously, threaded, etc. Never had any issues. Doesn't accept manual keyboard input. I'm looking into SendKeys but don't see a way to use it with subprocess. – themuffinman Oct 20 '15 at 21:45
@themuffinman: if `stdout=PIPE` works then you should not see *anything* (no output). The point of `SendKeys()` is to see whether you can send *any* command (e.g., start and immediately exit). At this point I can't confirm that *anything* works as it should. – jfs Oct 20 '15 at 21:51
Sorry, I'm actively taking the process.stdout and printing it out line by line. I don't think SendKeys() works with subprocess though. – themuffinman Oct 20 '15 at 22:11
@themuffinman: `SendKeys()` doesn't work with subprocess. The purpose is to find out whether you could send any commands (you could redirect the output to a file or a named pipe and read it in another thread similar to the above code) – jfs Oct 20 '15 at 22:17

How to get asynchronous input AND output using subprocess.POPEN

2 Answers2