Python subprocess communicate freezes when reading output

Question

I'm using Gphoto2 to take pictures on a DSLR. As its based on bash commands I tried to use subprocess.communicate but it freezes after the camera takes a picture.

If I try the gphoto2 --capture-image-and-download in the terminal it takes less than 2 seconds. I'm working on a Raspberry Pi.

Code:

import subprocess

class Wrapper(object):

    def __init__(self, subprocess):
        self._subprocess = subprocess

    def call(self,cmd):
        p = self._subprocess.Popen(cmd, shell=True, stdout=self._subprocess.PIPE, stderr=self._subprocess.PIPE)
        out, err = p.communicate()
        return p.returncode, out.rstrip(), err.rstrip()


class Gphoto(Wrapper):
    def __init__(self, subprocess):
        Wrapper.__init__(self,subprocess)
        self._CMD = 'gphoto2'

    def captureImageAndDownload(self):
        code, out, err = self.call(self._CMD + " --capture-image-and-download")
        if code != 0:
            raise Exception(err)
        filename = None
        for line in out.split('\n'):
            if line.startswith('Saving file as '):
                filename = line.split('Saving file as ')[1]
        return filename


def main():
    camera = Gphoto(subprocess)

    filename = camera.captureImageAndDownload()
    print(filname)

if __name__ == "__main__":
    main()

If I exit I get this:

Traceback (most recent call last):
  File "test.py", line 39, in <module>
   main()
  File "test.py", line 35, in main
    filename = camera.captureImageAndDownload()
  File "test.py", line 22, in captureImageAndDownload
    code, out, err = self.call(self._CMD + " --capture-image-and-download")
  File "test.py", line 11, in call
    out, err = p.communicate()
  File "/usr/lib/python2.7/subprocess.py", line 799, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 1409, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib/python2.7/subprocess.py", line 1463, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt

Any ideas?

Quote "Read data from stdout and stderr, until end-of-file is reached." - If the process hasn't exited, it will hang waiting for input. Are you sure your process has finished? — Torxed, Jan 04 '17 at 11:45
Im quite sure. I hear the shutter of the camera and a file is generated. Once I let it sit for at least 5mins but still freeze... I tested the process in Terminal and it works without further input. — Dennis, Jan 04 '17 at 11:59
Try something similar to this instead: https://gist.github.com/anonymous/0811f71ca0b290f2625c9ba768bd45fd — Torxed, Jan 04 '17 at 12:19
Thanks this helped. The problem was an existing jpg file in the directory of my project and it asked to overwrite it. When I tested it with the terminal I was in a different directory with no file and it worked fine. I added a function to rename the file and now it works fine even with subprocess.communicate — Dennis, Jan 04 '17 at 13:00

score 5 · Accepted Answer · edited May 23 '17 at 11:46

Based on the comments above, here's what we came up with. The .communicate() call hung the program, and much to my suspicion it was because the executed command didn't exit properly.

One thing you can use to get around this is by manually polling the process if it's finished and print the output as you go along.
Now the gist above was written on the phone so it didn't utelize this properly, but here's an example code you can use to catch the output as you go and manually poll the command.

import subprocess
from time import time
class Wrapper():
    def call(self, cmd):
        p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        O = ''
        E = ''
        last = time()
        while p.poll() is None:
            if time() - last > 5:
                print('Process is still running')
                last = time()
            tmp = p.stdout.read(1)
            if tmp:
                O += tmp
            tmp = p.stderr.read(1)
            if tmp:
                E += tmp
        ret = p.poll(), O+p.stdout.read(), E+p.stderr.read() # Catch remaining output
        p.stdout.close() # Always close your file handles, or your OS might be pissed
        p.stderr.close()
        return ret

Three important things to notice, using shell=True might be bad, unsafe and tricky.
I personally favor it because I rarely handle user input or "unknown variables" when I execute stuff. But a few words of caution - Never use it!

Second being, if you don't need to separate error and regular output, you can also do:

Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

It will give you one less file handle to worry about.

Finally thing, ALWAYS empty the stdout/stderr buffers, and always close them. These two things are important.
If you don't empty them, they might themselves hang the application because they're full and Popen can't put more data in them, so it will wait for you (in best case scenario) to empty them.
Second being not closing those file handles, that might render your OS to run out of possible file handles to open (there's only a certain ammount of collective file handles a OS can have open at any given time, so not closing them could render your OS useless for a bit).

(note: Depending on if you're using Python2 or 3, p.stdout.read() might give you bytes data back, meaning O = '' should be O = b'' instead etc)

Python subprocess communicate freezes when reading output

1 Answers1

Linked