Python Eclipse threaded subprocess.Popen()

Question

I am running python 2.7 on Ubuntu in Eclipse

I am trying to call subprocess.Popen from a thread other than the main thread.

When I run this code from Eclipse:

#lsbt.py

class someThread(threading.Thread):

    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        p = subprocess.Popen(["ls", "/usr"], stdout=subprocess.PIPE)
        out = p.communicate()
        print "Done" + out[0]

def main():

    test = someThread()
    test.daemon = True
    test.start()

    while True:
        time.sleep(3600)

The whole python program seems to exit at the subprocess.Popen() line.

Here is what eclipse says the call stack looks like:

<terminated>lsbt_1 lsbt.py [Python Run] 
    <terminated>lsbt.py 
    lsbt.py 
    <terminated, exit value: 137>lsbt.py

All debugging seems to stop as well and nothing is printed to the console.

When I run the subprocess code from the main thread in Eclipse, it seems to work well.

It does not seem to matter what command the subprocess.Popen runs, then only thing that seems to matter is that it is not being run from the main thread.

When I run the python code from the terminal, it works.

Could it be a problem with Eclipse?

@aabarnert commented: IIRC, errno 137 on linux is ENOTTY

Found a similar question here: http://stackoverflow.com/questions/984941/python-subprocess-popen-from-a-thread The asker @noahd says it was a problem with his environment in one of the comments, but does not specify the details. Still looking — ChrisVollo, Oct 07 '14 at 22:20
IIRC, errno 137 on linux is `ENOTTY`. In other words, for some reason, your subprocess thinks it's inherited a TTY, but hasn't, so when it tries to use TTY-specific ioctl's or similar, everything goes to hell. So… maybe it's something to do with Eclipse's virtual terminal? Does the same thing happen if you run the program from bash under gnome-terminal or whatever instead of from Eclipse? — abarnert, Oct 07 '14 at 22:35
Thanks. @abarnert. The same thing does not happen on my Raspberry Pi when I run the script from bash. Still looking for the solution, but I think this pointed me on a better path. — ChrisVollo, Oct 08 '14 at 13:28
If I set the daemon to false, it will also work. According to Pyhon's documentation: `A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property` — ChrisVollo, Oct 08 '14 at 14:29
As far as the TTY issue goes, some things that might be worth trying: (1) use `os.forkpty` or `pty.fork` followed by one of the `os.exec*` functions, instead of `subprocess`, to launch the child, so it actually _does_ have a TTY to write to. (2) `print sys.stdout.isatty()` and see what it says in Eclipse vs. actual TTYs vs. running your main program as a `subprocess`. (The latter two should be yes and no; the first is the real question.) (3) Try running some `termios` (or, better, the third-party `termcap`) code in Python inside Eclipse and see if you get `ENOTTY` exceptions. — abarnert, Oct 08 '14 at 18:39
Something I saw recently made me change my mind about the exit code 137. It actually may not be `ENOTTY`. It actually might be something like a `SIGTERM` or `SIGKILL`. — ChrisVollo, Oct 21 '14 at 05:34
IIRC, on linux, the exit status is a 16-bit number that contains an 8-bit exit code, a 6-bit signal number, a core-dumped flag, and a stopped-vs.-terminated flag. I suppose "exit value" could mean the exit status instead of the exit code, in which case signal 9 (SIGKILL) and the core dump flag could be 137 (it's 128 + 9). It seems a bit misleading to display that instead of, e.g., ``, but I guess it's not impossible. — abarnert, Oct 21 '14 at 18:25

score 0 · Answer 1 · answered Oct 08 '14 at 18:06

0

One way to do it is to set:

daemon = False

I'm not sure why this works for Eclipse, but it does.

From Python Documentation:

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property

answered Oct 08 '14 at 18:06

ChrisVollo

157
1
3
15

This shouldn't be relevant, given your `while True: time.sleep(3600)` on the main thread; as long as the main thread doesn't exit, daemon threads won't be killed by the OS. It would probably be better design to not daemonize the thread—and to replace the sleep loop with a `test.join()`. But that shouldn't actually _fix_ anything. – abarnert Oct 08 '14 at 18:36
@abarnert , Thank you! I agree that it shouldn't actually _fix_ anything, but for some reason it has stopped my code from crashing when running in eclipse. Perhaps daemon threads are not assigned a TTY in eclipse? Perhaps I will not daemonize the threads; this could be a better design pattern? – ChrisVollo Oct 16 '14 at 13:14
I'm not sure I'd call joinable threads a "better design pattern", just a different one. For example, if you have a bunch of background work that's broken down into safely atomic pieces, and it's easily resumable on next launch, putting it on daemon threads makes your application's shutdown a lot simpler. In this particular case, I don't think you want daemons, and if using them when you don't need to is causing problems you don't otherwise have, it may be better to just stop. I'd still want to understand the problem, but sometimes you have more pressing concerns. – abarnert Oct 16 '14 at 17:57

Python Eclipse threaded subprocess.Popen()

1 Answers1