Python application becomes non-responsive due to subprocess

Question

I have written a Python application, using Flask, that serves a simple website that I can use to start playback of streaming video on my Raspberry Pi (microcomputer). Essentially, the application allows be to use my phone or tablet as a remote control.

I tested the application on Mac OS, and it works fine. After deploying it to the Raspberry Pi (with the Raspbian variant of Debian installed), it serves the website just fine, and starting playback also works as expected. But, stopping the playback fails.

Relevant code is hosted here: https://github.com/lcvisser/mlbviewer-remote/blob/master/remote/mlbviewer-remote.py

The subprocess is started like this:

cmd = 'python2.7 mlbplay.py v=%s j=%s/%s/%s i=t1' % (team, mm, dd, yy)
player = subprocess.Popen(cmd, shell=True, bufsize=-1, cwd=sys.argv[1])

This works fine.

The subprocess is supposed to stop after this:

player.send_signal(signal.SIGINT)
player.communicate()

This does work on Mac OS, but it does not work on the Raspberry Pi: the application hangs until the subprocess (started as cmd) is finished by itself. It seems like SIGINT is not sent or not received by the subprocess.

Any ideas?

---

(I have posted this question also here: https://unix.stackexchange.com/questions/133946/application-becomes-non-responsive-to-requests-on-raspberry-pi as I don't know if this is an OS problem or if it a Python/Flask-related problem.)

UPDATE: Trying to use player.communicate() as suggested by Jan Vlcinsky below (and after finally seeing the warning here) did not help.

I'm thinking about using the proposed solution by Jan Vlcinsky, but if Flask does not even receive the request, I don't think that would receive the issue.

UPDATE 2: Yesterday night I was fortunate to have a situation in which I was able to exactly pinpoint the issue. Updated the question with relevant code.

I feel like the solution of Jan Vlcinsky will just move the problem to a different application, which will keep the Flask application responsive, but will let the new application hang.

UPDATE 3: I edited the original part of the question to remove what I now know not to be relevant.

UPDATE 4: After the comments of @shavenwarthog, the following information might be very relevant:

On Mac, mlbplay.py starts something like this:

rmtpdump <some_options_and_url> | mplayer -

When sending SIGINT to mlbplay.py, it terminates the process group created by this piped command (if I understood correctly).

On the Raspberry Pi, I'm using omxplayer, but to avoid having to change the code of mlbplay.py (which is not mine), I made a script called mplayer, with the following content:

#!/bin/bash

MLBTV_PIPE=mlbpipe

if [ ! -p $MLBTV_PIPE ]
then
    mkfifo $MLBTV_PIPE
fi

cat <&0 > $MLBTV_PIPE | omxplayer -o hdmi $MLBTV_PIPE

I'm now guessing that this last line starts a new process group, which is not terminated by the SIGINT signal and thus making my app hang. If so, I should somehow get the process group ID of this group to be able to terminate it properly. Can someone confirm this?

UPDATE 5: omxplayer does handle SIGINT:

https://github.com/popcornmix/omxplayer/blob/master/omxplayer.cpp#L131

UPDATE 6: It turns out that somehow my SIGINT transforms into a SIGTERM somewhere along the chain of commands. SIGTERM is not handled properly by omxplayer, which appears to be the problem why things keep hanging. I solved this by implementing a shell script that manages the signals and translates them to proper omxplayer commands (sort-of a lame version of what Jan suggested).

SOLUTION: The problem was in player.send_signal(). The signal was not properly handled along the chain of commands, which caused the parent app to hang. The solution is to implement wrappers for commands that don't handle the signals well.

In addition: used Popen(cmd.split()) rather than using shell=True. This works a lot better when sending signals!

Answer 1

The problem is marked in following snippet:

@app.route('/watch/<year>/<month>/<day>/<home>/<away>/')
def watch(year, month, day, home, away):
    global session
    global watching
    global player

    # Select video stream
    fav = config.get('favorite')
    if fav:
        fav = fav[0] # TODO: handle multiple favorites
        if fav in (home, away):
            # Favorite team is playing
            team = fav
        else:
            # Use stream of home team
            team = home
    else:
        # Use stream of home team
        team = home

    # End session
    session = None

    # Start mlbplay
    mm = '%02i' % int(month)
    dd = '%02i' % int(day)
    yy = str(year)[-2:]
    cmd = 'python2.7 mlbplay.py v=%s j=%s/%s/%s' % (team, mm, dd, yy)
    # problem is here ----->
    player = subprocess.Popen(cmd, shell=True, cwd=sys.argv[1])
    # < ------problem is here

    # Render template
    game = {}
    game['away_code'] = away
    game['away_name'] = TEAMCODES[away][1]
    game['home_code'] = home
    game['home_name'] = TEAMCODES[home][1]
    watching = game
    return flask.render_template('watching.html', game=game)

You are starting up new process for executing shell command, but do not wait until it completes. You seem to rely on a fact, that the command line process itself is single one, but your frontend is not taking care of it and can easily start another one.

Another problem could be, you do not call player.communicate() and your process could block if stdout or stderr get filled by some output.

Proposed solution - split process controller from web app

You are trying to create UI for controlling a player. For this purpose, it would be practical splitting your solution into frontend and backend. Backend would serve as player controller and would offer methods like

• start
• stop
• nowPlaying

To integrate front and backend, multiple options are available, one of them being zerorpc as shown here: https://stackoverflow.com/a/23944303/346478

Advantage would be, you could very easily create other frontends (like command line one, even remote one).

Answer 2

One more piece of the puzzle: proc.terminate() vs send_signal.

The following code forks a 'player' (just a shell with sleep in this case), then prints its process information. It waits a moment, terminates the player, then verifies that the process is no more, it has ceased to be.

Thanks to @Jan Vlcinsky for adding the proc.communicate() to the code.

(I'm running Linux Mint LMDE, another Debian variation.)

source

# pylint: disable=E1101

import subprocess, time

def show_procs(pid):
    print 'Process Details:'
    subprocess.call(
        'ps -fl {}'.format(pid),
        shell=True,
    )

cmd = '/bin/sleep 123'
player = subprocess.Popen(cmd, shell=True)

print '* player started, PID',player.pid
show_procs(player.pid)

time.sleep(3)

print '\n*killing player'
player.terminate()
player.communicate()

show_procs(player.pid)

output

* player started, PID 20393
Process Details:
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY        TIME CMD
0 S johnm    20393 20391  0  80   0 -  1110 wait   17:30 pts/4      0:00 /bin/sh -c /bin/sleep 123

*killing player
Process Details:
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY        TIME CMD