Restart ffmpeg process using monit if TOTAL CPU is less than 1%

Question

I have used kind of similar solution like this Restarting ffmpeg process using monit to restart my ffmpeg stream in case it fails for some reason. Remember its not duplicate problem/question, because I have other issues unlike the example question/solution Restarting ffmpeg process using monit, which I'm gonna explain below. So here is my monit configuration:

    check process FFMPEGStream with pidfile PATH-to-file/streampid.pid
    start program = "PATH-to-file/streambash.sh restart"
    stop program = "PATH-to-file/streambash.sh stop"
    if TOTAL CPU is less than 1% for 10 cycles then restart

Here is my streambash.sh file:

    #!/bin/bash
    pid_file="PATH-to-file/streampid.pid"

    case "$1" in
     restart)
        PATH-to-file/streambash.sh stop
        PATH-to-file/streambash.sh start
           ;;

     start)
        rm $pid_file
        /usr/bin/ffmpeg -i "INPUT-PATH" -c:v libx264 -b:v 900k -preset ultrafast -aspect 16:9 -s 640x376 -strict experimental -c:a aac -b:a 96k -f flv "RTMP-PATH" &> /dev/null &
        ch_pid=$! 
        echo "Start Stream1: ffmpeg = $ch_pid";
        echo $ch_pid > $pid_file
           ;;

     stop)
        echo "Stop ffmpeg Stream1";
        kill `cat $pid_file` &> /dev/null
           ;;

     *)
        echo "Usage: PATH-to-file/streambash.sh {start|stop|restart}"
        exit 1
           ;;

     esac
    exit 0
    echo $pid_file

Monit can start the bash file successfully, but when this condition "if TOTAL CPU is less than 1% for 10 cycles then restart" is matched in monit configuration, it tries to restart, it gives error that process is not running. But in actual the ffmpeg process still runs in the background and I can see that the stream is live on my website. Here is monit logs:

    [CET Jan 10 12:55:02] error    : 'FFMPEGStream' total cpu usage of 0.4% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:07] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:12] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:17] error    : 'FFMPEGStream' total cpu usage of 0.4% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:22] error    : 'FFMPEGStream' total cpu usage of 0.9% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:27] error    : 'FFMPEGStream' total cpu usage of 0.9% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:32] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:37] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:42] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:47] error    : 'FFMPEGStream' total cpu usage of 0.4% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:50] info     : 'FFMPEGStream' trying to restart
    [CET Jan 10 12:55:50] info     : 'FFMPEGStream' stop: PATH-to-file/streambash.sh
    [CET Jan 10 12:55:51] info     : 'FFMPEGStream' start: PATH-to-file/streambash.sh
    [CET Jan 10 12:55:56] error    : 'FFMPEGStream' process is not running
    [CET Jan 10 12:55:58] info     : 'FFMPEGStream' trying to restart
    [CET Jan 10 12:55:58] info     : 'FFMPEGStream' start: PATH-to-file/streambash.sh
    [CET Jan 10 12:56:04] error    : 'FFMPEGStream' process is not running
    [CET Jan 10 12:56:04] info     : 'FFMPEGStream' trying to restart
    [CET Jan 10 12:56:04] info     : 'FFMPEGStream' start: PATH-to-file/streambash.sh
    [CET Jan 10 12:56:09] error    : 'FFMPEGStream' process is not running
    [CET Jan 10 12:56:09] info     : 'FFMPEGStream' trying to restart
    [CET Jan 10 12:56:09] info     : 'FFMPEGStream' start: PATH-to-file/streambash.sh

Monit keeps trying to restart the process and on each retry, it dumps a new pid to the PATH-to-file/streampid.pid, but as I said it seems, it somehow can stop the actual ffmpeg stream/pid, which keep running in the background.

My problem is not CPU handling, my problem is process is not restarted or atleast not shown as started in monit if after its not running. CPU usage is just a way that I used to know that the process is not working or ffmpeg is encoding or any related problem. So doesn't make sense to use nice or cgroups. — Ghulam, Jan 10 '17 at 20:34

score 0 · Accepted Answer · answered May 28 '17 at 11:11

0

Your poll cycle / daemon check interval is very low, 5 seconds?

FFMpeg isn't starting within 5 seconds so monit tries to start it again, constantly in a loop.

If you want to have such a low check interval, you need to set a timeout on the start command, along the lines of:

start program = "PATH-to-file/streambash.sh restart" with timeout 30 seconds

This really helps me to understand monit's way of thinking, watch the live log in a terminal window while monit is doing things:

 tail -f /var/log/monit.log

answered May 28 '17 at 11:11

Paul

38
6

Another thing, monit / monit webif is very slow to detect and update cpu usage on a recently started process. Be patient and give it a minute or more! Even thoguh the process started and is using e.g., 10% CPU, monit webif states that the process is still using 0% and restarts the process, in a sometimes never ending loop. – Paul May 31 '17 at 10:52

Restart ffmpeg process using monit if TOTAL CPU is less than 1%

1 Answers1