Automatically kill process that consume too much memory or stall on linux

Question

I would like a "system" that monitors a process and would kill said process if:

• the process exceeds some memory requirements
• the process does not respond to a message from the "system" in some period of time

I assume this "system" could be something as simple as a monitoring process? A code example of how this could be done would be useful. I am of course not averse to a completely different solution to this problem.

Answer 1

For the first requirement, you might want to look into either using ulimit, or tweaking the kernel OOM-killer settings on your system.

Monitoring daemons exist for this sort of thing as well. God is a recent example.

Answer 2

I wrote a script that runs as a cron job and can be customized to kill problem processes:

#!/usr/local/bin/perl

use strict;
use warnings;
use Proc::ProcessTable;

my $table = Proc::ProcessTable->new;

for my $process (@{$table->table}) {
    # skip root processes
    next if $process->uid == 0 or $process->gid == 0;

    # skip anything other than Passenger application processes
    #next unless $process->fname eq 'ruby' and $process->cmndline =~ /\bRails\b/;

    # skip any using less than 1 GiB
    next if $process->rss < 1_073_741_824;

    # document the slaughter
    (my $cmd = $process->cmndline) =~ s/\s+\z//;
    print "Killing process: pid=", $process->pid, " uid=", $process->uid, " rss=", $process->rss, " fname=", $process->fname, " cmndline=", $cmd, "\n";

    # try first to terminate process politely
    kill 15, $process->pid;

    # wait a little, then kill ruthlessly if it's still around
    sleep 5;
    kill 9, $process->pid;
}

https://www.endpointdev.com/blog/2012/08/automatically-kill-process-using-too/

Answer 3

To limit memory usage of processes, check /etc/security/limits.conf

Answer 4

Try Process Resource Monitor for a classic, easy-to-use process monitor. Code available under the GPL.

There's a few other monitoring scripts there you might find interesting too.

Answer 5

If you want to set up a fairly comprehensive monitoring system, check out monit. It can be very chatty at times, but it will do a lot of monitoring, restart services, alert you, etc.

That said, don't be surprised if you're getting dozens of e-mails a day until you get used to configuring it and telling it what not to bug you about.

Answer 6

I have a shell script here that could be your start point. I did it because I also had some issues with processes exceeding memory limit. Actually it just checks a given limit of CPU usage, but you can easily change to watch memory, or the jobs list for an idle process.

file: pkill.sh

#!/bin/bash

if [ -z "$1" ]
  then
    maxlimit=99
else
  maxlimit=$1
fi

ps axo user,%cpu,pid,vsz,rss,uid,gid --sort %cpu,rss\
| awk -v max=$maxlimit '$6 != 0 && $7 != 0 && $2 > max'\
| awk '{print $3}'\
| while read line;\
    do\
      ps u --no-headers -p $line;\
      echo "$(date) - $(ps u --no-headers -p $line)" >> pkill.log;\
      notify-send 'Killing proccess!' $(ps -p $line -o command --no-headers | awk '{print $1}') -u normal -i dialog-warning -t 3000;\
      kill $line;\
  done;

Simple run it once like: sh ./pkill.sh <limit-cpu>

Or, to keep it running: watch -n 10 sh ./pkill.sh 90

In the case above it will keep running each 10 seconds, killing processes that exceeds 90% of CPU

Answer 7

Are the monitored processes ones you're writing, or just any process?

If they're arbitrary processes then it might be hard to monitor for responsiveness. Unless the process is already set up to handle and respond to events that you can send it, then I doubt you'll be able to monitor them. If they're processes that you're writing, you'd need to add some kind of message handling that you can use the check against.