How do you kill a specific process that exceeds CPU usage and runtime limit in Linux?

Question

I have a website (Wordpress site on Ubuntu OS and Apache Server) with special math calculators, many of which utilize python3 scripts to do the main calculations. The flow of data on these calculators is as such:

1.) User inputs numbers into html form, then hits submit button.

2.) PHP function is called, it assigns html user inputs to variables and does exec() on applicable python3 file with those variables (the user inputs are filtered and escapeshellarg is used so all good here).

3.) PHP function returns result of python3 script which is displayed via shortcode on the calculator web page.

The issue I am having is that occasionally the symbolic and numeric computations within my python scripts will hang up indefinitely. As that python3 process keeps running, it starts to use massive CPU and memory resources (big problem during peak traffic hours).

My question is this: is there some way to make a script or program on my server's backend that will kill a process instance of python3 if it has exceeded an arbitrary runtime and CPU usage level? I would like to restrict it only to instances of python3 so that it can't kill something like mysqld. Also, I am OK if it only uses runtime as a kill condition. None of my python scripts should run longer than ~10 seconds under normal circumstances and CPU usage will not be an issue if they don't run longer than 10 seconds.

Does this answer your question? [How to kill a child process after a given timeout in Bash?](https://stackoverflow.com/questions/5161193/how-to-kill-a-child-process-after-a-given-timeout-in-bash) — stark, Jun 25 '20 at 19:56
@stark that may possibly answer my question. Is the bash timeout something that would be called within the PHP exec() or would I need to do more research into making a Bash file that runs separately? — Nick481, Jun 25 '20 at 20:11
Regarding the Bash timeout, I found this thread: https://stackoverflow.com/questions/9419122/exec-with-timeout Using that method, it appears I may be able to implement it by doing something like this for example? $result = exec("timeout {$time} python_script.py "."variable1"."variable2"); — Nick481, Jun 25 '20 at 20:21
Linux has `setrlimit` syscall (and shell provides `ulimit` command)... either of those could be useful for you perhaps? — Ondrej K., Jun 25 '20 at 21:24
@OndrejK. it looks like that could be used in a similar way to the bash timeout. Both are beyond my current level of understanding but I think RLIMIT_CPU could be used to limit the cpu time that the process uses. I'm just not sure if it would apply it directly to the python process. — Nick481, Jun 26 '20 at 01:46

Daniel Ocando · Answer 1 · 2020-06-26T06:37:45.417

You can create another python script to serve as a health checker on your server based on the psutil and os modules.

The following code could serve as a base for your specific needs, notice that what it does is basically check for the PIDs for the python scripts on the script_name_list variable based on the name of the script and kill them after checking if your server's CPU is above some threshold or if the memory available is below some threshold as well.

#!/usr/bin/env python3

import psutil
import os
import signal

CPU_LIMIT = 80 #Change Me
AV_MEM = 500.0 #Change Me
script_name_list = ['script1'] #Put in the name of the scripts

def find_other_scripts_pid(script_list):
    pid_list = []
    for proc in psutil.process_iter(['pid','name', 'cmdline']):
        #this is not the PID of the process referencing this script and therefore we chould check inside the list of script name to kill them
        if proc.info['pid'] != os.getpid() and proc.info['name'] in ['python','python3']:
            for element in proc.info['cmdline']:
               for script_name in script_name_list:
                   if script_name in element:
                       pid_list.append(proc.info['pid'])
    return pid_list

def kill_process(pid):
    if psutil.pid_exists(pid):
        os.kill(pid,signal.SIGKILL)
    return None


def check_cpu():
    return psutil.cpu_percent(interval=1)


def check_available_memory():
    mem = psutil.virtual_memory()
    return mem.available/(2**(20))


def main():

    cpu_usage = check_cpu()
    av_memory_mb = check_available_memory()
    if cpu_usage > CPU_LIMIT or av_memory_mb < AV_MEM:
        pid_list = find_other_scripts_pid(script_name_list)
        for pid in pid_list:
            kill_process(pid)


if __name__ == "__main__":
    main()

You can afterwards run this script periodically on your server by using a crontab as explained on this post shared within the community.

How do you kill a specific process that exceeds CPU usage and runtime limit in Linux?

1 Answers1