Automatically shutdown all `Process` instances if the parent thread crashed on Linux

Question

One of my app's thread is responsible for running multiple processes in form of bash commands. It runs in a loop and for each found file runs specified command.

foreach (var file in files)
{
    string withoutExt = Path.GetFileNameWithoutExtension(file);
    string outputFile = Path.GetDirectoryName(file) + "/" + withoutExt + ".png";

    tasks.Add(Task.Run(delegate
    {
        ProcessStartInfo startInfo = new ProcessStartInfo()
            { FileName = "neato", Arguments = $"-Tpng {file} -o {outputFile}",  };

        Process.Start(startInfo)?.WaitForExit();
    }));
}

Task.WaitAll(tasks.ToArray());

Although it does its job, I noticed that those processes are alive after my app is either forcibly closed or when it crashes. I found it out when I've ran this code on too many files: hundreds of processes appeared eating all my resources and when I've finally managed to close an app, they still persisted.

I'm looking for a way to somehow bound those processes to main thread, so they all be cleaned as soon as I close the main thread.

I'm aware that there are already questions asked about such problem on StackOverflow, but as far as I can see, solutions presented there are strictly bound to Windows OS, not Linux.

On Windows, there's a concept called [Job](https://learn.microsoft.com/en-us/windows/win32/procthread/job-objects) — Thomas Weller, Feb 07 '22 at 12:59

Alex O · Answer 1 · 2022-02-16T19:15:31.283

1

One method to ensure that a set of created processes can be killed reliably is to start them in a new pid namespace.

In such a namespace, the "root" processes has pid 1 (init). As soon as that root process gets killed, the Linux kernel will ensure that all other processes in the namespace are killed. This will be independent from the sessions or the process groups which are associated to those processes.

This worked fine for me in a situation where I needed to reliably kill shell processes that I started from Python. Starting sub-processes in a new pid namespace is easy via the unshare shell command.

edited Feb 16 '22 at 19:15

answered Feb 13 '22 at 18:40

Alex O

7,746
2
25
38

In order to unshare pid namespace while unprivileged, one has to unshare user namespace as well. All but the current user id will be seen as nobody in e.g. a directory listing. – Nick Zavaritsky Feb 19 '22 at 19:10

score 0 · Answer 2 · answered Feb 15 '22 at 05:31

Maybe use a try-catch for the crash handling?

{
        List<string> pID = new List<string>();

        try
        {
            foreach (var file in files)
            {
                string withoutExt = Path.GetFileNameWithoutExtension(file);
                string outputFile = Path.GetDirectoryName(file) + "/" + withoutExt + ".png";

                tasks.Add(Task.Run(delegate
                {
                ProcessStartInfo startInfo = new ProcessStartInfo()
                { FileName = "neato", Arguments = $"-Tpng {file} -o {outputFile}",  };
                string pid = Process.Id.ToString();
                pID.Add(pid);
                Process.Start(startInfo)?.WaitForExit();
                }));
            }

            Task.WaitAll(tasks.ToArray());
        }
        catch (Exception e)
        {
            # kill all the processes
            foreach (var pid in pID)
            {
                try
                {
                    Process.GetProcessById(int.Parse(pid)).Kill();
                }
                catch (Exception)
                {
                    // ignore
                }
            }
        }
}

This will indeed kill the processes that were started in the first place. However, this won't reliable terminate child processes that were created from these top-level processes. If Piotrek's bash commands launched complex services/daemons or build processes, chances are that those will persist even after killing the initial shell command. — Alex O, Feb 16 '22 at 19:24
@AlexO I agree. Perhaps we could use a text file in which each subprocess simply writes its set of PIDs there when it gets the chance to, Or we could use separate text files for separate subprocesses in a distinct folder that can then be read and "taken care of" ... — mayank, Feb 17 '22 at 14:54

score 0 · Answer 3 · answered Feb 17 '22 at 19:54

Set process.EnableRaisingEvents = true before starting the process; apparently this is essential in Linux to have the child processes exit along with the parent process.

https://github.com/dotnet/runtime/issues/21661

If you're running in a docker container, make sure to pass --init so child processes are cleaned up.

How to use --init parameter in docker run

score -2 · Answer 4 · answered Feb 15 '22 at 17:45

First, view all of the running processes on the Linux machine:

~$ top

Look check the PID column of the resulting table and find the PID for the process you want to kill, and type the following:

~$ kill <PID of the target process>

However the PID of a program changes every time the process or the machine is restarted. So again with the top command:

~$ top

You can check the COMMAND column on the resulting table, and see if they have any common keywords or if the program that the processes are running on are the same, and enter those keywords into the following command:

~$ pkill -9 <common keyword or program name of the processes>

I don't know much C#, but maybe you can have it in your program so that it executes the above commands while/before/after it is being closed?

Indeed, one can implement something similar to ps/pkill, probably based on /proc. The problem with this is threefold. First, if the main process crashes, child processes are left behind. Second, we won’t be able to identify grand child processes if a child have already terminated as the grandchild will be reparented by pid 0. Lastly, killing processes this way is racy as new ones can spawn while we are nuking existing ones. — Nick Zavaritsky, Feb 19 '22 at 20:39

Automatically shutdown all `Process` instances if the parent thread crashed on Linux

4 Answers4