Mono crashes when running too many processes

Question

I want to build a video converting queue, so servers running a mono program, extracts a video from queue and runs ffmpeg to convert it. When running many ffmpeg processses mono gives me error.

So i created a test case with ping because it's a long running one as ffmpeg.

    static TextWriter tw = new StreamWriter("error.txt");
    private static int _started;
    public static void Main (string[] args)
    {

        for (int i=0; i<1000; i++) {
            Start ("ping", "localhost");

        }
        Console.WriteLine("Memory: " + (GC.GetTotalMemory(true) / 1024) + "KB");
        Console.WriteLine(_started);
        tw.Flush();
        Console.ReadLine();
    }

    public static Process Start (string name, string args)
    {
        try {
            var proc = new ProcessStartInfo {
                    FileName = name,
                    Arguments = args,
                    RedirectStandardOutput = true,
                    CreateNoWindow = false,
                    RedirectStandardError = false,
                    UseShellExecute = false,
                    RedirectStandardInput = false,
            };
            Process.Start (proc);
            _started++;
        } catch (Exception ex) {
            tw.WriteLine (ex.InnerException+Environment.NewLine+ex.Message + Environment.NewLine + ex.StackTrace, false);
            tw.Flush();
        }
        return null;
    }

And when running, the same error throws as in the case with ffmpeg

ApplicationName='ping', CommandLine='localhost', CurrentDirectory=''
  at System.Diagnostics.Process.Start_noshell
(System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process
process) [0x00000] in <filename unknown>:0 
  at System.Diagnostics.Process.Start_common
(System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process
process) [0x00000] in <filename unknown>:0 
  at System.Diagnostics.Process.Start (System.Diagnostics.ProcessStartInfo
startInfo) [0x00000] in <filename unknown>:0 
  at test1.MainClass.Start (System.String name, System.String args) [0x00000]
in <filename unknown>:0

and

CreateProcess: error creating process handle

So that's my environment:

Mono JIT compiler version 2.10.8.1 (Debian 2.10.8.1-5ubuntu1)
Copyright (C) 2002-2011 Novell, Inc, Xamarin, Inc and Contributors. www.mono-project.com
    TLS:           __thread
    SIGSEGV:       altstack
    Notifications: epoll
    Architecture:  amd64
    Disabled:      none
    Misc:          softdebug 
    LLVM:          supported, not enabled.
    GC:            Included Boehm (with typed GC and Parallel Mark)

And limit values

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31439
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31439
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I want to continue my project but i can't because of this anomalies, after working 1 year with mono i really love it, and don't want to learn other language because of this trouble.

I hope there is a problem with my code, not mono runtime.

So my temporary solution:

    public static void Start (string cmd, int time)
    {
        new Thread (p =>
        {
            try {
                Mono.Unix.Native.Syscall.system (String.Format("timeout {0} {1}", time, cmd));
            } catch (Exception ex) {
                //log it
            }
        }
        ).Start ();
    }

Excuse my off-topic question, but how do you control the limit values? I'm especially interested in controlling the stack size: http://stackoverflow.com/questions/14263148/mono-regex-and-stack-size — IneQuation, Feb 20 '13 at 18:23
That _started++ should be [Interlocked.Increment](https://msdn.microsoft.com/en-us/library/dd78zt0c(v=VS.110).aspx) otherwise it'll be inaccurate with threading. — Joseph Lennox, Jul 28 '15 at 15:47

Rolf Bjarne Kvinge · Answer 1 · 2013-02-21T23:37:20.090

3

You must call WaitForExit and Dispose on the Process instance.

So your Start method would look something like this:

public static void Start (string name, string args)
{
    new Thread (() =>
    {
        try {
            var proc = new ProcessStartInfo {
                    FileName = name,
                    Arguments = args,
                    RedirectStandardOutput = true,
                    CreateNoWindow = false,
                    RedirectStandardError = false,
                    UseShellExecute = false,
                    RedirectStandardInput = false,
            };
            using (var p = new Process ()) {
                p.StartInfo = proc;
                p.Start ();
                p.WaitForExit ();
            }
            _started++;
        } catch (Exception ex) {
            tw.WriteLine (ex.InnerException+Environment.NewLine+ex.Message + Environment.NewLine + ex.StackTrace, false);
            tw.Flush();
        }
    }).Start ();
}

edited Feb 21 '13 at 23:37

answered Feb 21 '13 at 13:54

Rolf Bjarne Kvinge

19,253
2
42
86

Could you please test this? On my computer it doesn't work, the same error appears. – croisharp Feb 21 '13 at 17:04
@croisharp: Sorry, I forgot you need to wait on the process on linux. See my updated answer. – Rolf Bjarne Kvinge Feb 21 '13 at 23:37
1

Well it's similar to what the [bug report tells you to do](https://bugzilla.novell.com/show_bug.cgi?id=536776#c4) (and arguably better -- with the `WaitForExit` call) but starting a thread and then waiting on a process that will run for an indeterminate period of time for each process seems a bit failure-prone. – ta.speot.is Feb 21 '13 at 23:43
1

I tested it with Dispose and WaitForExit and after starting 254 processes it gives same error. Now i call directly system() method (see question update) and it works fine after starting 5k processes. I also tested your solution in ubuntu x86 vps without success. – croisharp Feb 22 '13 at 10:21

Mono crashes when running too many processes

1 Answers1