Grep and tail on a huge file with process piping using .Net Core and System.Diagnostics.Process

Question

I want to grep a log file - which can be wery huge, often >= 600mb - and then tail the output to get exactly n lines. To be more precise I expect to obtain the same output as if I type this command on the Windows/Linux prompt:

grep.exe -i "STRING" "PATH_TO_FILE" | tail.exe -n LINES_TO_TAIL

I'm a C# and .Net (Core) newbie so be kind with me. The following code is my attempt of getting something useful:

executableName = @"PATH_TO\grep.exe";
executableArgs = String.Format(@"-i {0} {1} | PATH_TO\\tail.exe -n {2}", paramList["Text"], moduleInfo.GetValue("LogPath"), paramList["MaxLines"]);

var processStartInfo = new ProcessStartInfo
{
    FileName = executableName,
    Arguments = executableArgs,
    RedirectStandardOutput = true
};

Process process = Process.Start(processStartInfo);
string output = process.StandardOutput.ReadToEnd();

process.WaitForExit();

I'm pretty sure that the problem could be the fact that using | in the raw argument list is't the correct way to handle the pipe between two processes. If I set UseShellExecute to trueI get an exception with a message like "UseShellExecute" must always be false. Is there a reasonable and efficient way, even without using external executables, to obtain what I expect?

The problem is that `executableArgs` is going to be passed as arguments, it's not like writing that in the console. You need to create another process for `tail` and give the input to it as the output of the `grep` process. — dcg, Apr 03 '17 at 16:20
I found [this](http://stackoverflow.com/questions/1349543/redirecting-stdout-of-one-process-object-to-stdin-of-another). I think it's exactly what you want. — dcg, Apr 03 '17 at 16:41

Vikhram · Accepted Answer · 2017-04-04T14:34:13.450

If you want to use shell features (metacharacters, pipes, etc), then you should use the shell :)

What I mean to say is you should run cmd.exe /c command_line command to achieve what you want.

var pathToGrep = '"' + @"PATH_TO_GREP\grep.exe" + '"';
var pathToTail = '"' + @"PATH_TO_TAIL\tail.exe" + '"';
var pathToLogFile = '"' + @"PATH_TO_LOG\ganttproject.log" + '"';
var cmd = '"' + $@"{pathToGrep} -i SEARCH_TEXT {pathToLogFile} | {pathToTail} -n 10" + '"';
var processStartInfo = new ProcessStartInfo {
    FileName = @"C:\Windows\System32\cmd.exe",
    Arguments = $@"/c {cmd}",
    CreateNoWindow = true,
    RedirectStandardOutput = true,
    UseShellExecute = false
};

using (var process = Process.Start(processStartInfo)) {
    process.WaitForExit();
    var output = process.StandardOutput.ReadToEnd();
}

Take special care of quotes and related head-aches

BTW if the actual need is as simple as the commands you specified, then implementing it directly in C# is a lot easier. I provided the solution assuming, you might end up using this with other commands or potentially more command-line arguments or both.

Adding code to do grep and tail in C#.

This is just a simple implementation. You can benchmark and check if this gets you the desired performance. If not, you can happily stay with the external tools.

class TailQueue<T> {
    readonly T[] Buffer;
    bool Full = false;
    int Head = -1;

    public TailQueue(int quesize) {
        Buffer = new T[quesize];
    }

    public void Enqueue(T elem) {
        Head = Head == Buffer.Length - 1 ? 0 : Head + 1;
        Buffer[Head] = elem;
        if (Head == Buffer.Length - 1)
            Full = true;
    }

    public IEnumerable<T> GetAll() {
        if (Head == -1)
            yield break;

        var startIndex = 0;
        if (Full && Head != Buffer.Length - 1)
            startIndex = Head + 1;

        for (int i = startIndex; i <= Head; i = (i + 1) % Buffer.Length) {
            yield return Buffer[i];
        }
    }
}

static IEnumerable<string> GrepTail(string filePath, string expression, int lineCount) {
    var lineQ = new TailQueue<string>(lineCount);

    foreach (var line in File.ReadLines(filePath)) {
        if (line.IndexOf(expression, StringComparison.OrdinalIgnoreCase) != -1)
            lineQ.Enqueue(line);
    }

    return lineQ.GetAll();
}

"then implementing it directly in C# is a lot easier" what about performance? Will a "native" C# implementation be at least fast as using the original grep? — fitzbutz, Apr 04 '17 at 07:10
I haven't benchmarked, but I can whip up some code for you to try, if you wish — Vikhram, Apr 04 '17 at 12:44

Grep and tail on a huge file with process piping using .Net Core and System.Diagnostics.Process

1 Answers1