SSH.NET real-time command output monitoring

Question

There is a long running script script.sh on a remote Linux machine. I need to start it and monitor it's activity in real time. The script during it's activity may output to stdout and stderr. I am searching for a way to capture both of the streams.

I use Renci SSH.NET to upload script.sh and start it, so it would be great to see a solution bounded to this library. In my mind the perfect solution is the new method:

var realTimeScreen= ...;

var commandExecutionStatus = sshClient.RunCommandAsync(
    command: './script.sh',
    stdoutEventHandler: stdoutString => realTimeScreen.UpdateStdout(stdString)
    stderrEventHandler: stderrString => realTimeScreen.UpdateStderr(stderrString));
...
commandExecutionStatus.ContinueWith(monitoringTask =>
{
    if (monitoringTask.Completed)
    {
        realTimeScreen.Finish();
    }
});

Answer 1

Use SshClient.CreateCommand method. It returns SshCommand instance.

The SshCommand class has OutputStream (and Result) for stdout and ExtendedOutputStream for stderr.

See SshCommandTest.cs:

public void Test_Execute_ExtendedOutputStream()
{
    var host = Resources.HOST;
    var username = Resources.USERNAME;
    var password = Resources.PASSWORD;

    using (var client = new SshClient(host, username, password))
    {
        #region Example SshCommand CreateCommand Execute ExtendedOutputStream

        client.Connect();
        var cmd = client.CreateCommand("echo 12345; echo 654321 >&2");
        var result = cmd.Execute();

        Console.Write(result);

        var reader = new StreamReader(cmd.ExtendedOutputStream);
        Console.WriteLine("DEBUG:");
        Console.Write(reader.ReadToEnd());

        client.Disconnect();

        #endregion

        Assert.Inconclusive();
    }
}

---

See also a full code for similar WinForms question Execute long time command in SSH.NET and display the results continuously in TextBox.

Answer 2

So, here is the solution I came up with. Of course, it can be improved, so it is open to critique.
I used

await Dispatcher.Yield(DispatcherPriority.ApplicationIdle);

instead of Task.Yield() because Task.Yield() will make continuation a higher priority than GUI events, but, as a bad consequence, it demands your application to use WindowsBase.dll.

public static class SshCommandExtensions
{
    public static async Task ExecuteAsync(
        this SshCommand sshCommand,
        IProgress<ScriptOutputLine> progress,
        CancellationToken cancellationToken)
    {
        var asyncResult = sshCommand.BeginExecute();
        var stdoutStreamReader = new StreamReader(sshCommand.OutputStream);
        var stderrStreamReader = new StreamReader(sshCommand.ExtendedOutputStream);

        while (!asyncResult.IsCompleted)
        {
            await CheckOutputAndReportProgress(
                sshCommand,
                stdoutStreamReader,
                stderrStreamReader,
                progress,
                cancellationToken);

            await Dispatcher.Yield(DispatcherPriority.ApplicationIdle);
        }

        sshCommand.EndExecute(asyncResult);

        await CheckOutputAndReportProgress(
            sshCommand,
            stdoutStreamReader,
            stderrStreamReader,
            progress,
            cancellationToken);
    }

    private static async Task CheckOutputAndReportProgress(
        SshCommand sshCommand,
        TextReader stdoutStreamReader,
        TextReader stderrStreamReader,
        IProgress<ScriptOutputLine> progress,
        CancellationToken cancellationToken)
    {
        if (cancellationToken.IsCancellationRequested)
        {
            sshCommand.CancelAsync();
        }
        cancellationToken.ThrowIfCancellationRequested();

        await CheckStdoutAndReportProgressAsync(stdoutStreamReader, progress);
        await CheckStderrAndReportProgressAsync(stderrStreamReader, progress);
    }

    private static async Task CheckStdoutAndReportProgressAsync(
        TextReader stdoutStreamReader,
        IProgress<ScriptOutputLine> stdoutProgress)
    {
        var stdoutLine = await stdoutStreamReader.ReadToEndAsync();

        if (!string.IsNullOrEmpty(stdoutLine))
        {
            stdoutProgress.Report(new ScriptOutputLine(
                line: stdoutLine,
                isErrorLine: false));
        }
    }

    private static async Task CheckStderrAndReportProgressAsync(
        TextReader stderrStreamReader,
        IProgress<ScriptOutputLine> stderrProgress)
    {
        var stderrLine = await stderrStreamReader.ReadToEndAsync();

        if (!string.IsNullOrEmpty(stderrLine))
        {
            stderrProgress.Report(new ScriptOutputLine(
                line: stderrLine,
                isErrorLine: true));
        }
    }
}

public class ScriptOutputLine
{
    public ScriptOutputLine(string line, bool isErrorLine)
    {
        Line = line;
        IsErrorLine = isErrorLine;
    }

    public string Line { get; private set; }

    public bool IsErrorLine { get; private set; }
}

Answer 3

In addition to Wojtpl2's answer. For commands like "tail -f" one of the streamer tasks will lock on ReadLine method:

var stderrLine = await streamReader.ReadLineAsync();

To overcome this, we need to pass token to streamReader with extension method:

        public static Task<T> WithCancellation<T>(this Task<T> task, CancellationToken cancellationToken)
        {
            return task.IsCompleted // fast-path optimization
                ? task
                : task.ContinueWith(
                    completedTask => completedTask.GetAwaiter().GetResult(),
                    cancellationToken,
                    TaskContinuationOptions.ExecuteSynchronously,
                    TaskScheduler.Default);
        }

thx to Can I cancel StreamReader.ReadLineAsync with a CancellationToken?

and use it like this:

var stderrLine = await streamReader.ReadToEndAsync().WithCancellation(cancellationToken);

Answer 4

The following code waits for output and error output independently and has good performance.

using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Renci.SshNet;

namespace DockerTester
{
    public static class SshCommandExtensions
    {
        public static async Task ExecuteAsync(
            this SshCommand sshCommand,
            IProgress<ScriptOutputLine> progress,
            CancellationToken cancellationToken)
        {
            var asyncResult = sshCommand.BeginExecute();
            var stdoutReader = new StreamReader(sshCommand.OutputStream);
            var stderrReader = new StreamReader(sshCommand.ExtendedOutputStream);

            var stderrTask = CheckOutputAndReportProgressAsync(sshCommand, asyncResult, stderrReader, progress, true, cancellationToken);
            var stdoutTask = CheckOutputAndReportProgressAsync(sshCommand, asyncResult, stdoutReader, progress, false, cancellationToken);

            await Task.WhenAll(stderrTask, stdoutTask);

            sshCommand.EndExecute(asyncResult);
        }

        private static async Task CheckOutputAndReportProgressAsync(
            SshCommand sshCommand,
            IAsyncResult asyncResult,
            StreamReader streamReader,
            IProgress<ScriptOutputLine> progress,
            bool isError,
            CancellationToken cancellationToken)
        {
            while (!asyncResult.IsCompleted || !streamReader.EndOfStream)
            {
                if (cancellationToken.IsCancellationRequested)
                {
                    sshCommand.CancelAsync();
                }

                cancellationToken.ThrowIfCancellationRequested();

                var stderrLine = await streamReader.ReadLineAsync();

                if (!string.IsNullOrEmpty(stderrLine))
                {
                    progress.Report(new ScriptOutputLine(
                        line: stderrLine,
                        isErrorLine: isError));
                }

                // wait 10 ms
                await Task.Delay(10, cancellationToken);
            }
        }
    }

    public class ScriptOutputLine
    {
        public ScriptOutputLine(string line, bool isErrorLine)
        {
            Line = line;
            IsErrorLine = isErrorLine;
        }

        public string Line { get; private set; }

        public bool IsErrorLine { get; private set; }
    }
}

You can use it:

var outputs = new Progress<ScriptOutputLine>(ReportProgress);

using (var command =
    sshClient.RunCommand(
        "LONG_RUNNING_COMMAND"))
{
    await command.ExecuteAsync(outputs, CancellationToken.None);
    await Console.Out.WriteLineAsync("Status code: " + command.ExitStatus);
}

And example implementation of the method for reporting progress:

private static void ReportProgress(ScriptOutputLine obj)
{
    var color = Console.ForegroundColor;
    if (obj.IsErrorLine)
    {
        Console.ForegroundColor = ConsoleColor.Red;
        Console.WriteLine(obj.Line);
        Console.ForegroundColor = color;
    }
    Console.WriteLine(obj.Line);
}