1

I am trying to add Windows natives notifications for the lftp linux process. To do that I'm trying to launch the command from c# Console App to catch the status of lftp. The problem is that I don't have the same output if I launch it from c# that if I run it from bash.

I have an helper class to do my Command Line calls : 

public static class ShellHelper
{
    public static void Bash(this string cmd)
    {
        var escapedArgs = cmd.Replace("\"", "\\\"");

        var process = new Process()
        {
            StartInfo = new ProcessStartInfo
            {
                FileName = "C:/cygwin64/bin/sh.exe",
                Arguments = $"-c \"{escapedArgs}\"",
                RedirectStandardError = true,
                RedirectStandardOutput = true,
                UseShellExecute = false,
                CreateNoWindow = true,
                StandardOutputEncoding = Encoding.UTF8,
                WindowStyle = ProcessWindowStyle.Hidden,
            }
        };

        process.OutputDataReceived += new DataReceivedEventHandler(ReadOutput);
        process.ErrorDataReceived += new DataReceivedEventHandler(ErrorOutput);

        process.Start();
        process.BeginOutputReadLine();
        process.BeginErrorReadLine();
        process.WaitForExit();
    }

    private static void ErrorOutput(object sender, DataReceivedEventArgs e)
    {
        if (e.Data != null)
        {
            Console.WriteLine("Error: " + e.Data);
        }
    }

    private static void ReadOutput(object sender, DataReceivedEventArgs e)
    {
        if (e.Data != null)
        {
            Console.WriteLine(e.Data);
        }
    }
}

And my call is :

ShellHelper.Bash("unbuffer lftp -p 22 -u login,password sftp://XXXXXXXXX -e \"set mirror:use-pget-n 20; mirror --parallel=3 -c -P5 /home29/qualinost/files/tests ./tests; quit; \"");

This is the expected output that I get when I launch the command from bash

cd `/home29/qualinost/files/tests' [Connecting...]
cd `/home29/qualinost/files/tests' [Connected]      
`Kaamelott - S01E17 - Le Signe.mkv', got 0 of 48941955 (0%)
    ............................................................................................................................................................................................................................................
[A`Kaamelott - S01E17 - Le Signe.mkv', got 32768 of 48941955 (0%)
................................................................................    ................................................................................    .............................................................................
[A`Kaamelott - S01E17 - Le Signe.mkv', got 131072 of 48941955 (0%)  
................................................................................    ................................................................................    .............................................................................
[A`Kaamelott - S01E17 - Le Signe.mkv', got 458752 of 48941955 (0%) 235.7K/s
oo..............................................................................    ................................................................................    .............................................................................
[A`Kaamelott - S01E17 - Le Signe.mkv', got 720896 of 48941955 (1%) 265.0K/s

And this is what I get when I launch the same command from c# :

cd `/home29/qualinost/files/tests' [Connecting...]
cd `/home29/qualinost/files/tests' [Connected]
`...1E17 - Le Signe.mkv', got 0 of 48941955 (0%)
`...1E17 - Le Signe.mkv', got 131072 of 48941955(0%)..........................
`...1E17 - Le Signe.mkv', got 917504 of 48941955 (1%) 598.5K/s.................
`...1E17 - Le Signe.mkv', got 1671168 of 48941955 (3%) 730.4K/s  ..............
`...1E17 - Le Signe.mkv', got 1900544 of 48941955 (3%) 734.7K/s   .............
`...1E17 - Le Signe.mkv', got 2588672 of 48941955 (5%) 650.1K/s   .............
`...1E17 - Le Signe.mkv', got 3473408 of 48941955 (7%) 727.3K/s   .............
`...1E17 - Le Signe.mkv', got 4096000 of 48941955 (8%) 718.8K/s   .............
`...1E17 - Le Signe.mkv', got 4816896 of 48941955 (9%) 747.9K/s   .............
`...1E17 - Le Signe.mkv', got 5668864 of 48941955 (11%) 954.0K/s  .............

As you can see the name of the file is truncate. My guess is it's maybe a buffer size problem or an encoding format problem with some specials chars that c# doesn't like.

I also tried to directly redirect the output in a file from the command like so:

ShellHelper.Bash("unbuffer lftp -p 22 -u login,password sftp://XXXXXX -e \"set mirror:use-pget-n 20; mirror --parallel=3 -c -P5 /home29/qualinost/files/tests ./tests; quit; \" > output.log");

But it's the same, works like a charm from bash but missing parts when I launch from c#..

Any ideas on what's happening here?

EDIT : Someone posted a comment (and deleted it??) and pointed out that the maximum number of characters that "c#" give me by output line is 80. He asked me to resize the Console with

Console.SetWindowSize(400, 800);

at the begining of my program but it doesn't changed anything. But the 80 chars max is probably an interresting remark!

EDIT 2: I cleaned up the output of lftp with a sed command to remove the progress bar and only get the name of the file and it's progression. But still the output is truncate when I launch my script from C#. I did a test with an extremly short file name (yo.mkv) and everything is here! So I think it's definitly a buffer size problem of StandardOutput. Is there a way to define a bigger size of this buffer?

qualinost
  • 103
  • 8
  • I just tried to add : Console.SetWindowSize(400, 800); before the call but nothing changed... You're probably in the good way because the 3 dots '...' before the truncate file name is probably because it can't display the whole string... – qualinost Nov 07 '18 at 16:40
  • I believe your problem is due to the progress bars, those use lots of control characters that are difficult to capture in standard output as they are constantly changing. Have you looked at the logging options of this utility 'unbuffer' to see if there is a way to set a more standard logging that doesn't use progress bars? – raterus Nov 07 '18 at 17:17
  • Sadly I don't have the hand on the lftp output. unbuffer is just a script from the expect package which I added to cygwin. It allows me to unbuffer the "dynamic" output of lftp. – qualinost Nov 07 '18 at 17:39
  • Is a backtick could be a problem in the stream output in C# ? In the full output I can see that I always have those characters before the cut [A` ? – qualinost Nov 07 '18 at 17:49
  • perhaps you could try to implement your own filtering logic in the ReadOutput/ErrorOutput to catch and discard invalid characters. You'll probably have to watch the incoming stream to determine which characters you want to ignore. – raterus Nov 07 '18 at 18:55
  • How can I do that? If a character splits/truncate the string I only get the already splited/truncated string, don't I? – qualinost Nov 07 '18 at 18:58
  • 1
    Lose the async reads here (BeginOutputReadLine) and process the ProcessInfo.StandardOutput stream directly by using the Read() method. See: https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.processstartinfo.redirectstandardoutput?view=netframework-4.7.2 – raterus Nov 07 '18 at 20:28
  • Sorry for the late answer. I tried to do it in a sync way with `while (process.StandardOutput.Peek() >= 0) { Console.Write((char)process.StandardOutput.Read()); }` but the same things happened. When I check the value of StandardOutput in debug mode, the output content is already truncate in the same way. So far, the only way I found to get all the output is to create a batch file, which will launch a shell script file which will redirect the output in a file with the unix '>' command. Really ugly solution cause I have no vision of what's going on... – qualinost Nov 09 '18 at 07:12

1 Answers1

2

I finally found a workaround on this broken MS async output redirection. I used this code https://stackoverflow.com/a/13860551/1502444 to be able to read each chars of the output by using StandardOutput.Read() method in a different thread and changing the Read buffer size. Would like to thanks the author of this solution and raterus to pointed out the async read problem.

qualinost
  • 103
  • 8