C# Run multiple non-blocking external programs in parallel

Question

I need to run several instances of an external executable from my app. The average run time for this executable is about 3 minutes. I want to redirect output from these processes, and update a progress bar in my GUI. Of course I don't want to wait for them to return before I can continue using my app.

I think I should create a thread for every instance, and update my progress bar when a thread finishes.

Is this the right approach ?

Also, do you recommend a good resource / documentation to understand how it works ? I've found http://www.dotnetperls.com/threadpool only.

edit : these processes are network-based, ie: the run time may vary a lot depending on the link latency/bandwidth.

Concerning the progress bar, I would like to update it every time a process finishes. Is there a handler for that ? Later i will add more detailed update, based on the processes output to increase the progress done at each execution step.

edit 2 :

Thanks for your inputs. As I may have to run a lot of process (up to 20), and I don't want to saturate bandwidth, i'll run 5 in parallel max. Every time a process finishes, I increment the progress counter (for my progress bar) and I run another one until they're all completed, using :

Process p = new Process();
p.StartInfo.FileName = pathToApp;
p.EnableRaisingEvents = true;
p.Exited += OnCalibrationProcessExited;
p.Start();

private void OnCalibrationProcessExited(object sender, EventArgs e)
{
  runAnotherOne function
}

Is it correct or is there a more elegant way to achieve this ? I don't want my app to be blocked during execution of course. Is it better to use background workers for this ?

Answer 1

You should be using Process and ProcessStartInfo. You'll need to set ProcessStartInfo.UseShellExecute to false, ErrorDialog to false, RedirectStandardOutput to true (and possibly RedirectStandardError too).

You'll also need to provide a delegate to the Process object to handle to output generated by the external process through OutputDataReceived (and possibly ErrorDataReceived as well).

There's also an Exited delegate you can set that will be called whenever the process exits.

Example:

ProcessStartInfo processInfo = new ProcessStartInfo("Write500Lines.exe");
processInfo.ErrorDialog = false;
processInfo.UseShellExecute = false;
processInfo.RedirectStandardOutput = true;
processInfo.RedirectStandardError = true;

Process proc = Process.Start(processInfo);
proc.ErrorDataReceived += (sender, errorLine) => { if (errorLine.Data != null) Trace.WriteLine(errorLine.Data); };
proc.OutputDataReceived += (sender, outputLine) => { if (outputLine.Data != null) Trace.WriteLine(outputLine.Data); };
proc.BeginErrorReadLine();
proc.BeginOutputReadLine();

proc.WaitForExit();

Answer 2

Just waiting for each thread to end before updating the progress bar results in nothing happening ... then a quick jump .. 3 times. You may as well skip the progress bar.

The correct way to do it IMHO would be to calculate the toal work done across all 3 process:

totalwork = time1 + time2 + time3

Now, if you have multiple processors, it will take more like max(time1, time2, time3) but thats ok. It's a representation of work.

Have a shared variable for work-done. Each time a process does some more work, update the progress bar by calculating work-done += my-work-increment. The progress is just work-done/totalwork.

This will give good results regardless of whether the threads run sequentially or in parallel. Since you don't know how things are going to run (you might have a single processor cpu) this is the best approach.

Answer 3

Create a single thread.

In that thread (pseudocode):

Thread begins here
for each externalApp
   Run the application with redirect output 
   Wait for exit
   Update progress bar
end for
Thread ends here

See http://msdn.microsoft.com/en-us/library/ty0d8k56.aspx for wait for exit

Or... do you want to run the external apps in parallel?

Edit: based on the latest updates in the orginal post:

If you don't know the actual progress then don't use a regular progress bar. How about an infinite progress bar or a "working" icon?

An infinite progress bar could be a progress bar that fills up and the starts from beginning until everything is done. A working icon is like the Windows busy cursor (the ever spinning circle).

Answer 4

Just create several instances of the Process class by calling the constructor, set the properties to redirect output stream, and then start them.

Your program won't wait for the called process to exit, as long as you don't call the WaitForExit method. No multi-threading is required.

Answer 5

How about creating an ObservableCollection of TaskProgressInfo,
where TaskProgressInfo is a custom class, to which you write your progress.

Bind a WPF listview to that collection, using a datatemplate (with target type = TaskProgressInfo) to show a progressbar for each item (task).

Create an array of BackgroundWorkers that launch the external app and monitor it.
Each background worker should update its TaskProgressInfo, thus updating the datasource of a progressbar.

Upon completion each BackgroundWorker should remove its TaskProgressInfo from the ObservableCollection, thus removing a progress bar from the UI.

Since BackgroundWorker uses the background (UI) thread for report progress and completion, the changes to the ObservableCollection will be done by its creating thread (thread safe).

Behind the scenes, .NET will use a ThreadPool - some backgroundworkers will share threads.