Can Pipelines run concurrently in the same Runspace?

Question

How to run two cmdlets parallely in same Runspace. I am using C#.

InitialSessionState iss = InitialSessionState.CreateDefault();
iss.AuthorizationManager = new AuthorizationManager("MyShellId");
iss.ImportPSModule(new string[] { "MSOnline" });
Runspace powerShellRunspace = RunspaceFactory.CreateRunspace(iss);

In two threads, i will run cmdlets using the same runspace.

Pipeline pipeLine = powerShellRunspace.CreatePipeline();
pipeLine.Commands.Add(shellCommand);
pipeLine.Input.Close();
pipeLine.Invoke();
pipeLine.Output.DataReady += new EventHandler(processData);    //processData is a method which processes data emitted by pipeline as and when it comes to avoid out of memory
if (pipeLine.Error != null && pipeLine.Error.Count > 0) {
    Collection<Object> errors = (Collection<Object>)(pipeLine.Error.ReadToEnd());
    //process those errors
}

But when two threads simultaneously using the same runspace to run cmdlets. I am getting exception, "Pipeline not executed because a pipeline is already executing. Pipelines cannot be executed concurrently."

I need to use same runspace for performance reasons. How to achieve my objective?

Runspaces lack the necessary synchronization to support concurrency. Assuming your objective is "good performance", you should expand on why multiple runspaces do not give you good performance. — Jason Shirk, Jun 20 '14 at 22:10
Hi @JasonShirk , mine is web application and since i am importing powershell modules in runspace, it will be slow when many users give http request. Do you have any idea how to do error handling in Runspacepool? and i have to bind that eventhandler to avoid out of memory (updated my code).. — Praveen Kumar, Jun 23 '14 at 17:40
I have run into the same issue. Any body tried the solutions and succeeded? @Praveen Kumar — choudhury smrutiranjan parida, Aug 22 '18 at 09:02
Hi @ch.smrutiranjanparida, i think i used RunspacePool and assigned that pool to the PowerShell obj and used powerShellObj.Streams.Error for error handling and used powerShellObj.BeginInvoke to pass a PSDataCollection to it to process the data emitted by powershell command — Praveen Kumar, Aug 23 '18 at 06:27

StephenP · Answer 1 · 2015-02-11T23:15:59.223

Have you looked at the System.Management.Automation.Runspaces.RunspacePool class? Using it and the InitialSessionState can assist with eliminating overhead from module imports as it is only done once per pool and not per runspace. If you are looking for asynchronous execution of powershell commands here is a very basic, not production-ready example: (note I am not at a computer with Visual Studio but this should be right)

InitialSessionState iss = InitialSessionState.CreateDefault();
iss.AuthorizationManager = new AuthorizationManager("MyShellId");
iss.ImportPSModule(new string[] { "MSOnline" });
#set commands we want to run concurrently
string[] commands = new string[4] {
    "Start-Sleep -Seconds 5; 'Hi from #1'",
    "Start-Sleep -Seconds 7; 'Hi from #2'",
    "Start-Sleep -Seconds 3; 'Hi from #3'",
    "throw 'Danger Will Robinson'"
};
Dictionary<PowerShell, IAsyncResult> dict = new Dictionary<PowerShell, IAsyncResult>();
//this loads the InitialStateSession for all instances
//Note you can set the minimum and maximum number of runspaces as well
using(RunspacePool rsp = RunspaceFactory.CreateRunspacePool(iss))
{
    rsp.SetMinRunspaces(5);
    rsp.SetMaxRunspaces(10);
    rsp.Open();
    foreach(string cmd in commands)
    {
        PowerShell ps = PowerShell.Create();
        ps.AddScript(cmd);
        ps.RunspacePool = rsp;
        //Add parameters if needed with ps.AddParameter or ps.AddArgument
        dict.Add(ps,ps.BeginInvoke());            
    }
    do{
        List<PowerShell> toBeRemoved = new List<PowerShell>();
        foreach(KeyValuePair<PowerShell, IAsyncResult> kvp in dict)
        {
            if(kvp.Value.IsCompleted)
            {
                try
                {
                    PSDataCollection<PSObject> objs = kvp.Key.EndInvoke(kvp.Value);
                    foreach(PSObject obj in objs)
                    {
                        Console.WriteLine(obj);
                    }
                }
                catch (Exception e)
                {
                    Console.WriteLine(e);
                }
                finally
                {
                    toBeRemoved.Add(kvp.Key);
                }
            }
        }
        foreach(PowerShell item in toBeRemoved)
        {
            dict.Remove(item);
        }
        //Wait before we check again
        Thread.Sleep(200);
    } while (dict.Count > 0)
    rsp.Close();
}
//Added to keep console open
Console.Read();

This should give:

Hi from #3
Hi from #1
Hi from #2

Hi @StephenP, but how to do error handling when using RunspacePool, i have updated my code to show how i handle error when using single Runspace. Thing to be noted is, mine is web application, so i cant run all powershell commands at a time asynchronously, powershell commands will run as and when http requests come. — Praveen Kumar, Jun 23 '14 at 17:26
And before invoking powershell command, i should bind `pipeLine.Output.DataReady += new EventHandler(processData);` to process data emitted by pipeline as and when it comes to avoid out of memory. — Praveen Kumar, Jun 23 '14 at 17:33
Errors thrown by the powershell commands will be picked up as runtime errors when you make the EndInvoke call. You will need to add error handling logic. You can also check the [HadErrors](http://msdn.microsoft.com/en-us/library/system.management.automation.powershell.haderrors(v=vs.85).aspx) and [Streams](http://msdn.microsoft.com/en-us/library/system.management.automation.powershell.streams(v=vs.85).aspx) properties. You can still use the runspacepool to eliminate some of the overhead from the module imports, you will just to need implement caching of the runspacepool. — StephenP, Jun 24 '14 at 00:01
@StephenP I don't think you can modify your dictionary while you're iterating over it. The toBeRemoved should be added to a Collection of items to be removed, then each item in that collection should be removed from the dictionary outside of your foreach loop (the same place where your sleep statement is at). Otherwise I'm pretty sure you'll get an exception. — ksun, Feb 09 '15 at 18:58
@ksun good catch. I had the } in the wrong spot. The if(toBeRemoved) should be after the foreach loop. I had a } before the thread.sleep call instead of the if(toBeRemoved). I've updated the answer. — StephenP, Feb 09 '15 at 23:56
@StephenP Thanks, that's better, but I still think there's a small issue. What if multiple powershell scripts in the dictionary have completed when you run through the foreach loop? Then toBeRemoved will be set to the last Powershell object. You should probably have a collection of ToBeRemoved items - Then after the foreach loop iterate over them and remove them all from the dictionary and dispose of them. — ksun, Feb 11 '15 at 19:59
@ksun I have altered it to handle this case. Thanks for pointing this out. — StephenP, Feb 11 '15 at 23:18

Can Pipelines run concurrently in the same Runspace?

1 Answers1

Linked