5

I have a State-full service fabric application running in a cluster.. I have about 20 State-full applications running in the same cluster. i have used TraceEventSession for correlation purposes. My cluster is having 10 nodes. When i deploy an application i am able to see the TPL session running active in Performance Monitor. When the number of application deployed in the cluster increased i started receiving the error specified

Insufficient system resources exist to complete the requested service. (Exception from HRESULT: 0x800705AA)

System.Runtime.InteropServices.COMException (0x800705AA): Insufficient system resources exist to complete the requested service. (Exception from HRESULT: 0x800705AA) at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at Microsoft.Diagnostics.Tracing.Session.TraceEventSession.EnableProvider(Guid providerGuid, TraceEventLevel providerLevel, UInt64 matchAnyKeywords, TraceEventProviderOptions options) at TestApp.Service.Program.Main() in C:\Agent_work\12\s\TestApp\TestApp.Service\Program.cs:line 61

this is the code i wrote in program.cs to enable the tpl session

session = new TraceEventSession("TestApp_TplSession");  
session.EnableProvider(TplEtwProviderTraceEventParser.ProviderGuid, TraceEventLevel.Verbose,
                        (ulong)TplEtwProviderTraceEventParser.Keywords.TasksFlowActivityIds);

session.EnableProvider(ServiceEventSource.Current.Name);

I am not sure of how many state full service application will be deployed in the same cluster. How will i get rid of this error??

Rijas Kb
  • 212
  • 3
  • 10

1 Answers1

5

Windows has a limit of 64 ETW sessions that can be running concurrently. Consider using a single stateless app running on every node to create a single session.

Matt Trower - MSFT
  • 361
  • 1
  • 6
  • So do you mean that all the other applications deployed in the cluster should use the session created by the stateless session so that much of the memory can be utilized?? – Rijas Kb Oct 12 '16 at 19:44
  • I am monitoring correlation in my application to track the flow of request.. can you tell me if the activityId and relativeActivityId would repeat for different requests?? – Rijas Kb Oct 14 '16 at 04:40
  • For what it's worth, I've noticed this even when running a single listener e.g. PerfView or Visual Studio's Diagnostic Events window. They don't seem to always clean up after themselves so eventually a reboot is required just to start a single new session. From the code above, at the very least, TraceEventSession is IDisposable and should be disposed. As ETW is machine-wide, I think Matt's advice was to run a single additional stateless service with an ETW listener that takes all events from the node and pushes them somewhere. Consider using Microsoft Diagnostics EventFlow for this. – George Helyar Aug 11 '17 at 09:25