4

Summary:

  • A series of memory dumps is gathered for the purpose of discovering the source of memory leakage within a 32-bit process.
  • An attempt to use Debug Diagnostics to analyse a dump fails after two hours, reporting a timeout exception.
  • Supplying an “AnalysisCompletedTimeout” parameter in the DebugDiag.Analysis.exe.config specifying a timeout of 4 hours does succeed in inhibiting timeout exceptions from being reported. Instead after 2 hours, I see a completely empty report generated.

It seems that my attempt to work past the timeout issue is only partially successful. My question is to ask for suggestions in how to get the analysis to succeed in generating an analysis report.

Further detail:

As requested a customer supplies us with a sequence of memory dumps for a 32-bit COM+ server application. The idea is to use the LeakTrack dll to help track down the source of some memory leakage observed in this process.

The Debug Diagnostics Analysis tool is started, and a dump file is added. Having checked the ‘MemoryAnalysis’ checkbox, the analysis is started. After two hours, the Analysis tool displays a message box reporting a problem:

No report file was generated
---------------------------
An error occurred while generating the analysis report

Exception:
    Type:   TimeoutException

    Message:    This request operation sent to net.pipe://localhost/15466de6-db7d-477f-aac4-42980eb2f27f did not receive a reply within the configured timeout (02:00:00).  The time allotted to this operation may have been a portion of a longer timeout.  This may be because the service is still processing the operation or because the service was unable to send a reply message.  Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.

    StackTrace: 

Server stack trace: 
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.SyncDuplexRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)


Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at DebugDiag.DotNet.x86Analysis.IAnalysisService.RunAnalysisRules(List`1 analysisRuleInfos, List`1 dumpFiles, String symbolPath, String imagePath, String reportFileFullPath, TimeSpan timeout, Boolean twoTabs, Boolean includeSourceAndLineInformationInAnalysisReports, Boolean setContextOnCrashDumps, Boolean doHangAnalysisOnCrashDumps, Boolean includeHttpHeadersInClientConns, Boolean groupIdenticalStacks, Boolean includeInstructionPointerInAnalysisReports, List`1& facts)
   at DebugDiag.DotNet.NetAnalyzer.RunX86Analysis(NetProgress progress, List`1 dumpFiles, List`1 analysisRuleInfos, String symbolPath, String imagePath, String reportFileFullPath, Boolean twoTabs, NetResults& results, List`1& facts)
   at DebugDiag.DotNet.NetAnalyzer.RunAnalysisRulesInternal(DumpFileType bitness, NetProgress progress, String symbolPath, String imagePath, String reportFileFullPath, Boolean twoTabs, AnalysisModes analysisMode)
   at DebugDiag.DotNet.NetAnalyzer.RunAnalysisRules(NetProgress progress, String symbolPath, String imagePath, String reportFileDirectoryOrFullPath, Boolean twoTabs, AnalysisModes analysisMode)
   at DebugDiag.Analysis.AnalyzerClient.RunAnalysisAsyncInternal(NetProgress progress, String symbolPath, String imagePath, List`1 dumpFiles, List`1 analysisRules, String reportFileDirectoryOrFullPath, Boolean IncludeSourceAndLineInformationInAnalysisReports, Boolean SetContextOnCrashDumps, Boolean DoHangAnalysisOnCrashDumps, Boolean IncludeHttpHeadersInClientConns, SynchronizationContext synchContext, Boolean ExcludeIdenticalStacks, Boolean IncludeInstructionPointerInAnalysisReports)

After having a rummage through the DebugDiag analysis tool assemblies using JetBrains dotPeek, I made a change to the DebugDiag.Analysis.exe.config to try to alter the timeout setting to 4 hours:

<?xml version="1.0"?>
<configuration>
  <configSections>
    <sectionGroup name="applicationSettings" type="System.Configuration.ApplicationSettingsGroup, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
      <section name="DebugDiag.DotNet.Properties.Settings" type="System.Configuration.ClientSettingsSection, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false" />
    </sectionGroup>
  </configSections>
  <applicationSettings>
    <DebugDiag.DotNet.Properties.Settings>
      <setting name="AnalysisCompletedTimeout" serializeAs="String">
        <value>04:00:00</value>
      </setting>
    </DebugDiag.DotNet.Properties.Settings>
  </applicationSettings>
  <startup> 
      <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5"/>
  </startup>
  <system.net>
    <defaultProxy useDefaultCredentials="true">
    </defaultProxy>
  </system.net>
</configuration>

Upon retrying the analysis, the tool again works for around 2 hours. Upon completion, Internet Explorer is then fired up with its report, which turns out to be completely blank. A check of the DebugDiag reports folder reveals that the .mht report file is 0 bytes in size. No 'timeout exception' message box was displayed this time however.

So the questions I have are: why is there no report generated? Are there additional configuration settings I need to add/amend to allow the report to complete, either in the debug diag analysis host process or indeed in the calling (UI) process?

In an effort to try to see what else might be going wrong I did attach windbg to the process DebugDiag.x86AnalysisHost.exe once analysis was under way. My hope was that I might be able to see evidence of other exception conditions which could offer clues as to what’s happening. However the process appears to exit in a controlled fashion, exhibiting no exception conditions.

Suggestions for where to go next welcome.

Ed Barrett
  • 41
  • 5
  • In general, Debug Diag can only cover certain patterns, so it is not surprising to see it broken in other cases. If you don't know how to manually analyze dumps, open a support case via http://support.microsoft.com and work with Microsoft support guys. – Lex Li May 30 '19 at 14:37
  • Hi Lex, thanks for your comment. I appreciate that Debug Diag, though powerful, is not as flexible as WinDbg. I am reasonably familiar with using the latter. The particular exercise we have been going through with the customer has been trying to use the LeakTrack dll to help identify the source of leakage. Obviously the nice thing with using Debug Diag here is that it has the smarts to be able to work through the LeakTrack data gathered and produce output in easily readable form. I haven't tried doing this manually, but it's probably rather tricky... – Ed Barrett May 30 '19 at 22:51
  • In the meanwhile, further investigation yields an intriguing clue, making me suspect this behaviour results from the size of the dumps concerned. As stated these are full memory dumps of a 32-bit process. However in this instance the COM+ server app has been configured to ‘Enable 3GB support’. What I notice is that those dump files which are less than 2GB in size yield a Debug Diag analysis memory report in minutes. Whereas with those greater than 2GB in size the tool executes for a couple of hours before yielding an empty report. – Ed Barrett May 30 '19 at 22:54
  • Same issue here, but for web application pools and the dump files are under 2 GB. We were able to get around it by unchecking the Microsoft symbol server in the settings, but then the analysis isn't that useful. Did you ever find a solution? – user0474975 Jan 14 '21 at 20:43
  • Unfortunately no, I have no solution. – Ed Barrett Jan 28 '21 at 15:08

2 Answers2

1

I'll also add that I am receiving this issue with the timeout.

Regrettably, attempting to find any information on how the .config should be configured is a bit of problem, since it doesn't seem to exist.

Message reported by DebugDiag I will update this post if I resolve but I hope including the error message will help.

timkly
  • 793
  • 6
  • 14
1

It can be configured in \AnalysisRules\DebugDiag.AnalysisRules.dll.config

Vasile
  • 31
  • 1