2

I have a web application, developed with Visual Studio 2019, targeting .Net Framework 4.8 and using an Angular front end. In production, using IIS 10, after running without problems for well over two weeks, the application just stopped responding, requiring recycling of the application pool. We are struggling to figure out why.

In the application error logs, I found a lot of the following entries (this is just one example):

2022-06-03 08:49:05.300 -05:00 [ERR] Failed writing message. Aborting connection. System.InvalidOperationException: Writing is not allowed after writer was completed. at System.IO.Pipelines.Pipe.GetMemory(Int32 sizeHint) at System.IO.Pipelines.Pipe.DefaultPipeWriter.GetSpan(Int32 sizeHint)
at System.IO.Pipelines.PipeWriter.WriteAsync(ReadOnlyMemory`1 source, CancellationToken cancellationToken) at Microsoft.AspNetCore.SignalR.HubConnectionContext.d__49.MoveNext() 2022-06-03 08:49:06.643 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:06.6435092+00:00"". 2022-06-03 08:49:07.955 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:07.9557024+00:00"". 2022-06-03 08:49:09.158 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:09.1585443+00:00"". 2022-06-03 08:49:10.267 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:10.2676601+00:00"". 2022-06-03 08:49:11.283 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:11.2830451+00:00"". 2022-06-03 08:49:12.298 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:12.2984324+00:00"". 2022-06-03 08:49:13.313 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:13.3138198+00:00"". 2022-06-03 08:49:14.329 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:14.3292071+00:00"". 2022-06-03 08:49:15.344 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:15.3445944+00:00"". 2022-06-03 08:49:16.359 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:16.3599817+00:00"". 2022-06-03 08:49:17.375 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:17.3753694+00:00"". 2022-06-03 08:49:18.390 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:18.3907585+00:00"". 2022-06-03 08:49:19.406 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:19.4061440+00:00"". 2022-06-03 08:49:20.421 -05:00 [WRN] Heartbeat took longer than ""00:00:01"" at ""2022-06-03T13:49:20.4215310+00:00"".

I did some research on the Signal R error (Microsoft.AspNetCore.SignalR (1.1.0)), and found it reported by multiple people here: https://github.com/dotnet/aspnetcore/issues/6701 No solution is offered, but one person commented that it may be “a benign error”. I don’t know either the cause of it, or whether it is serious.

Regarding the Heartbeat warning messages, my research suggests that they are caused because the CPU on the server is overworked. See, for example:How to resolve Heartbeat took longer than "00:00:01" failure?.

However, CPU monitoring shows that CPU usage never exceeded 20% at the time of the incident.

No relevant errors in IIS logs or Event Viewer logs.

eglease
  • 2,445
  • 11
  • 18
  • 28
JRS
  • 569
  • 9
  • 26
  • 1
    You are running ASP.NET Core SignalR on .NET Framework and that cannot be any worse. To resolve, either you stick to the legacy ASP.NET 4.x SignalR on .NET Framework, or completely upgrade to ASP.NET Core + .NET Core. I don't think there is any middle ground. – Lex Li Jun 05 '22 at 00:25
  • Until after .net core 2.1, it was fine to run .net core on top of the .net framework. It was an out of the box option in Visual Studio. See: https://stackoverflow.com/questions/39865054/why-use-the-full-net-framework-with-asp-net-core We have been running this for years without seeing these errors. – JRS Jun 05 '22 at 16:13
  • 1
    But time is your enemy, and all ASP.NET Core on .NET Framework versions are end of life, https://dotnet.microsoft.com/en-us/download/dotnet – Lex Li Jun 05 '22 at 16:49
  • `after running without problems for well over two weeks` When the problem occurs, did the server make some changes such as automatic system upgrades or automatic patch updates? – Bruce Zhang Jun 06 '22 at 01:49
  • @BruceZhang I don't know. I will ask our infrastructure person. Thanks for the idea. – JRS Jun 06 '22 at 17:44
  • @BruceZhang I guess the answer is no. The last Windows Update on this server was installed on 4/2/2022 – JRS Jun 06 '22 at 17:52
  • If so, I think it is necessary to use failed request tracing check what happen when the issue occur. https://learn.microsoft.com/en-us/iis/troubleshoot/using-failed-request-tracing/troubleshooting-failed-requests-using-tracing-in-iis However, if it is caused by network, network monitor is a better tool to troubleshoot this issue. – Bruce Zhang Jun 07 '22 at 05:07

0 Answers0