1

We have a very simple .NET isolated v4 Azure function app running .NET7 that has three functions all using the ServiceBusTrigger.

We've found a small proportion of executions are failing with the following exception:

System.Threading.Tasks.TaskCanceledException at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess

The exception seems to be being thrown from inside the function's own framework rather than our own code. Full stack trace below. A web search suggests that this can occur when there are unawaited async tasks, but I've confirmed there is no such issue in our code.

Each function performs the following simple tasks:

  1. Deserialize the message content
  2. Call and HTTP endpoint
  3. Log a metric to App Insights

What's worse is that these exceptions are not causing the service bus message to be deadlettered as would be the case if our own code threw an exception. So we're losing critical prod data.

Does anyone have any idea why this might be happening?

Here is the full stack trace:

{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":0,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":1,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Script, Version=4.15.0.0, Culture=neutral, PublicKeyToken=null","method":"Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__9.MoveNext","level":2,"line":96,"fileName":"/src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":3,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":4,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":5,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Script, Version=4.15.0.0, Culture=neutral, PublicKeyToken=null","method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext","level":6,"line":82,"fileName":"/src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":7,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":8,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":9,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext","level":10,"line":20,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":11,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":12,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":13,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext","level":14,"line":52,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":15,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":16,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":17,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext","level":18,"line":581,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":19,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":20,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":21,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext","level":22,"line":527,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":23,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":24,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":25,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext","level":26,"line":306,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":27,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext","level":28,"line":352,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":29,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":30,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":31,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext","level":32,"line":108,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}

Update

Same Issue on the GitHub Azure Functions Host Issues Official Forum - 9147

Tom Troughton
  • 3,941
  • 2
  • 37
  • 77
  • Is the message being completed? It would become visible again in the queue if the function did not mark it as completed. – juunas Mar 13 '23 at 13:36
  • I guess they must be getting completed. No build up of active messages, and no deadletters either. This is only affecting around 2% of total messages so difficult to isolate. – Tom Troughton Mar 13 '23 at 13:44
  • @TomTroughton can you please share the code snippet of what you are trying to do. – RitikaNalwaya Mar 17 '23 at 11:53

1 Answers1

1

Please check if the below findings helpful:

System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
/src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs
\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs

I have seen the complete error message (Stack Trace) you have given, it contains 4 to 5 common errors repeatedly.

As per my research, all the error messages coming because of Timeout property on HttpClient and make sure the underlying connection should not be closed.

Each function performs the following simple tasks:

  1. Deserialize the message content
  2. Call and HTTP endpoint
  3. Log a metric to App Insights

Check how long your Http Endpoint Request and Response is taking time or any timeouts in between the Connection.

I found few SO Issues 1, 2 & 3 with the similar exception messages and the resolutions given as to check the underlying Http Client Connection in the Function Trigger execution.

Also, every exception message are being logged in Application Insights even App Insights instance is integrated and logger class is implemented in the Function Code where you can analyze complete telemetry data, info on this MS Doc.

enter image description here

  • 1
    Thank you for the advice. We ended up trying everything - ensuring Polly retries on the HTTP client, ensuring cancellation token passed right through from function to HTTP client, verifying the total time waiting on HTTP response didn't exceed message lock etc. In the end the thing that fixed the issue was updating the `Microsoft.Azure.Functions.Worker` library from 1.10.0 to 1.13.0. This "magically" fixed the problem, I still don't know why. – Tom Troughton Mar 31 '23 at 08:51