NullReferenceException Error while using Polly package for retries

Question

We have .net 6 service where we are using polly package (Polly, Version=7.0.0.0) for retrying transient api calls. We are seeing System.NullReferenceException: Object reference not set to an instance of an object error a lot. Though, the retry seems to be working, I think we are missing something from the code.

we are able to see logs for retries:

NotExtended delaying for 1000 ms, then making retry 1 {...data..}
NotExtended delaying for 1000 ms, then making retry 2 {...data..}
NotExtended delaying for 1000 ms, then making retry 3 {...data..}

Since I am new to how Polly works, not sure what exactly we are missing from the below code. any suggestions would be much helpful. Thanks in advance.

P.S.: I am pasting snippet of code where we use polly package but not the lines of code NOT related to polly implementation to avoid confusion.

Not sure if this error is occurring for few api calls or for all of them.

using Polly;
using Polly.Extensions.Http;
using Polly.Wrap;

services.AddHttpClient<IApiService, ApiService>()
        .AddPolicyHandler(GetPollyRetryPolicy(builder.services));


private static AsyncPolicyWrap<HttpResponseMessage> GetPollyRetryPolicy(IServiceCollection services)
{
    var asyncPolicy = HttpPolicyExtensions
                        .HandleTransientHttpError()
                        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(retryAttempt),
                            onRetryAsync: async (outcome, timespan, retryAttempt, context) => 
                            {
                                var serviceProvider = services.BuildServiceProvider();
                                var data = new                                                      
                                {
                                    url = outcome.Result?.RequestMessage?.RequestUri?.ToString(),
                                    method = outcome.Result?.RequestMessage?.Method?.Method,
                                    resultContent = await outcome.Result?.Content?.ReadAsStringAsync()
                                };
                                serviceProvider.GetService<ILogger>()
                                ?.Warn($"{outcome.Result.StatusCode} delaying for {timespan.TotalMilliseconds} ms, " +
                                   $"then making retry {retryAttempt}", data: data, ex: outcome.Exception);
                            });

    return Policy.TimeoutAsync(60).WrapAsync(asyncPolicy);
}

Error we are receiving is below:

System.NullReferenceException: Object reference not set to an instance of an object.

--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Polly.Wrap.AsyncPolicyWrapEngine.<>c__DisplayClass2_0`1.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Func`2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func`5 onTimeoutAsync, Boolean continueOnCapturedContext)
   at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Func`2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func`5 onTimeoutAsync, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Polly.Wrap.AsyncPolicyWrapEngine.ImplementationAsync[TResult](Func`3 func, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext, IAsyncPolicy outerPolicy, IAsyncPolicy`1 innerPolicy)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

You are sending an HTTP Request. I do not think the connection is completing. If connection completed than you should of gotten a response with either a 200 OK status or an error message. You code just keeps on retrying. I would check if URL is correct. Then verify if you should be using HTTP or HTTPS (secure). It possible the issue is with TLS which is used to encrypt the request and there may be a certificate issue. It may also be an issue with version of Net/Core and the version that your library is using. — jdweng, Aug 30 '22 at 16:37
Can you post a minimal example that reproduces the problem? Building a whole new service provider within each retry is very odd but should work (it's more normal to pass the logger as part of the context). — Stephen Cleary, Aug 30 '22 at 16:50
Are the typed client interface and implementation called ApiService? — Peter Csala, Aug 30 '22 at 19:04
Inside the logger call this can cause NRE: outcome.Result.StatusCode — Peter Csala, Aug 30 '22 at 19:06
@PeterCsala - my bad, interface - IApiService and implementation - ApiService. Updated. — Vicky, Aug 30 '22 at 19:54
*Not sure if this error is occurring for few api calls or for all of them.* < Could you please figure this out? It would help a lot with the investigation. — Peter Csala, Aug 31 '22 at 11:34

Peter Csala · Accepted Answer · 2022-08-31T11:46:49.687

As Stephen Cleary has mentioned it in his comment: Building a whole new service provider within each retry is very odd. It is not just odd, but not necessary at all.

The AddPolicyHandler has an overload which provides access to the IServiceProvider:

services.AddHttpClient<IApiService, ApiService>()
        .AddPolicyHandler((provider, _) => GetPollyRetryPolicy(provider))

With this technique in our hand you can simply ask for an ILogger like this inside the onRetryAsync delegate:

var logger = provider.GetRequiredService<ILogger>();

I would also suggest to prefer the static Wrap method over the instance method:

Policy.WrapAsync(Policy.TimeoutAsync(60), asyncPolicy);

Here I have detailed the differences between the alternatives.

Please also note that the ordering matters:

Timeout outer, Retry inner: Timeout act a global, over-aching constraint
Retry outer, Timeout inner: Timeout act a local constraint, so it resets itself for each retry attempt

NullReferenceException Error while using Polly package for retries

1 Answers1