How to deal with 100 seconds timeouts while using Poly retry policy

Question

I'm using retry policy in .net core application and am getting timeouts after exceeding 100 seconds period. Might I use Poly in some incorrect way or it's by design and only timeout period increase might help?

Here is the way I use Poly: Startup:

// populate timeouts array from appsettings
var resilencyOptions = services.BuildServiceProvider().GetRequiredService<IOptions<ResiliencyOptions>>().Value;
var attempts = resilencyOptions.TimeOutsInSeconds.Count;
TimeSpan[] timeouts = new TimeSpan[attempts];
int i = 0;

foreach (var timeout in resilencyOptions.TimeOutsInSeconds)
{
    timeouts[i++] = TimeSpan.FromSeconds(timeout);
}

// register
services.AddTransient<LoggingDelegatingHandler>();
services.AddHttpClient<IMyClient, MyClient>()
    .AddHttpMessageHandler<LoggingDelegatingHandler>()
    .AddPolicyHandler(ResiliencyPolicy.GetRetryPolicy(attempts, timeouts))
    .AddPolicyHandler(ResiliencyPolicy.GetCircuitBreakerPolicy());

Library:

/// <summary>
/// Resiliency policy.
/// </summary>
public class ResiliencyPolicy
{
    /// <summary>
    /// Get a retry policy.
    /// </summary>
    /// <param name="numberofAttempts"> Количество попыток.</param>
    /// <param name="timeOfAttempts"> Массив с таймаутами между попытками, если передается неполный или пустой, попытки делаются в секундах 2^.</param>
    /// <returns></returns>
    public static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy(int numberofAttempts = 5, TimeSpan[] timeOfAttempts = null)
    {
        //  In case timeOfAttempts is null or its elements count doesnt correspond to number of attempts provided,
        //  we will wait for:
        //  2 ^ 1 = 2 seconds then
        //  2 ^ 2 = 4 seconds then
        //  2 ^ 3 = 8 seconds then
        //  2 ^ 4 = 16 seconds then
        //  2 ^ 5 = 32 seconds

        return HttpPolicyExtensions
            .HandleTransientHttpError()
            .WaitAndRetryAsync(
                retryCount: numberofAttempts,
                sleepDurationProvider: retryAttempt =>  ((timeOfAttempts == null) || (timeOfAttempts.Length != numberofAttempts)) ?
                                                        TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)) :
                                                        timeOfAttempts[retryAttempt],
                onRetry: (exception, retryCount, context) =>
                {
                    Logging.Global.LogError($"Retry {retryCount} of {context.PolicyKey} at {context.OperationKey}, due to: {exception}.");
                });
    }

    /// <summary>
    /// Get circuit breaker policy.
    /// </summary>
    /// <param name="numberofAttempts">количество попыток</param>
    /// <param name="durationOfBreaksInSeconds">количество секунд (таймаут) между попытками</param>
    /// <returns></returns>
    public static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy(int numberofAttempts = 5, int durationOfBreaksInSeconds = 30)
    {
        return HttpPolicyExtensions
            .HandleTransientHttpError()
            .CircuitBreakerAsync(
                handledEventsAllowedBeforeBreaking: numberofAttempts,
                durationOfBreak: TimeSpan.FromSeconds(durationOfBreaksInSeconds)
            );
    }
}

Calling from custom http client:

public class MyClient : IMyClient
{
    private readonly HttpClient _httpClient;
    private readonly ILogger<MyClient> _logger;

    public MyClient(HttpClient httpClient, ILogger<MyClient> logger)
    {
        _httpClient = httpClient;
        _logger = logger;
    }

    public async Task<bool> Notify(string url, Guid id, string orderId, int state, int category, DateTime date, CancellationToken cancellationToken)
    {
        // prepare request
        var request = new
        {
            Id = id,
            OrderId = orderId,
            State = state,
            Category = category,
            Date = date
        };

        var data = new StringContent(JsonSerializer.Serialize(request), Encoding.UTF8, "application/json");

        // send request
        _logger.LogInformation("sending request to {url}", url);
        var response = await _httpClient.PostAsync(url, data, cancellationToken);

        // process response
        if (response.IsSuccessStatusCode)
            return true;

        var content = await response.Content.ReadAsStringAsync(cancellationToken);

        response.Content?.Dispose();

        throw new HttpRequestException($"{response.ReasonPhrase}. {content.Replace("\"", "").TrimEnd()}", null, response.StatusCode);
    }
}

Controller simulating endpoint availability:

[ApiController]
[Route("[controller]")]
public class RabbitController : ControllerBase
{
    private static int _numAttempts;

    public RabbitController(IBus client)
    {
        _client = client;
    }

    [HttpPost("ProcessTestREST")]
    public IActionResult ProcessTestREST(Object data)
    {
        _numAttempts++;
        if (_numAttempts%4==3)
        {
            return Ok();
        }
        else
        {
            return StatusCode((int)HttpStatusCode.InternalServerError, "Something went wrong");
        }
    }
}    

I am getting this error:

"The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing."

Answer 1

The important thing to note here, and it's definitely not intuitive, is that the HttpClient.Timeout applies to the ENTIRE collection of calls, which includes all retries and waits: https://github.com/App-vNext/Polly/wiki/Polly-and-HttpClientFactory#use-case-applying-timeouts

The default for HttpClient is 100 seconds, so if your retries and waits exceed that then Polly will throw the TimeoutException.

There's a couple ways to address this:

1. Set HttpClient.Timeout to the max length of time you'd expect it to take for all your retries.
2. Put the timeout policy BEFORE the retry policy, which makes it act like a global timeout policy.

In my case I did #1, because I want my timeout policy to apply independently to each request, so I kept my timeout policy AFTER my retry policy. The docs further explain how this works.

Answer 2

Check https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-6.0#dynamically-select-policies

var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(
    TimeSpan.FromSeconds(10));
var longTimeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(
    TimeSpan.FromSeconds(30));

builder.Services.AddHttpClient("PollyDynamic")
    .AddPolicyHandler(httpRequestMessage =>
        httpRequestMessage.Method == HttpMethod.Get ? timeoutPolicy : longTimeoutPolicy);

The timeout policy should be set during the AddHttpClient phase to override the 100 seconds default value as defined in the official documentation.

Your timeout for polly related requests, should cover the biggest value of your retry policy.

Beware to use custom clients in case you want to ignore the retries, so that the timeout is the default one.

Answer 3

You need to make sure that the timeout for the HttpClient is greater than any of the timeouts for your Polly policies. You need to use the AddHttpClient overload, changing the default timeout for the client from 100 seconds.

var notFoundTimeout = TimeSpan.FromMinutes(5);
var transientTimeout = TimeSpan.FromSeconds(5);
var clientTimeout = notFoundTimeout.Add(new TimeSpan(0, 1, 0));

var notFoundRetryPolicy = Policy.Handle<HttpRequestException>() // 404 not found errors
            .OrResult<HttpResponseMessage>(response => response.StatusCode == System.Net.HttpStatusCode.NotFound)
            .WaitAndRetryAsync(3, (int tryIndex) => notFoundTimeout);

services.AddHttpClient(CLIENT_NAME, config => config.Timeout = clientTimeout)
            .AddPolicyHandler(notFoundRetryPolicy)
            .AddTransientHttpErrorPolicy(
                builder => builder.WaitAndRetryAsync(3, (int tryIndex) => transientTimeout));

Answer 4

I might be late to the game but allow me to put my 2 cents.

All the other answers are focusing on the 100 seconds default value of HttpClient's Timeout property and try to solve that issue. The real problem is how the AddPolicyHandler works under the hood.

I have detailed here how the PolicyHttpMessageHandler ruins the party. In case of typed HttpClient the solution is to move the policy inside the typed client to avoid the usage of AddPolicyHandler.

---

You have already separated the policies into a dedicated class ResiliencyPolicy. (BTW you can declare the class as static). I would recommend to expose a combined policy instead of exposing two policies.

public static IAsyncPolicy<HttpResponseMessage> GetCombinedPolicy(int attempts = 5, TimeSpan[] timeouts = null)
  => Policy.WrapAsync<HttpResponseMessage>(GetRetryPolicy(attempts, timeouts), GetCircuitBreakerPolicy())

Answer 5

You may try this during construct your HttpClient:

HttpClient client = new();
client.Timeout = TimeSpan.FromMinutes(5); // or your desire