Polly WaitAndRetryAsync hangs after one retry

Question

I'm using Polly in very basic scenario to do exponential backoff if an HTTP call fails:

protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
    return await HandleTransientHttpError()
        .Or<TimeoutException>()
        .WaitAndRetryAsync(4, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)))
        .ExecuteAsync(async () => await base.SendAsync(request, cancellationToken).ConfigureAwait(false));
}

private static PolicyBuilder<HttpResponseMessage> HandleTransientHttpError()
{
    return Policy
        .HandleResult<HttpResponseMessage>(response => (int)response.StatusCode >= 500 || response.StatusCode == System.Net.HttpStatusCode.RequestTimeout)
        .Or<HttpRequestException>();
}

I have a test API that just creates an HttpListener and loops in a while(true). Currently, I'm trying to test if the client retries correctly when receiving 500 for every single call.

while (true)
{
    listener.Start();
    Console.WriteLine("Listening...");
    HttpListenerContext context = listener.GetContext();
    HttpListenerRequest request = context.Request;

    HttpListenerResponse response = context.Response;
    response.StatusCode = (int)HttpStatusCode.InternalServerError;

    //Thread.Sleep(1000 * 1);
    string responseString = "<HTML><BODY> Hello world!</BODY></HTML>";
    byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
    response.ContentLength64 = buffer.Length;
    System.IO.Stream output = response.OutputStream;
    output.Write(buffer, 0, buffer.Length);
    output.Close();
    listener.Stop();
}

With the above code all works well and the retries happen after 3, 9, 27 and 81 seconds of waiting, respectively.

However, if I uncomment the Thread.Sleep call, the client retries once and then just hangs until the call times out for the other 3 retries, which is not the correct behavior.

The same thing also happens with the actual production API, which leads me to believe it's not a problem with my test API.

Try to `ConfigureAwait(false)` the awaiting of `HandleTransientHttpError()`. — Theodor Zoulias, Jun 26 '19 at 09:54
@Theodor Zoulias: Unfortunately, that did not solve it. The context in which I'm executing this is similar to a console application, so I don't think it would make a difference theoretically either. — Tudor, Jun 26 '19 at 10:11
Is this on .NET Framework? 4.6.1? It looks like an example of this issue: https://github.com/App-vNext/Polly/issues/642 . That issue is outside (not caused by) Polly - see discussion on .https://github.com/App-vNext/Polly/issues/642 and https://github.com/App-vNext/Polly/issues/658 — mountain traveller, Jun 26 '19 at 11:52
@AstroBoy: Yes, I moved the Polly retry logic outside the actual http call, as Stephen Cleary suggests below. — Tudor, Nov 22 '19 at 12:20

score 5 · Accepted Answer · answered Jun 26 '19 at 19:12

5

Using Polly within HttpClient doesn't work very well. A single SendAsync is intended to be a single call. I.e.:

Any HttpClient timeouts will apply to the single SendAsync call.
Some versions of HttpClient also dispose their content, so it can't be reused in the next SendAsync call.
As noted in the comments, this kind of hang is a known issue and cannot be fixed by Polly.

Bottom line: overriding SendAsync is great for adding pre-request and post-request logic. It's not the right place to retry.

Instead, use a regular HttpClient and have your Polly logic retry outside the GetStringAsync (or whatever) call.

answered Jun 26 '19 at 19:12

Stephen Cleary

437,863
77
675
810

Any thoughts on the workaround by disposing the results inside the RetryAsync callback? – scuba88 Mar 03 '22 at 17:16
1

@scuba88: I haven't tried it. As long as it doesn't close the underlying socket, it might work fine. – Stephen Cleary Mar 03 '22 at 18:23

score 0 · Answer 2 · answered Mar 03 '22 at 17:10

This seems to be an appropriate workaround for the known issue with .NET Framework and using Polly within the HttpClient. We must dispose the result on retry in order to allow multiple requests. See discussion on the original issue here and another discussion describing the workaround here. I have only briefly tested this to determine that it works, but have not fully researched what side-effects might be present.

Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(msg => RetryableStatusCodesPredicate(msg.StatusCode))
.RetryAsync(retryCount, onRetry: (x, i) =>
{
    x.Result.Dispose(); // workaround for https://github.com/aspnet/Extensions/issues/1700
}));

Polly WaitAndRetryAsync hangs after one retry

2 Answers2

Linked