Polly's CircuitBreakerAsync does not retry if exception occur

Question

I am using Polly library for transient fault handling. For synchronous operations Polly circuit breaker policy works fine but when I created its async version it does not retries the execution. Kindly suggest :

Asynchronous method:

private async static Task HelloWorld()
    {
        if (DateTime.Now < programStartTime.AddSeconds(10))
        {
            Console.WriteLine("Task Failed.");
            throw new TimeoutException();
        }
        await Task.Delay(TimeSpan.FromSeconds(1));
        Console.WriteLine("Task Completed.");
    }

Polly circuit breaker async policy:

private static void AsyncDemo3(Func<Task> action)
    {
        programStartTime = DateTime.Now;

        AsyncPolicy policy = Policy
            .Handle<TimeoutException>()
            .CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));
        try
        {
            var a = policy.ExecuteAndCaptureAsync(action, true).GetAwaiter().GetResult();
        }
        catch (AggregateException ex)
        {
            Console.WriteLine("Exception: " + ex.Message);
        }
        catch (Exception ex)
        {
            Console.WriteLine("Exception: " + ex.Message);
        }
    }

Executing the Polly circuit breaker policy :

AsyncDemo3(HelloWorld);

Kindly help the find and resolve the problem.

Answer 1

I believe you misunderstood what the circuit breaker policy does.

What it does is that if you call it the given number of times and it fails each time, then it will stop calling the given method for a certain amount of time. But it does not retry by itself.

So to do what I think you want to do, you need to combine retry policy with circuit breaker policy. One way to do that would be:

AsyncPolicy retryPolicy = Policy.Handle<TimeoutException>().RetryAsync(3);

AsyncPolicy circuitBreakerPolicy = Policy
    .Handle<TimeoutException>()
    .CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));

try
{
    retryPolicy.ExecuteAsync(() => circuitBreakerPolicy.ExecuteAsync(action, true))
        .GetAwaiter().GetResult();
}
…

The output of this code is:

Task Failed.
Task Failed.
Task Failed.
Exception: The circuit is now open and is not allowing calls.

Answer 2

Recommend creating both policies and combining the same using PolicyWrap as follows.

Policy Creation

 var circuitBreakerPolicy = Policy
        .Handle<TimeoutException>()
        .CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));
 var retryPolicy = Policy.Handle<TimeoutException>().RetryAsync(3);

 // Combined policy: outermost first, innermost last
 var policy = Policy.WrapAsync(retryPolicy, circuitBreakerPolicy);

Policy Usage

 await this.policy.ExecuteAsync(async () => await SomeFooMethodAsync(cancellationToken));

Answer 3

svick's and user3613932's answers are showing only one half of the story. Namely, how to execute an operation several times to open the Circuit Breaker. But what should we do next?

Let me try to tackle this question in this post.

Circuit breaker in general

You should consider the CB as a proxy. It allows each request to go trough if the downstream system is considered health. If CB detects transient failure by examining the subsequent responses (if any) then it will shortcut the new requests by throwing an exception.

And here comes the important part: It will block the new requests only for certain amount of time. If that elapsed then it allows one request to reach the downstream system as a probe:

• If it fails then it shortcuts all new requests for a given time period again
• If it succeeds then it allows all new requests to reach the downstream system

So, in other words the Circuit Breaker is a pattern to avoid flooding the downstream system (which is considered either overloaded or temporarily unavailable) until it becomes health/available again.

Revising the suggested retries

Let's keep your Circuit Breaker definition, so after 3 subsequent TimeoutExceptions it should deny all new requests for 2 seconds.

Both of the above mentioned SO members are suggested this retry policy:

Policy.Handle<TimeoutException>().RetryAsync(3);

3 retries means 4 attempts (1 initial and 3 retries). Because the CB has been setup in a way that it should open after 3 consecutive TimeoutExceptions that's why at the 4th attempt the CB will throw a BrokenCircuitException.

Because there is no policy which should trigger for BrokenCircuitException that's why the ExecuteAsync throws that exception to the caller.

Let's modify the retry policy to trigger for this exception as well and change the retryCount from 3 to 6

Policy.Handle<TimeoutException>().Or<BrokenCircuitException>().RetryAsync(6);

Now the policy triggers for the BrokenCircuitException as well, but the CB shortcuts 4 of our retry attempts. Why? Because the CB is open for 2 seconds and denies all retry attempts. (an attempt can be denied in only several milliseconds).

So, a better solution would be to wait between each reach attempt rather than retry immediately.

Policy.Handle<TimeoutException>()
      .Or<BrokenCircuitException>()
      .WaitAndRetryAsync(6, _ => TimeSpan.FromSeconds(1));

With this approach we are retrying longer than the CB's break duration. In other words, now one of our retry attempt is the probe which might succeed. And that's the overall goal of introducing resiliency logic overcome transient failures.

Pushing further the basic idea

I've already posted a lots of sample code on StackOverflow about Circuit Breaker.
Let me share with you the most relevant ones here:

• If you want to combine Timeout, Circuit Breaker and Retry policies
• If the circuit breaker is used to detect throttling
• Debugging and fine-tuning Circuit Breaker and Retry integration
• How to make the Retry aware of the Circuit Breaker's break duration