Retries are not happening after the first retry on a broken circuit

Question

I have the following policies:

var sharedBulkhead = Policy.BulkheadAsync(
            maxParallelization: maxParallelizations, 
            maxQueuingActions: maxQueuingActions,
            onBulkheadRejectedAsync: (context) =>
            {
                Log.Info($"Bulk head rejected => Policy Wrap: {context.PolicyWrapKey}, Policy: {context.PolicyKey}, Endpoint: {context.OperationKey}");
                return TaskHelper.EmptyTask;
            }
        );

var retryPolicy = Policy.Handle<Exception>(e => (e is HttpRequestException)).WaitAndRetryAsync(
            retryCount: maxRetryCount,
            sleepDurationProvider: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)),
            onRetryAsync: (exception, calculatedWaitDuration, retryCount, context) =>
            {
                Log.Error($"Retry => Count: {retryCount}, Wait duration: {calculatedWaitDuration}, Policy Wrap: {context.PolicyWrapKey}, Policy: {context.PolicyKey}, Endpoint: {context.OperationKey}, Exception: {exception}.");
                return TaskHelper.EmptyTask;
            });

            var circuitBreaker = Policy.Handle<Exception>(e => (e is HttpRequestException)).CircuitBreakerAsync(
            exceptionsAllowedBeforeBreaking: maxExceptionsBeforeBreaking, 
            durationOfBreak: TimeSpan.FromSeconds(circuitBreakDurationSeconds), 
            onBreak: (exception, timespan, context) =>
            {
                Log.Error($"Circuit broken => Policy Wrap: {context.PolicyWrapKey}, Policy: {context.PolicyKey}, Endpoint: {context.OperationKey}, Exception: {exception}");
            },
            onReset: (context) =>
            {
                Log.Info($"Circuit reset => Policy Wrap: {context.PolicyWrapKey}, Policy: {context.PolicyKey}, Endpoint: {context.OperationKey}");
            }
        );

var fallbackForCircuitBreaker = Policy<bool>
         .Handle<BrokenCircuitException>()
         .FallbackAsync(
             fallbackValue: false,
             onFallbackAsync: (b, context) =>
             {
                 Log.Error($"Operation attempted on broken circuit => Policy Wrap: {context.PolicyWrapKey}, Policy: {context.PolicyKey}, Endpoint: {context.OperationKey}");
                 return TaskHelper.EmptyTask;
             }
         );

var fallbackForAnyException = Policy<bool>
            .Handle<Exception>()
            .FallbackAsync(
                fallbackAction: (ct, context) => { return Task.FromResult(false); },
                onFallbackAsync: (e, context) =>
                {
                    Log.Error($"An unexpected error occured => Policy Wrap: {context.PolicyWrapKey}, Policy: {context.PolicyKey}, Endpoint: {context.OperationKey}");
                    return TaskHelper.EmptyTask;
                }
            );


var resilienceStrategy = Policy.WrapAsync(retryPolicy, circuitBreaker, sharedBulkhead);
        var policyWrap = fallbackForAnyException.WrapAsync(fallbackForCircuitBreaker.WrapAsync(resilienceStrategy));

I execute the policy like so:

Task.Run(() =>
        {
            foreach (var changeMessage in changeMessages)
            {
                policyWrap.ExecuteAsync((context) => CallApi(changeMessage), new Context(endPoint));
            }
        });

No retries happen after the first call to fallbackForCircuitBreaker. I want the retries to happen as per the wait duration regardless of the state the circuit is in.

Why doesn't this work?

The posted code is executing the async tasks without awaiting them: `await Task Run(...)` and `await policyWrap.ExecuteAsync(...)` would be more usual - unless you intentionally want to fire-and-forget the async tasks. If fire-and-forget is intended, it may be worth considering what the code should do for exceptions which would otherwise become [unobserved task exceptions](https://blogs.msdn.microsoft.com/pfxteam/2011/09/28/task-exception-handling-in-net-4-5/). — mountain traveller, Dec 26 '18 at 23:13
Yes. That is intentional. Won't the unhandled exceptions be caught by `fallbackForAnyException`? — Nimish David Mathew, Dec 27 '18 at 05:13
The unhandled exceptions will not be caught be `fallbackForAnyException`, because the outcomes of the `Task`s representing the async work are not being observed in any way, in the posted code. I would suggest reading [any general articles on exception handling in async code](https://www.google.com/search?q=exception+handling+in+async+c%23), and the [previously linked article](https://blogs.msdn.microsoft.com/pfxteam/2011/09/28/task-exception-handling-in-net-4-5/) for a deep discussion of unobserved task exceptions. — mountain traveller, Dec 27 '18 at 09:17

score 2 · Accepted Answer · answered Dec 26 '18 at 23:07

2

No retries happen after the first call to fallbackForCircuitBreaker. I want the retries to happen as per the wait duration regardless of the state the circuit is in.

The retry policy is not configured to handle BrokenCircuitException, only HttpRequestException.

To make the retry policy retry also for BrokenCircuitException, configure the policy:

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .Or<BrokenCircuitException>()
    .WaitAndRetryAsync( /* etc */ );

Note: If only the above change is made, the sequencing of policies in the PolicyWrap posted in the question:

.., fallbackForCircuitBreaker, retryPolicy, circuitBreaker, sharedBulkhead

will mean fallbackForCircuitBreaker will only be invoked if all retries fail, and if the last retry fails with BrokenCircuitException. See PolicyWrap documentation.

answered Dec 26 '18 at 23:07

mountain traveller

7,591
33
38

What changes should be made in order for `fallbackForCircuitBreaker` to be invoked every time a retry is made on a broken circuit? – Nimish David Mathew Dec 27 '18 at 04:47
One more question, I have declared `sharedBulkhead` as an instance property in a service. It is initialized in the constructor. Is that a good practice? – Nimish David Mathew Dec 27 '18 at 04:51
Please raise these questions as separate Stack Overflow questions; Stack Overflow prefers us to avoid answering questions in comments, and answers may be too long for comments. – mountain traveller Dec 27 '18 at 09:31
Will do that. Meanwhile, can you take a look at this: https://stackoverflow.com/questions/53942022/how-to-wrap-transienthttperror-retry-policy-with-circuit-breaker-and-bulkhead-po – Nimish David Mathew Dec 27 '18 at 09:34
Have added the question: https://stackoverflow.com/questions/53944139/fallback-for-circuit-breaker-not-invoked-on-all-retries-on-the-broken-circuit – Nimish David Mathew Dec 27 '18 at 11:14

Retries are not happening after the first retry on a broken circuit

1 Answers1