Polly: BulkheadPolicy combined with RetryPolicy

Question

I've got a C# API that communications with a web service that throws exceptions if it's too busy. I want to limit the number of parallel calls to this WCF service.

As the web service quickly throws exceptions when it's too busy, I want to use a Bulkhead in combination with a retry, and I explicitly want to have the Bulkhead to apply first to limit the number of calls waiting in the Retry Policy. The documentation of PolicyWrap says it usually should be the other way around, but in my case I need it like this.

However, it is not working. I'm running a performance test with 40 concurrent calls, and it keeps retrying all 40 calls in parallel.

This is the policy definition:

var retryPolicy = Policy.HandleResult<wcfResponse>(r => !r.IsSuccessStatus())
  .Or<TimeoutException>()
  .Or<CommunicationException>()
  .WaitAndRetryAsync(60, i => TimeSpan.FromSeconds(1),

// setting maxParallel to 1 and a large queue, 
// so I only expect one call to be retrying at the same time. 
// However, it is doing all 40 calls in parallel, 
// the bulkhead seems to be ignored and the retry policy 
// kicks in for all requests in parallel.

var bulkHeadPolicy = Policy.BulkheadAsync(1, 100);
return bulkHeadPolicy.WrapAsync(retryPolicy);

What am I missing here?

Can you add to the question, the code of the calls placed through the policy? — mountain traveller, May 16 '19 at 05:54
Also, it would be useful to provide the sample code giving rise to the comment that calls are being made unexpectedly concurrently. — mountain traveller, May 16 '19 at 17:01
I found out what the problem was in the meantime: I created a new Bulkhead policy every time, for every request... — Boland, May 17 '19 at 00:05

score 3 · Answer 1 · answered May 17 '19 at 00:07

I've found out what the problem was. I created a new Bulkhead policy for every request, which is wrong. I've changed it to only create it once, and that's working.

Found out after reading the documentation again, this specifically:

Scoping bulkhead instances

An instance of BulkheadPolicy maintains internal state of the number of operations executing through the bulkhead: you must re-use the same BulkheadPolicy instance for each execution through a call site, not create a fresh instance on each traversal of the code.

Polly: BulkheadPolicy combined with RetryPolicy

1 Answers1