How to asynchronously delay all future calls while handling exception

Question

I have an asynchronous method that uses a limited resource. If the resource becomes unavailable, I want to delay future calls until the resource becomes available again.

Basically, if an exception occurs while accessing the limited resource, a single thread will handle the error by delaying all threads that also caught the exception, and other threads that call the method. After 5 seconds the thread will retry accessing the resource. It's kind of like throttling.

I've implemented this by abusing TaskCompletionSource combined with a SemaphoreSlim. It appears to work. Can this be improved to be less... hacky?

// Use SemaphoreSlim to make sure only one thread handles an error at a time.
private static readonly SemaphoreSlim mySemaphore = new SemaphoreSlim(1);

// Use TaskCompletionSource as a flag to delay threads while an error is handled.
private static volatile TaskCompletionSource<bool> myFlag;

static MyClass()
{
    myFlag = new TaskCompletionSource<bool>();
    myFlag.SetResult(false); // At startup there is no error being handled.
}

public async Task DoSomethingAsync()
{
    while (true)
    {
        await myFlag.Task; // Wait if an error is being handled.

        try
        {
            await ... // Call some asynchronous operations here that can cause errors.
            return;
        }
        catch
        {
            await mySemaphore.WaitAsync(); // Wait so only one thread handles an error.
            bool wasHandled = await myFlag.Task; // Wait and check if error was handled.

            if (wasHandled == false)
            {
                // Reset TaskCompletionSource so error handling on other threads waits.
                myFlag = new TaskCompletionSource<bool>();
                mySemaphore.Release();

                await Task.Delay(5000); // "Handle" the error by waiting 5 seconds.
                myFlag.SetResult(true); // Notify waiting threads an error was handled.

                // Reset TaskCompletionSource 
                myFlag = new TaskCompletionSource<bool>();
                myFlag.SetResult(false);
            }
            else // (wasHandled == true)
            {
                mySemaphore.Release(); // Move along, nothing to see here.
            }
        }
    }
}

To clarify why I think this should be improved: I am using TaskCompletionSource to create an awaitable boolean state, and to reset it I must instantiate a new TaskCompletionSource every time. I don't think this is an intended use for TaskCompletionSource.

I have looked at ManualResetEvent and AutoResetEvent as they seem to do what I need, but they don't offer asynchronous functionality.

Answer 1

Some ideas for you here, with code samples. However, you really should investigate the Circuit Breaker pattern, so you could implement it by yourself or get the working implementation.

As you already now, reset event implementations could work for you, as the AutoResetEvent cover the case for the handling the exception one by one (as it allows only one thread to run through it), and ManualResetEventSlim cover the case for running all the threads in case of normal execution:

var manual = new ManualResetEventSlim(true);
var auto = new AutoResetEvent(true);
while (true)
{
    // check for normal work
    manual.Wait();
    try
    {
    }
    catch
    {
        auto.Wait();
        // only one thread here
        // stop all the worker threads
        manual.Reset();
        // handling here

        // start all the workers
        manual.Set();
        // start exception handlers
        auto.Set();
    }
}

Note that you can simulate the slim version for auto event with SemaforSlim (as you already did). You should use SemaforSlim(1, 1) constructor, for exactly 1 thread to be able to proceed, and the semafor is initially set.

Still, this version is non-async, so you have some choice here.

1. Semafor methods could be awaited, and there is a AsyncManualResetEvent implementation from @StephenCleary.

2. Also, you can use some retry logic with timeout parameters for waiting loops. In this case, you can either yield the execution time with Thread.Yield method or, if you want to set a retry timeout, simply use the Delay task from TPL:

   while (!manual.Wait(SMALL_TIMEOUT_FOR_WAITING)
   {
       // Thread.Yield();
       await Task.Delay(LARGE_RETRY_TIMEOUT);
   }
   

You should also note that it's a common practice to introduce some CancellationToken for your loops and task-oriented methods so your code became more managed in case of stopping the whole system.

Answer 2

If I understand your situation correctly: Consumer --> Your service --> Another service/resource.

You shouldn't implement a back-off strategy in your service, because the end-client is waiting on the other end of the line. Let the client implement the back-off, your service just needs to implement a circuit-breaker and even that's optional.

So...

On the Service side:

You should have a private worker method (e.g. DoSomething) which only take care of 'Doing Something'. It should be wrapped in a public method, which implements Circuit Breaker pattern.

On the 'Consumer' side:

Consumers of your 'service' (the public method) should implement Exponential Backoff Strategy.