1

Let's suppose I have an operation that uses a resource protected by a quota (reqs/s). Think for example any rate-limited Cloud service:

public class ExternalService
{
    public async ValueTask UseResource();
}

The code that uses this resource can be triggered both by legitimate users and automatically by the system when doing housekeeping:

public class MyService
{
    readonly ExternalService _externalService;

    public async ValueTask TriggerByUser() => await _externalService.UseResource();

    public async ValueTask TriggerBySystem() => await _externalService.UseResource();
}

Because this external service has a rate limit, the code has a throttle that prevents too many concurrent requests. The rate limit is high enough that just limiting by concurrency prevents any further issues... 99% of the time, which is why I'm asking now :)

public class MyService
{
    readonly ExternalService _externalService;
    readonly SemaphoreSlim _throttle;

    public async ValueTask TriggerByUser()
    {
        await _throttle.WaitAsync();
        try
        {
            await _externalService.UseResource();
        }
        finally
        {
            _throttle.Release();
        }
    }

    public async ValueTask TriggerBySystem()
    {
        await _throttle.WaitAsync();
        try
        {
            await _externalService.UseResource();
        }
        finally
        {
            _throttle.Release();
        }
    }
}

However, TriggerByUser is more important than TriggerBySystem. The later is a housekeeping task that can be delayed, whilst the former must finish ASAP. In the end, there is a real person at the other end of the wire waiting for their operation to finish!

But sometimes the housekeeping tasks (which can be delayed, but once they start they must run to completion) hog the throttles, and the user has to wait for their operations to finish.

So I'd like the throttle used on TriggerBySystem to be smaller than the throttle used on TriggerByUser. More specifically, the throttle used by housekeeping tasks should consume only a subset of the global throttle. This way there would always be space for user operations.

This would be an example of the sequence of events I want to achieve if the "global" throttle has a count of 10 and the "housekeeping" throttle has a count of 2:

  • Global throttle count: 10, Housekeeping throttle count: 2
  • Trigger by user ENTER -> 9/2
  • Trigger by user ENTER -> 8/2
  • Trigger by system ENTER -> 7/1
  • Trigger by system ENTER -> 6/0
  • Trigger by system ENTER -> Has to wait
  • Trigger by user ENTER -> 5/0
  • Trigger by user LEAVE -> 6/0
  • Trigger by user LEAVE -> 7/0
  • Trigger by system LEAVE -> 8/1

I suppose that this can be achieved "daisy-chaining" SemaphoreSlim(s):

public class MyService
{
    readonly ExternalService _externalService;
    readonly SemaphoreSlim _globalThrottle = new(10);
    readonly SemaphoreSlim _housekeepingThrottle = new(2);

    public async ValueTask TriggeredByUser()
    {
        _globalThrottle.WaitAsync();
        try
        {
            await _externalService.UseResource();
        }
        finally
        {
            _globalThrottle.Release();
        }
    }

    public async ValueTask TriggeredBySystem()
    {
        _housekeepingThrottle.WaitAsync();
        try
        {
            _globalThrottle.WaitAsync();
            try
            {
                await _externalService.UseResource();
            }
            finally
            {
                _globalThrottle.Release();
            }
        }
        finally
        {
            _housekeepingThrottle.Release();
        }
    }
}

However, because of the archiquecture of the system, the place where the throttles are defined and the place where the throttles are actually used are miles apart.

Is there an already existing synchronization mechanism that achieves this behavior that I can use to replace SemaphoreSlim altogether - while maintaining the semantics of Wait/Release?

I'm thinking about replacing the SemaphoreSlim (that only models concurrent requests) with something that actually models what a rate is. But that would be a whole different story.

S_Luis
  • 502
  • 5
  • 13
  • `I'm thinking about replacing the SemaphoreSlim (that only models concurrent requests) with something that actually models what a rate is.` Definitely the best approach. Check out Polly. – Stephen Cleary Aug 24 '23 at 16:29
  • 1
    You can find a `RateLimiter` implementation [here](https://stackoverflow.com/questions/65825673/partition-how-to-add-a-wait-after-every-partition/65829971#65829971), that is configured by two parameters: `int maxActionsPerTimeUnit` and `TimeSpan timeUnit`. – Theodor Zoulias Aug 24 '23 at 16:33
  • Throwing stuff in the air - producer\consumer priority queue, where system calls has less priority. – Monsieur Merso Aug 24 '23 at 16:50
  • 1
    Also related: [SemaphoreSlim Await Priority](https://stackoverflow.com/questions/39474370/semaphoreslim-await-priority). – Theodor Zoulias Aug 24 '23 at 17:08

0 Answers0