Managing Azure Search 503

Question

Is there a well-known pattern to manage the ServiceUnavailable ? I'm NOT so happy with something like this

catch (CloudException e) when (System.Net.HttpStatusCode.ServiceUnavailable.Equals(e.Response?.StatusCode))
{
    var howMuchWait = TimeSpan.FromMinutes(1);
    if (e.Response.Headers.TryGetValue("Retry-After", out var hValue))
    {
        if(RetryConditionHeaderValue.TryParse(hValue.FirstOrDefault(), out var time) && time.Delta.HasValue)
        {
            howMuchWait = time.Delta.Value;
        }
    }
    logger.LogWarning(() => $"Service Unavailable... Let him rest a bit, I will wait for {howMuchWait}.");
    await Task.Delay(howMuchWait);
    return indexEntities.Select(x => (string)x[indexKeyFieldName]).ToList();
}

This code simply delay the current call to it but does not prevent calls from other threads. Now I'm implementing something different using a Stopwatch and I would like to know if there is a well-known pattern. NOTE: everything using the SDK.

score 2 · Answer 1 · answered Jul 05 '17 at 18:29

When managing for 503's, it is best to implement more of an incremental back-off mechanism as opposed to a fixed time delay. For example, start with 1 second, then 2, then 4, then 8, etc... The key reason for this, is if you are sending a lot of work to Azure Search, it is best to let it catch up as opposed to continually trying to send more work to it.

By the way, for others reading this, sometimes you might think that is a good idea to add partitions while you are seeing these 503's (typically for mass uploads) to give more resources, when in fact, this can cause more 503's as this is more work the service needs to do to configure the partitions. If you do think you need more partitions it would be best to do this before you do the work and then scale down afterwards, if needed.

Also, another side note is that if you leveraged the Azure Search .NET SDK, retry is already integrated. Bruce, has some good information on this here: Azure Search RetryPolicy

Hi Liam. I know what happens trying sending more work to the service and for that reason, I would avoid even the RetryPolicy (read it as: "do not insists (retry) if you know I'm busy"). I'm working on something like this: ```private async Task> InsertBatch(ISearchIndexClient index, List indexEntities) { if(IsInDoNotDisturb()) { return indexEntities.Select(x => (string)x[indexKeyFieldName]).ToList(); } ``` — Fabio Maulo, Jul 05 '17 at 18:45
Just one more thing.. Liam/Bruce would be really useful to have the `Retry-After` header with the 503 to give us the opportunity to respect it. — Fabio Maulo, Jul 05 '17 at 18:56

Managing Azure Search 503

1 Answers1