8

I have a Durable Orchestration that scales up and down Azure Cosmos DB throughput on request. The scale up is triggered via HTTP, and the scale down happens later via a Durable Timer that is supposed to wake up the Azure Function at the end of the current or next hour. Here is the Orchestrator Function:

public static class CosmosDbScalerOrchestrator
{
    [FunctionName(nameof(CosmosDbScalerOrchestrator))]
    public static async Task RunOrchestrator(
        [OrchestrationTrigger] IDurableOrchestrationContext context)
    {
        var cosmosDbScalerRequestString = context.GetInput<string>();

        var didScale = await context.CallActivityAsync<bool>(nameof(ScaleUpActivityTrigger), cosmosDbScalerRequestString);

        if (didScale)
        {
            var minutesUntilLastMinuteOfHour = 59 - context.CurrentUtcDateTime.Minute;
            var minutesUntilScaleDown = minutesUntilLastMinuteOfHour < 15
                ? minutesUntilLastMinuteOfHour + 60
                : minutesUntilLastMinuteOfHour;
            var timeUntilScaleDown = context.CurrentUtcDateTime.Add(TimeSpan.FromMinutes(minutesUntilScaleDown));
            await context.CreateTimer(timeUntilScaleDown, CancellationToken.None);
            await context.CallActivityAsync(nameof(ScaleDownActivityTrigger), cosmosDbScalerRequestString);
        }
    }
}

Here is the ScaleUpActivityTrigger:

public class ScaleUpActivityTrigger
{
    [FunctionName(nameof(ScaleUpActivityTrigger))]
    public static async Task<bool> Run([ActivityTrigger] string cosmosDbScalerRequestString, ILogger log)
    {
        var cosmosDbScalerRequest =
            StorageFramework.Storage.Deserialize<CosmosDbScalerRequest>(cosmosDbScalerRequestString);

        var scaler = new ContainerScaler(cosmosDbScalerRequest.ContainerId);
        var currentThroughputForContainer = await scaler.GetThroughputForContainer();

        // Return if would scale down
        if (currentThroughputForContainer > cosmosDbScalerRequest.RequestedThroughput) return false;

        var newThroughput = cosmosDbScalerRequest.RequestedThroughput < 25000
            ? cosmosDbScalerRequest.RequestedThroughput
            : 25000;
        await scaler.Scale(newThroughput);

        return true;
    }
}

and the ScaleDownActivityTrigger:

public class ScaleDownActivityTrigger
{
    [FunctionName(nameof(ScaleDownActivityTrigger))]
    public static async Task Run([ActivityTrigger] string cosmosDbScalerRequestString, ILogger log)
    {
        var cosmosDbScalerRequest =
            StorageFramework.Storage.Deserialize<CosmosDbScalerRequest>(cosmosDbScalerRequestString);
        var scaler = new ContainerScaler(cosmosDbScalerRequest.ContainerId);
        var minimumRusForContainer = await scaler.GetMinimumRusForContainer();
        await scaler.Scale(minimumRusForContainer);
    }
}

However, what I observe is that the Function is not awakened until something else triggers the Durable Orchestration. Notice the difference in the timestamps for when this was scheduled and when it happened.

Azure Portal screenshot showing different timestamps

Is the fact that it did not wake up until then by design, or a bug? If it is by design, how can I wake it up when I actually want to?

Scotty H
  • 6,432
  • 6
  • 41
  • 94
  • can you post a bit more of your orchestrator code? I dont spot anything in the bit that you posted – silent Feb 10 '20 at 20:49
  • @silent I edited the question to add more code. – Scotty H Feb 10 '20 at 21:04
  • hm looks all ok to me. Did you try to add some logging, especially in your orchestrator to get a bit more visibilty in what it is doing? Also, have you tried running it locally in debugger (with a timer of only 1 or 2 minutes)? – silent Feb 10 '20 at 21:15
  • Are you running on the [app service plan](https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#app-service-plan) by any chance? If so, make sure you have set the [Always On](https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#always-on) setting. – PramodValavala Feb 18 '20 at 04:19
  • @PramodValavala-MSFT This is on the consumption plan. – Scotty H Feb 18 '20 at 16:44
  • Consumption plan stops after 5 minutes of no activity, so once dead, it does not restart correctly on timer event, (most likely a bug, I had similar issue), I just ran a simply 5 minutes timer to do an empty Ping log to keep process alive. – Akash Kava Feb 25 '20 at 10:11
  • 3
    We upgraded to Azure Functions v3 about a week ago and have not encountered the issue since then. – Scotty H Feb 27 '20 at 18:41
  • sorry guys, but I am struggling to understand how Consumption Plan is related to Durable Timer which is supposed to wake up the function no matter what. The plan itself has nothing to do with the ability to wake up – user3647324 Nov 28 '21 at 13:49

0 Answers0