6

I am trying to implement an Azure Durable Function workflow.

Every 6 minutes I have an Azure TimerTrigger function calls an Azure Orchestration Function (OrchestrationTrigger), that in turn starts a number of activity functions (ActivityTrigger).

Sometimes, however, the Orchestration function gets called twice in the span of a few seconds! This is a big problem as my activity functions are not idempotent!

Below is how my code gets called.

TimerTriggered Function: 

[FunctionName("StartupFunc")]
public static async Task Run([TimerTrigger("0 */6 * * * *", RunOnStartup = true, UseMonitor = false)]TimerInfo myStartTimer, [OrchestrationClient] DurableOrchestrationClient orchestrationClient, TraceWriter log)
{
    List<OrchestrationModel> ExportModels = await getData();

    string id = await orchestrationClient.StartNewAsync("OrchestratorFunc", ExportModels);
}

Orchestration Function:

[FunctionName("OrchestratorFunc")]
public static async Task<string> TransformOrchestration([OrchestrationTrigger] DurableOrchestrationContext context, TraceWriter log)
{
    var dataList = context.GetInput<List<OrchestrationModel>>();
    var tasks = new List<Task>();

    foreach (var data in dataList)
    {
        tasks.Add(context.CallActivityAsync<string>("TransformToSql", new TransformModel(data));
    }
    await Task.WhenAll(tasks);
}

Activity function:

[FunctionName("TransformToSql")]
[public static async Task<string> RunTransformation([ActivityTrigger] DurableActivityContext context, TraceWriter log)
{
    TransformModel = context.GetInput<TransformModel>();

    //Do some work with TransformModel
}
Jeppe
  • 1,424
  • 2
  • 15
  • 36

2 Answers2

10

This behaviour is perfectly fine - this is how Durable Functions works by design.

You have the following orchestration:

[FunctionName("OrchestratorFunc")]
public static async Task<string> TransformOrchestration([OrchestrationTrigger] DurableOrchestrationContext context, TraceWriter log)
{
    var dataList = context.GetInput<List<OrchestrationModel>>();
    var tasks = new List<Task>();

    foreach (var data in dataList)
    {
        tasks.Add(context.CallActivityAsync<string>("TransformToSql", new TransformModel(data));
    }
    await Task.WhenAll(tasks);
}

When an activity is called, the flow returns to a concept called Dispatcher - it is an internal being of Durable Functions, responsible for maintaining the flow of your orchestration. As it awaits until a task is finished, an orchestration is deallocated temporarily. Once a task is completed, the whole orchestration is replayed until the next await happens.

The important thing there is, though an orchestration is replayed, an activity is not called once more - its result is fetched from the storage and used. The make things more explicit, please consider the following example:

[FunctionName("OrchestratorFunc")]
public static async Task<string> TransformOrchestration([OrchestrationTrigger] DurableOrchestrationContext context, TraceWriter log)
{
    var dataList = context.GetInput<List<OrchestrationModel>>();
    var tasks = new List<Task>();

    foreach (var data in dataList)
    {
        await context.CallActivityAsync<string>("TransformToSql1", new TransformModel(data);
        await context.CallActivityAsync<string>("TransformToSql2", new TransformModel(data);
    }
}

When TransformToSql1 is awaited, the orchestration is deallocated and the whole flow waits until this activity is finished. Then the orchestration is replayed - it once more awaits TransformToSql1 but since the result of it is saved, it just goes back to the orchestration and awaits TransformToSql2 - then the process is repeated.

kamil-mrzyglod
  • 4,948
  • 1
  • 20
  • 29
  • 2
    I am aware of the replay functionality of the activities, and that works just fine with no problems whatsoever. My issue is two concurrent executions of the Orchestrator function. – Jeppe Sep 13 '18 at 03:20
  • @kamil-mrzyglod - so if my orchestration function needs to perform some one-off code, to take place before the individual activity requests are carried out and not re-run, what would you recommend to do? Perform it in a separate activity of its own, called beforehand? – Stephen Holt Jun 06 '19 at 13:39
  • 1
    @StephenHolt - if your orchestration needs some kind of "warmup" before running, it should get the data from an external source(e.g. it should be passed as an orchestration input). – kamil-mrzyglod Jun 06 '19 at 20:44
6

The orchestration function will run more frequently as it is replayed by the Durable Function framework. Do you see that your activity functions are also triggered again? If so then this is indeed a strange behavior.

Durable Functions uses storage queues and tables to control the flow and capture state of the orchestration.

Each time the orchestration function is triggered (by receiving a message from the control queue) it will replay the orchestration. You can check this in code by using the IsReplaying property on the DurableOrchestrationContext. 

if (!context.IsReplaying)
{
   // this code will only run during the first execution of the workflow
}

Don't use the IsReplaying to determine to run the activities, but use it for logging/debugging purposes.

When an activity function is reached a message is put on the work-item queue. The Durable Functions framework will then look into the history table to determine if this activity function has already been run before (with the given parameters). If it is, then it won't run the activity function again and it will continue with the rest of the orchestration function.

The checkpointing and replay functionality of Durable Functions only works when the orchestration code is deterministic. So never use decisions based on DateTime or random numbers.

More info: https://learn.microsoft.com/en-us/azure/azure-functions/durable-functions-checkpointing-and-replay

Marc
  • 1,798
  • 8
  • 15
  • 1
    I see the second orchestration function (try to) do exactly the same execution of the activities as the first orchestration function. This leads to a lot of exceptions and aborted work. – Jeppe Sep 13 '18 at 03:26