0

I have an Azure Function that has EventHub trigger, with Consumption plan. In my test I shoot 3000 events to event hub using in a few batches. Since time for those 3000 events was almost 10 times bigger than the time for 300 events, I suspected that this Azure Function didn't scale to multiple VMs/instances.

To verify that hypothesis, I used a Guid static variable, which I initialized once and logged in every run of the function. All 3000 runs logged the same Guid.

That happens even if I specify following configuration in host.json: "eventHub": { "maxBatchSize": 1, "prefetchCount": 10 }

Logic was that this would limit parallel processing within single instance and multiple instances would be started because of that, but again only 1 Guid is logged.

As a note, this is not the only function in App Service. Could that be the issue? What is the condition that needs to be satisfied so that Function is started on multiple VMs?

Edit: I have 32 partitions and 20 throughput units. First issue was that I was using SendBatchAsync, which doesn't partition events. Even SendAsync didn't bring any scale, like it wasn't partitioning. So I created partitioned eventhub senders and did round robin partitioning when sending events in client application.

That increased number of events processed by AzureFunction, but still didn't create more than 1 VM. Furthermore, number of events processed per second was much larger in the beginning (~200 in each moment), and after 2000 events, or near the end, they dropped to ~5. This has nothing to do with load of the system, as same behavior was observed with 9000 events, where slowing down happened after ~5k events.

This Azure function lasts 50-250 ms, depending on the load. It also sends event to another Azure function through Azure Storage Queue trigger. What is interesting is that neither that function which is triggered by Queue trigger scales to more than 1 VM, and it has ~1k messages in queue at the beginning, before slowness of eventhub triggered azure function. Queue settings in host.json are "queues": { "maxPollingInterval": 2000, "visibilityTimeout" : "00:00:10", "batchSize": 32, "maxDequeueCount": 5, "newBatchThreshold": 1 }

Thanks.

Janusz Nowak
  • 2,595
  • 1
  • 17
  • 36
Vukasin
  • 90
  • 1
  • 7

1 Answers1

2

It depends on a few factors:

  • the number of partitions your event hub has and whether the events you are writing are being distributed across your partitions. Azure Functions uses Event Processor Host to process your workload and the maximum scale you can get in this mode is one VM per partition.
  • the per-event workload you're executing. For example if your function does nothing but log, those 3000 events could be processed in less than 5 seconds on a single VM. This would not warrant scaling your application onto multiple instances.

However if you're writing a batch of events across several partitions which takes several minutes in total to process and you don't see your throughput accelerating as your function scales up then that could indicate that something is not working right and would warrant further investigation.

Paul Batum
  • 8,165
  • 5
  • 40
  • 45
  • I have 32 partitions. First issue was that I was using SendBatchAsync, which doesn't partition events. Even SendAsync didn't bring any scale, like it wasn't partitioning. So I created partitioned eventhub senders and did round robin partitioning when sending events in client application. – Vukasin Apr 19 '17 at 20:03
  • I have edited the question with more info. Thanks for the answer. – Vukasin Apr 19 '17 at 20:14
  • 1
    Adding on to Paul's comment, each Function instance is backed by 1 EventProcessorHost (EPH). EventHub allows only 1 EPH to hold a lease on a partition but >1 partition can be assigned an EPH. On start, you have 1 Function instance => 1 EPH (EPH0). EventHub detects that EPH0 is trying to connect to it and assigns all 32 partitions to it. If EPH0 is able to process all events before the scaling logic kicks in, then you will only ever need 1 Function instance. For details, kindly refer to http://stackoverflow.com/questions/42901284/azure-functions-event-hub-trigger-bindings/42911842#42911842 – Ling Toh Apr 19 '17 at 21:30
  • I see, it is actually able to process all events in short time. My application logic is such that each pair of input events generate 1 event, so that is why later function slows down - because there are no events in eventhub. However, it is strange that they are not produced faster. As it is queue triggered function's duty to generate events, I will try to investigate how to increase scale for queue triggered function. I've already have some ideas from other questions. Thanks! – Vukasin Apr 19 '17 at 22:43