Azure Queue trigger - execute one message at the time not working

Question

I have an azure queue trigger associated to a queue, and I want to ensure that the trigger only reads and executes one message at the time. So, when the message is executed (successfuly or not) it processes the next message.

What is happening is that the queue executes one message, yet it begins to execute other message. My host.json:

"queues": {
    "maxPollingInterval": 20000,
    "visibilityTimeout": "00:01:00",
    "batchSize": 1,
    "maxDequeueCount": 5,
    "newBatchThreshold": 1
  }

Following instructions from MS link:

https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue#trigger---configuration

If you want to avoid parallel execution for messages received on one queue, you can set batchSize to 1

So it would be expected to run only one message at the time (I'm using consumption plan).

This is critical because I need to ensure that only one message it processed at the time.

Is there any setting that I could change?

Or is queue trigger not a good option to address this requirement?

score 4 · Answer 1 · answered Mar 23 '18 at 16:14

4

Storage Queue does NOT guarantee ordering - so if needing sequential processing because order of delivery matters you need to consider Azure Service Bus and set in Function setting (host.json)

maxConcurrentCalls = 1

Even if you do the trick by setting maximum number of instances that a function app can scale to as follow, ordering is still not guaranteed with Azure Storage Queue.

WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1

Microsoft documentation is not the perfect one. It's being continuously updated.

answered Mar 23 '18 at 16:14

EagleDev

1,754
2
11
31

1

Processing one message at a time does not necessarily mean processing in order. According to question content, I think former is the requirement and nothing mentioned about ordering. So this answer is appropriate. @OP -if you need ordering then only option is `Service Bus` with `Session`. – Nikhil Vartak Apr 04 '18 at 22:46
@NikhilVartak I believe he asked to let Azure Functions to process each message one time. I worked some projects which required the same requirement. This was to avoid duplicate when function added info from message to database. – EagleDev Apr 05 '18 at 01:10

Karishma Tiwari - MSFT · Answer 2 · 2018-03-28T18:37:41.823

4

If you want to minimize parallel execution for queue-triggered functions in a function app, you can set the batch size to 1. But this setting eliminates concurrency only so long as your function app runs on a single virtual machine (VM).

If you have multiple Virtual Machines and function instances on each VM, there will be one message processed for each function instance running in each virtual machine.

This microsoft document explains concurrency on triggers.

edited Mar 28 '18 at 18:37

answered Mar 27 '18 at 18:59

Karishma Tiwari - MSFT

1,455
9
16

Any idea how this pertains to durable functions because I have one running on a single instance in an App Service Plan with the queue settings as per the documentation and it tried to process another queue message while it was still processing a previous message – Simon Jun 20 '19 at 14:13

score 3 · Answer 3 · answered Mar 03 '21 at 09:22

3

For those coming across this question looking to debug locally and getting issue with multiple queue items making this difficult, you can add the following to your local.settings.json to override the default functionality on your machine only:

{
  "IsEncrypted": false,
  "Values": {
     "AzureFunctionsJobHost__extensions__queues__batchSize": 1
   }
}

Documentation

answered Mar 03 '21 at 09:22

Liam

27,717
28
128
190

Thanks for finding this quirk, nowhere does it say you must override but it appears to ignore host.json on the local debug machine. Like the OP, I set it in host.json and assumed that would be enough. – MarkD May 16 '22 at 20:51

Azure Queue trigger - execute one message at the time not working

3 Answers3