2

We are using Azure services for this, and we have a web api (Flask) running on Azue Web Service App, connected to a PostgreSQL in a different service (so far so good, although the connection is a bit slow). Anyway, there is an action performed by users that triggers a process. Basically, we need to pre-process thousands of records (between 1000 and 5000), and processing each one takes between 5 and 10 seconds.

We thought in using Azure Functions to this, and trigger thousands of http requests. But the requests are not parallelized, even when we added more instances. It seems they are queued. I talked to Azure support and they told me this may not the best solution, since Azure functions are expected to run quick, not take several seconds to perform.

So my question is: what's the best architecture for doing something like this on Azure? I read about something called Event Grid, but I have no clue (first time using Azure). The basic solution would be to have a VM, that can receive parameters and start processes, but that's something that would be too complicated (and we don't have a devops in our team that could take care of that).

Thanks in advance.

acasla
  • 131
  • 1
  • 9
  • How do you trigger the function? – CSharpRocks Sep 19 '20 at 14:28
  • Right now, we are sending async http requests to Azure function. They are sent all at the same time, we are measuring that, and I am seeing in the Azure function dashboard that all the request are received too. But they are not executed instantly (we add logs to the beggining of the function and at the end). We also try with a function that is just 1) Logging, 2) Sleep, 3) Loggin again, and it's the same behaviour, not parallelization. – acasla Sep 19 '20 at 14:36
  • Have you tried sending a message to a queue and have the function being triggered by new messages? Also, this may be provide some clues: https://stackoverflow.com/questions/46191002/requests-are-queuing-in-azure-appservice-though-it-has-enough-threads-in-threadp – CSharpRocks Sep 19 '20 at 14:49
  • No, I did not try using a queue. As I said, I am new to Azure so I dont now all the services available :( Do you think that would make them run in parallel? – acasla Sep 19 '20 at 14:57
  • Are you using Consumption hosting option in Azure Function or App Service Plan hosting. Because if it is consumption then Functions should scale-out very quickly. See this: https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference#parallel-execution – Hassan Raza Sep 20 '20 at 13:25
  • I tried with Azure Consumption, then with the Service plan hosting with different plans, but it didn't scale out with any of those. – acasla Sep 20 '20 at 13:44

3 Answers3

2

I would strongly recommend evaluating azure durable functions for this, the runtime supports fan out automatically meaning you can get massive scale very easily. Make sure the systems these functions talk to can also handle the load!

The basic structure is that you kick off an orchestrator function which can then kick off multiple activity functions, which if necessary can themselves be chained together or even fanned out again. The timings you’ve quoted would not be an issue see here. You can run all this on the consumption plan happily.

Of course you need to look into cost here, if you do use the consumption plan you will pay by the Mb-second, ie cost is proportional to memory used and time taken.

Matt
  • 12,569
  • 4
  • 44
  • 42
1

You can try the Azure Queue storage trigger it will trigger the azure function as you add a message in the azure queue and yeah it will run parallel But it has a limit but you can scale it as it's mention in the documentation.

Azure Queue storage trigger for Azure Functions

Jay Dadhaniya
  • 171
  • 4
  • 15
0

Matt's answer is will work, however I specifically caught that you would prefer the requests to be parallelized (i.e. time to execute is a priority). Have you looked at Azure Batch ? It is specifically designed for large scale parallel workloads.