Calling Azure Function 150 times by HTTP results in exception

Question

I am running an asp.net mvc Web App on Azure. In one method I make several HTTP calls to an Azure Function Web API. Within this Azure Function I use a DbContext for inserting new records into the database.

// Method in web application making http requests to azure function web api
public async Task CreateRecords() {
    int amountOfCalls = 150;
    var allTasks = new List<Task<HttpResponseMessage>>();

    for (int i = 0; i < amountOfCalls; i++) {
        var task = HttpClientInstance.Instance.PostAsync(<uri>, new StringContent(i.ToString()));
        allTasks.Add(task);
    }

    await Task.WhenAll(allTasks); // Gives an exception
}

When amountOfCalls is set to 25, 50 or 100, everything is fine. However, when I set the amountOfCalls to 150, initially it inserts some records into the database, but eventually it results in an error:

Unable to connect to the remote server.

So first I thought it had to be something with the maximum concurrent logins of the Azure sql database. So I upgraded that one to the pricing tier S9 (1600 DTU's with a maximum of 3200 concurrent logins noted here).

But this didn't fix the problem. So then I started remotely debugging the Azure Function. And it doesn't give an exception in the Azure Function itself. So the next step was to look into Application Insights. And I saw that the constructor of the DbContext gave an exception (without a stacktrace unfortunately):

type: System.InvalidOperationException

method: WebApp.DataAccess.PocDbContext..ctor

requestName: InsertRecordFunction

requestDuration: 55,643.2201

The Azure Function looks like this:

[DependencyInjectionConfig(typeof(DependencyConfig))] // Autofac
public static class InsertRecordFunction
{
    [FunctionName("InsertRecordFunction")]
    public static async Task<HttpResponseMessage> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post", Route = "InsertRecordFunction")]HttpRequestMessage req,
        TraceWriter log,
        [Inject]IRecordDataService recordDataService)
    {
        log.Info("InsertRecordFunction was triggered by Http.");
        try {
            await recordDataService.AddRandomRecordAsync();
        }
        catch (Exception ex) {
            return req.CreateErrorResponse(HttpStatusCode.InternalServerError, ex);
        }
 
        return req.CreateResponse(HttpStatusCode.OK);
    }

And the ctor of PocDbContext looks like this:

public PocDbContext() : base("DefaultConnection") {
    Database.SetInitializer<PocDbContext>(null);
}

I still have a feeling that it has something to do with connection pooling to the Azure sql database. Before this issue, I ran into another one telling me to set the execution strategy to SqlAzureExecutionStrategy (and I did it this way).

Is there some kind of maximum on web api calls from an Azure Web App using a (static) HttpClient? What else can I do to find out what the exact exception is? Thanks.

Update

In Application Insights I found additional information about the exception:

Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.

Not what I would expect after upgrading the pricing tier of the Azure sql database.

A thread here discussing on the maximum concurrent calls which can be handled in azure functions. https://stackoverflow.com/questions/44257660/azure-functions-limiting-parallel-execution You can check about maxconcurrent request as mentioned in this thread. https://github.com/Azure/Azure-Functions/issues/419 — Baskar Rao, Dec 14 '17 at 03:56
@Mikhail, I can use a queue, but I am just playing around and find it weird that I can't manage to get it to work. I did find the inner exception in Application Insights though: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. — Melissa, Dec 14 '17 at 09:25
What's the value of `Max Pool Size` in your connection string? — Mikhail Shilkov, Dec 14 '17 at 09:30
@Mikhail the connectionstring doesn't contain a `Max Pool Size`. This is the value of the `DefaultConnection` in the Application Settings of the Function application: `Data Source=tcp:declapp-prd2.database.windows.net,1433;Initial Catalog=declapp-prd2;User ID=****;Password=*********` — Melissa, Dec 14 '17 at 09:59

score 3 · Accepted Answer · answered Dec 30 '17 at 00:40

From the docs for ADO.NET, which underpins Entity Framework (SQL Server Connection Pooling (ADO.NET)):

By default, connection pooling is enabled in ADO.NET. Unless you explicitly disable it, the pooler optimizes the connections as they are opened and closed in your application.

A little further down in the Adding Connections section:

Connections are added to the pool as needed, up to the maximum pool size specified (100 is the default).

So explicitly setting your connection string's Max Pool Size to >= 150 should do the trick.

Calling Azure Function 150 times by HTTP results in exception

1 Answers1