2

According to the below diagram on https://learn.microsoft.com/en-us/azure/cosmos-db/change-feed-processor, at least 4 partition key ranges are distributed between two hosts. What I'm struggling to understand in this diagram is the distinction between a host and a consumer. In the context of Azure Functions, would it be true to say that a host is a Function app whereas a consumer is an active/warm instance?

enter image description here

I'd like to create a setup with N many Function apps each with 0-200 active instances (depending on workload). At the same time, I'd like to read Change Feed. If I use a CosmosDBTrigger with the same connection string and lease container in each app, is this taken care of automatically or do I need a manual implementation?

user246392
  • 2,661
  • 11
  • 54
  • 96
  • If you're using [trigger binding](https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-cosmosdb-v2-trigger?tabs=python), do you really have to bother about load balancing? Won't Azure Function Runtime take care of that automatically (spawn required number of function instances based on traffic)? Are you using Consumption plan or one of the premium? – Kashyap Jan 08 '21 at 17:09
  • Consumption. As for load balancing, ideally the runtime should take care of this, but that's what I'm trying to find out. My unique case involves running multiple Function apps (each with the same source code). There is a limit on max number of instances per app which is 200. This is to avoid that by running multiple apps, but it will also help with rolling out prod updates incrementally. – user246392 Jan 08 '21 at 17:24
  • 1
    Now I see your pain spread across all these posts. :-) – Kashyap Jan 08 '21 at 17:25
  • I wish MS understood me as fast as you did. :) – user246392 Jan 08 '21 at 17:26
  • Very curious case, I don't really have an answer for you, but have you already done a PoC to see if this works? This is like multiple consumers (Function Apps) consuming change messages from same Feed, isn't it? Does that even work? Or are you building your own layer between Feed and Functions to split the Feed into multiple feeds (same as "number of Function Apps")? – Kashyap Jan 08 '21 at 17:37
  • Haven't had a chance to do a PoC since I've been busy with other things at work, but at this rate, it looks like I will need to PoC this to get my own answers. Yes to your first question. Multiple apps consuming messages from the same feed. The messages should be distributed across apps (and their instances) automatically by the runtime. This is my expectation. – user246392 Jan 08 '21 at 17:41
  • _"The messages should be distributed across apps (and their instances) automatically by the runtime."_ IMHO that is a big assumption and I would PoC it with some high traffic to be sure. – Kashyap Jan 08 '21 at 17:52
  • Yes. That's on my list to do. At the same time, I'm hoping for an authoritative answer on this matter. – user246392 Jan 08 '21 at 17:59

1 Answers1

1

The documentation you linked is mainly for the Change Feed Processor, but the Azure Functions binding actually runs the Change Feed Processor underneath.

When just using CFP, it's maybe easier to understand because you are mainly in control of the instances and distribution, but I'll try to map it to Functions.

The document mentions a deployment unit concept:

A single change feed processor deployment unit consists of one or more instances with the same processorName and lease container configuration. You can have many deployment units where each one has a different business flow for the changes and each deployment unit consisting of one or more instances.

For example, you might have one deployment unit that triggers an external API anytime there is a change in your container. Another deployment unit might move data, in real time, each time there is a change. When a change happens in your monitored container, all your deployment units will get notified.

The deployment unit in Functions is the Function App. One Function App can span many instances. So each instance/host within that Function App deployment, will act as a available host/consumer.

Further down, the article talks about the dynamic scaling and what it says is basically that, within a Deployment Unit (Function App), the leases will get evenly distributed. So if you have 20 leases and 10 Function App instances, then each instance will own 2 leases and process them independently from the other instances.

One important note on that article is, scaling enables a higher CPU pool, but not a necessarily a higher parallelism.

As the documentation mentions, even on a single instance, CFP will process and read each lease it owns on an independent Task. The problem is, all these parallel processing is sharing the same CPU, so adding more instances will help if you currently see the instance having a CPU thread/bottleneck.

Now, in your example, you want to have N Function Apps, I assume that each one, doing something different. Basically, microservice deployments which would trigger on any change, but do a different task or fire a different business flow.

This other article covers that. Basically you can either, have each Function App use a separate Lease collection (having the monitored collection be the same) or you can share the lease collection but use a different LeaseCollectionPrefix for each Function App deployment. If the number of Function Apps you will be shared the lease collection is high, please check the RU usage on the lease collection as you might need to increase it (there is a note about it on the article).

Matias Quaranta
  • 13,907
  • 1
  • 22
  • 47
  • Thanks for your answer, Matias. Just to clarify, each Function app will be doing the same thing (with the same source code). This is to help me distribute the processing across many regions, unlock a higher active instance count than 200 max in a single consumption app, and push prod changes in phases. If I keep the LeaseCollectionPrefix the same in each app, what happens then? What I want to achieve is, if I have N apps each with M instances, create N * M leases each with a unique partition key range. Is this possible? – user246392 Jan 08 '21 at 22:59
  • If you deploy N Function Apps with exactly the same code, then they all conform the same Deployment Unit conceptually. If each Function App has 10 instances, you have N * 10 instances to distribute the leases. Keep in mind that if you have more instances than leases, then those extra instances will sit idle. Currently there is one lease per physical partition in your monitored collection. You can use this for high availability because if a Function App region goes down, the leases will be distributed to the instances in the other region, but you cannot control the load balancing process. – Matias Quaranta Jan 08 '21 at 23:52
  • So, just to confirm, are you saying that there could be at most 10 leases in this case and (N-1)*10 instances will sit idle? I was hoping to distribute the Change Feed stream across N * 10 instances. – user246392 Jan 09 '21 at 00:15
  • The number of leases vary, they are (currently) a reflection of your physical partitions. Any instance that is running above the number of leases will sit idle. If any of the running instances goes down, then one of the idle ones can automatically take its place. But you will be having idle instances. – Matias Quaranta Jan 11 '21 at 16:47