2

I have a function that works fine locally but is not triggered in Azure unless the function is "awake". The function just copies items from one container to another, with the primary key and id columns swapped (as an eventually-consistent secondary index). There are two functions in this app, both of which are Cosmos triggers (different source containers and lease prefixes); all my other functions are in other apps.

The function is in a Linux consumption plan, deployed via the GitHub Action Azure/functions-action@v1, which uses zipdeploy, which should be syncing triggers. There are no other functions listening on that container (local instances listen to the emulator), and this function has a unique lease prefix anyway.

If I view the function app in the portal (even just the Overview page), the function "wakes up" and processes all the Cosmos DB items just fine. If I then close the Overview page, wait ~30 minutes, and change an item in the watched container, the function does not fire. Nothing goes into the Azure Monitor logs and the corresponding item does not change in the destination container. I can wait several (10) minutes and refresh, and there is still nothing in the logs or the destination container. Then when I re-open the Overview page, the function immediately triggers, and the item in the destination container is updated pretty much immediately; the Azure Monitor logs show up 2-3 minutes later (a normal amount of lag). This behavior is 100% repeatable.

I've checked all the troubleshooting suggestions multiple times, but I don't see anything that would help.

How do I get this function to trigger without me observing it in the portal?

Stephen Cleary
  • 437,863
  • 77
  • 675
  • 810
  • Do you have an app setting that disables the function? Looks like that could cause this behavior: https://github.com/Azure/azure-functions-host/issues/4844 – sveinungf Feb 24 '21 at 00:02
  • Nope; the only app setting I have that starts with `AzureWebJobs` is `AzureWebJobsStorage`. The function apps are set up via ARM just before the function app deployment, so the environment is brand new each time. – Stephen Cleary Feb 24 '21 at 01:46
  • I did something hacky to avoid sleep/cold starts on some of my functions.. I keep my function "always on" by adding a timer function which calls back to itself every 15 minutes keeping it alive. Also, I think you can upgrade from the "consumption" plan to the "Premium" plan and there is an option to keep the function on at all times. I know this doesn't answer your question, but if you get desperate there are kludges you can implement. – Andy Feb 24 '21 at 04:14
  • @Andy: Yes; I can combine all my function apps into one. One of them is a 5-minute timer, so that would keep them essentially always on. It just seems like this might be some kind of a bug in the functions host/orchestrator that should be addressed, or a limitation that needs documentation. If no one from the functions team chimes in, I'll bring this to their attention. – Stephen Cleary Feb 24 '21 at 12:56
  • If the Function is in consumption plan, it should be waking up automatically. The only scenario where I saw this not fully happening is if your Cosmos DB account might be configured with Firewall or VPNs. – Matias Quaranta Feb 24 '21 at 18:19
  • @MatiasQuaranta: No firewall or VPNs here. "All networks, including the internet, can access this Azure Cosmos DB account." Feel free to reach out to me directly (my email is [here](https://stephencleary.com/contact/)) if you'd like me to repro. – Stephen Cleary Feb 24 '21 at 20:03
  • 1
    You'd have to contact Azure Support then to know why the Function is not automatically scaling or reacting. My understanding is that there is a component that checks every couple seconds if there is any pending work and if there is, then it wakes up an instance. – Matias Quaranta Feb 24 '21 at 22:04
  • I'm experiencing the exact same problem after implementing a few new Azure Functions. The same things happened the last time I worked with Functions, 2 years ago - it's amazing that it hasn't been solved yet. I tried contacting Azure support, but trying troubleshooting the issue with them was a waste of time. – J.N. Aug 20 '21 at 11:41
  • @J.N.: In my case it was [PEBKAC](https://stackoverflow.com/a/68862656/263693). – Stephen Cleary Aug 20 '21 at 13:00

1 Answers1

1

Turns out I had my connection strings wrong. I had the CosmosDb trigger connection string in ConnectionStrings when it was supposed to be in AppSettings.

The function would actually work locally (and if it was already running in the cloud) because the bindings were looking in both places: AppSettings where it was supposed to be, and falling back to ConnectionStrings. However, when the function "slept", the Azure scale controller was only watching AppSettings, so that's why it would never wake up.

Stephen Cleary
  • 437,863
  • 77
  • 675
  • 810
  • 1
    The same thing was happening to me, but with a slightly different cause. I put my databaseName and collectionName as variables and since I was so used to using linux app services but was using windows on my functions, I was using `Cosmos__database` and `Cosmos__collection` instead of `Cosmos:database` and `Cosmos:collection`. The weirdest part is that it works when it does, but just doesn't wake up on new changes unless I visit the portal. – arvil Sep 07 '21 at 11:45