4

I have an Azure Function that has both Timer Trigger and Service Bus Trigger functions. In Portal Azure configuration of that AF, I have the correct storage account connection string AzureWebJobsStorage. There were no changes in ARM of both resources, everything was working properly until few days ago. Now only Service Bus Trigger functions are working, but not Timer Trigger ones.

Connecting string to storage is 100% correct, I've even regenerated the access key.

I've found logs for starting Host:

2021-05-25T15:23:10.703 [Error] The listener for function 'XXX' was unable to start.
Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException : The listener for function 'XXX' was unable to start. ---> System.InvalidOperationException : Could not create BlobServiceClient to obtain the BlobContainerClient using Connection: Storage
   at Microsoft.Azure.WebJobs.Script.AzureStorageProvider.GetBlobContainerClient() at D:\a\1\s\src\WebJobs.Script\StorageProvider\AzureStorageProvider.cs : 69
   at Microsoft.Azure.WebJobs.Extensions.Timers.AzureStorageScheduleMonitor.GetStatusBlobReference(String timerName) at D:\a\1\s\src\WebJobs.Script\Timer\AzureStorageScheduleMonitor.cs : 146
   at async Microsoft.Azure.WebJobs.Extensions.Timers.AzureStorageScheduleMonitor.GetStatusAsync(String timerName)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Extensions.Timers.Listeners.TimerListener.StartAsync(CancellationToken cancellationToken) at C:\azure-webjobs-sdk-extensions\src\WebJobs.Extensions\Extensions\Timers\Listener\TimerListener.cs : 99
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener.StartAsync(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonListener.cs : 70
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs : 68
   End of inner exception

I've checked diagnostics in New Support Request -> Solutions and everything is ok. Also, the same error appears (but less descriptive) when trying to start AF from Visual Studio using the same configuration as from Portal Azure.

What could be wrong? What more I could check to find the solution for this problem?

db_k
  • 364
  • 1
  • 5
  • 19
  • Having the exact problem in azure. I actually can run it fine locally in VS, but every night i get that same error in deployed envs – acaputo May 28 '21 at 13:11

2 Answers2

3

There is a possible regression in the latest function host release(3.0.15733.0). Please set the app setting FUNCTION_EXTENSION_VERSION=3.0.15584.0 (previous function runtime version).

The regression will be fixed in the next function host release: https://github.com/Azure/azure-functions-host/releases

Alexey Rodionov
  • 1,436
  • 6
  • 8
  • Do you have "Startup.cs" defined? – Alexey Rodionov May 27 '21 at 22:49
  • Thanks for the answer. I don't have "Startup.cs" defined. And in Portal Azure configuration I had setting FUNCTION_EXTENSION_VERSION=~3 which took always newest version. I changed it for the specific version you've mentioned and its starting properly. There is still some error "Forbidden" but it's most likely connected with access to KV so your answer is perfect. Thanks again. – db_k May 28 '21 at 10:30
  • Do you have AzureWebJobsStorage app setting defined? Does it come from KV? (trying to reproduce the issue) – Alexey Rodionov May 28 '21 at 11:52
  • Yes, I do have `AzureWebJobsStorage` app setting defined. It's the same value as for `AzureWebJobsStorage` which is plain connection string to the storage. – db_k May 30 '21 at 22:54
  • Hi @db_k, can you please share your function app name (we trying to reproduce the issue): https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately – Alexey Rodionov Jun 01 '21 at 17:27
  • sure: 2021-05-30 12:00:00.349, 661c2963-1225-45ef-af50-6229234643e9, West Europe – db_k Jun 02 '21 at 08:43
  • @db_k, do you use the file (app setting named "c...Name") to store AzureWebJobsStorage and then read it on the function app startup? Could you please clarify exactly how you are setting that app setting? – Alexey Rodionov Jun 02 '21 at 20:24
  • No, the file you've mentioned has only business settings. AzureWebJobsStorage is stored as a connection string in AF configuration and from this place is used on startup. For local env I have specific file defined, but all cloud settings are defined only on AF level in Portal Azure. – db_k Jun 07 '21 at 10:20
  • @AlexeyRodionov The work around setting the FUNCTION_EXTENSION_VERSION to 3.0.15584.0 _had_ been working but has now stopped (as of today). We're once again getting the `Could not create BlobServiceClient to obtain the BlobContainerClient using Connection: Storage` error. We've tried setting the runtime to ~3 but that hasn't fixed the issue. Our AzureWebJobsStorage comes from KV. Did something change? – BigPigVT Jun 17 '21 at 20:31
  • I am getting this error too for our Python function app. Tried to reference `3.0.15960.0` as another comment said, but the function app says it is not supported anymore. We do not have our AzureWebJobsStorage from a keyvault, it is directly on the Configurations strings – user3369494 Aug 03 '21 at 15:28
2

Earlier we have updated the FUNCTION_EXTENSION_VERSION from ~3 to 3.0.15584.0, and looks like the issue with 3.0.15733.0 was fixed and I could see latest version is 3.0.15960 (https://github.com/Azure/azure-functions-host/releases)

However, since morning the timer trigger is failing again, so I tried to revert the FUNCTION_EXTENSION_VERSION to ~3, but no luck as function is throwing the same error. But timer trigger is working fine if we set the FUNCTION_EXTENSION_VERSION to 3.0.15960.0.

Naveen Vemunoori
  • 21
  • 4
  • If the AzureWebJobsStorage is present in Configuration section and not reading from Keyvault, then we can put the FUNCTION_EXTENSION_VERSION to ~3. No need to point it to specific version. However, if you see the possibility of storage key is being exposed, then you need to keep updating it with the working version, as the previous versions may be removed over a period of time. – Naveen Vemunoori Jul 13 '21 at 17:21
  • In my case problem was that I use appType `functionApp` for `AzureFunctionApp@1` pipe taks istead of `functionAppLinux` – zolty13 Jul 14 '21 at 08:35
  • My Python function app cannot reference `3.0.15960.0`. I get a `Your custom runtime version (3.0.15960) is not supported. As a result your app is using the latest version 3.0.15828.0, and you will always be running the latest version of the runtime even across major version updates which might contain breaking changes. It is advised to set that value to a specific major version (e.g. ~2) and you will get notified with newer versions for update.` error.... Seems like this version isn't available in the Python function app? This was all working fine until recently – user3369494 Aug 03 '21 at 15:18