Intermittent Deployment Issues with Azure App Service

Question

We have been intermittently losing our ability to deploy any updates to our Premium App Service Plan since August 1st. Our primary means of deployment is via Azure DevOps Pipelines (Microsoft Hosted, not self-hosted) and these pipelines have been stable for months or even > 1 year. We have made no changes to our pipelines or App Service Plan (other than requested changes from Microsoft support) in the past months.

When these intermittent issues happen, our ability to deploy updates is impacted for up to a day at a time and it also impacts our ability to publish directly from Visual Studio 2022 (as a backup means of deploying). Additionally, while these deployment issues are occurring, the hosted web applications are accessible and responsive, as is the Kudu (SCM) endpoint.

The error that we see in our Pipeline logs is as follows (we are using the AzureRmWebAppDeployment@4 task):

Error: Error Code: ERROR_COULD_NOT_CONNECT_TO_REMOTESVC More Information: 
Could not connect to the remote computer ("XXXXX.scm.azurewebsites.net") using the specified process ("Web Management Service") because the server did not respond. 
Make sure that the process ("Web Management Service") is started on the remote computer. 
Learn more at: https://go.microsoft.com/fwlink/?LinkId=221672#ERROR_COULD_NOT_CONNECT_TO_REMOTESVC. 
Error: The remote server returned an error: (500) Internal Server Error. 
Error count: 1.

FWIW, our applications are running ASP.NET 6.

Things we have tried to resolve via either searching stack overflow or via Microsoft support include:

1. Running as a ZipDeploy.
2. Running as a ZipDeploy with the app setting WEBSITE_RUN_FROM_PACKAGE = 1.
3. Adding the app setting WEBSITE_WEBDEPLOY_USE_SCM = true.
4. Deploying through Visual Studio’s publish profiles.
5. Deleting Logs in Kudu.
6. Deleting Deployments in Kudu.
7. Freeing up space by deleting old Memory Dumps
   • NOTE: Azure is reporting that we are using 4% of our plan's total storage space.
8. Restarting the App Service / Kudu.
9. Review other stackoverflow articles https://stackoverflow.com/search?tab=newest&q=%5bazure-web-app-service%5d%20ERROR_COULD_NOT_CONNECT_TO_REMOTESVC&searchOn=3

We have been working with Microsoft support on this, but their questions lead me to believe that they are not understanding that we are using an App Service Plan for hosting our applications (vs self-hosting or using a VM).

Answer 1

• One of the reason for intermittent connection problems is reaching a limit when creating new outbound connections. TCP connections and SNAT ports are a few of the restrictions you may reach. Refer this MS documentation on outbound connections includes information on SNAT port limits and how they effect connections. In order to connect with public IP addresses, Azure uses load Balancers that are hidden from customers and source network address translation, or SNAT.

• 128 SNAT ports are originally pre-allocated for each Azure App Service instance. Connections made to the same address and port combination are affected by the SNAT port limit. Your SNAT ports won't run out if your application establishes connections to a variety of address and port combinations.

• You can get around the SNAT port limits using a few different methods. They consist of NAT gateways, connection pools, service endpoints, and private endpoints. One of the more obvious solutions to that issue is connection pools. Utilising regional VNet Integration, service endpoints, or private endpoints will help you avoid SNAT port depletion concerns if your destination is an Azure service that supports service endpoints.

• Your app's outbound traffic to those services won't have outbound SNAT port restrictions if you utilise regional VNet Integration and put service endpoints on the integration subnet. Likewise, you won't see any outgoing SNAT port difficulties to that location if you use regional VNet Integration and private endpoints.

In order to resolve this, You need to visit Diagnose and solve problems

There are other sections where you can check the issue related to your Web app to resolve it like below:-

My Azure Web app DevOps Task:-

trigger:
- master

variables:

  
  azureSubscription: 'xxxxx28-8xx6-xxx'

  
  webAppName: 'valleywebapp09'

  
  environmentName: 'valleywebapp09'

  
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build stage
  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)

    steps:
    - task: ArchiveFiles@2
      displayName: 'Archive files'
      inputs:
        rootFolderOrFile: '$(System.DefaultWorkingDirectory)'
        includeRootFolder: false
        archiveType: zip
        archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
        replaceExistingArchive: true

    - upload: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
      artifact: drop

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build
  condition: succeeded()
  jobs:
  - deployment: Deploy
    displayName: Deploy
    environment: $(environmentName)
    pool:
      vmImage: $(vmImageName)
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureRmWebAppDeployment@4
            displayName: 'Azure App Service Deploy: valleywebapp09'
            inputs:
              azureSubscription: $(azureSubscription)
              appType: webAppLinux
              WebAppName: $(webAppName)
              packageForLinux: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'
              RuntimeStack: 'NODE|10.10'
              StartupCommand: 'npm run start'
              ScriptType: 'Inline Script'
              InlineScript: |
                npm install
                npm run build --if-present