1

Situation
I've installed Jenkins using the solution template provided by Microsoft and created a fairly basic Jenkins pipeline job on Azure. The job currently fetches the code from my personal Git repo on GitHub, performs some simple operations, zips some build files, attempts to deploy the zip file to my WebApp.

I've ensured that exactly the same set of files are included within the zip file so the zip file's size remains virtually identical throughout.

When ready to publish, it calls the azureWebAppPublish function (part of the Azure App Service: https://plugins.jenkins.io/azure-app-service) with all the correct parameters required by Azure. For a few job runs, everything works, and I can SSH into my Web App and see my zip file - great. However, after a while, it begins to slow and eventually break.

During the azureWebAppPublish call, the following is logged to the console as standard.

Starting Azure Web App Deployment
Cloning repository https://<my-web-app-name>.scm.azurewebsites.net:443/<my-web-app-name>.git
 > git init /var/lib/jenkins/workspace/<my-jenkins-pipeline-name>/.azure-deploy # timeout=30
Fetching upstream changes from https://<my-web-app-name>.scm.azurewebsites.net:443/<my-web-app-name>.git
 > git --version # timeout=30
using GIT_ASKPASS to set credentials 
 > git fetch --tags --progress https://<my-web-app-name>.scm.azurewebsites.net:443/<my-web-app-name>.git +refs/heads/*:refs/remotes/origin/* # timeout=30

The Problem
The git fetch --tags operation seen in the last line of the above code snippet on a relatively new WebApp (i.e. no prior deployments) runs fairly quickly. However, after numerous deployments to the same WebApp it begins to take longer and longer each time. On other tests I've done, this has gone way into 30 minutes which is clearly undesirable.

Question 1: Why is it taking so much longer each time for git fetch --tags to execute and can this be prevented? I'm assuming it's due to the Git deployments that Azure performs for non-Java apps, but still, should it really cause a huge increase in time?

Below are some results to support this.

Deployment 1: 2m 25s (Assumed one-off due to cold-start as a result of creating the WebApp)
Deployment 2: 1m 26s
Deployment 3: 1m 37s
Deployment 4: 1m 58s
Deployment 5: 2m 13s
Deployment 6: 2m 49s
Deployment 7: 3m 17s
Deployment 8: 6m 56s
Deployment 9: 8m 3s (FAILURE)
Deployment 10: 9m 3s (FAILURE)

As shown above, there are two failures. Any job runs beyond Deployment 10 would also result in the same errors along with ever-increasing execution time. All failed with the same error message:

hudson.plugins.git.GitException: Command "git fetch --tags --progress https://<my-web-app-name>.scm.azurewebsites.net:443/<my-web-app-name>.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
...
remote: Compressing objects: n% (x/y)
...
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header

Question 2: Why is this bad pack header error happening frequently as the time taken by the git fetch --tags call increases?

I appreciate any and all help/explanations on this as I'm pretty new to Azure/DevOps in general. Many thanks!

What I've tried so far

  • Increased the timeout from 10m for standard git operations to 30m (org.jenkinsci.plugins.gitclient.Git.timeOut)
  • Increased the timeout for clone operations (Advanced clone behaviours) and increased the timeout for checkout operation (Advanced checkout behaviours)
  • Set shallow clone to be true, along with a depth of 1
  • Added the following git config settings to my Jenkins VM: Git pull fails with bad pack header error
  • Checked the number of tags within the Jenkins pipeline workspace after the result of the git fetch (git tag | wc -l), it reported there was only 9
ojbx
  • 41
  • 4

1 Answers1

0

TL;DR - Quick Answer
This for me was fixed by upgrading (scaling-up) my App Service Plan (see: https://azure.microsoft.com/en-gb/pricing/details/app-service/linux/). Originally it was at B1, upgraded to B3 and the problem disappeared.

The Longer Answer
Azure Web Chat was incredibly helpful, a high load average on my WebApp was identified and I was recommended to upgrade both the Jenkins VM and the App Service Plan.

After doing both, the deployment speeds dropped instantly. Now, deployments are consistently working and are only taking 10s longer each time which is great. I decided to test further and found that the Jenkins VM upgrade had little impact in comparison to the App Service Plan upgrade. I have since scaled down the Jenkins VM to use plan B1ms (1 vCPU and 2GB of RAM) and it's working perfectly.

From this experience, due to the tendency for each job run to take longer and longer each time. I'd recommend increasing the timeout from 10m for standard git operations to something larger (Done by setting org.jenkinsci.plugins.gitclient.Git.timeOut) as mentioned above. I can't comment if any of the other items I laid out above had much effect, but give it a go anyway if you face similar issues.

Many thanks to Prem from the Azure support team

ojbx
  • 41
  • 4