6

Running Jenkins on a Windows Server 2012 slave, jobs are successfully fetching data from Git, served locally (inside firewall) by Bitbucket; its last task is to execute a "git rev-list" on the change it just checked out. This command is taking about 8 minutes to complete. This is repeatable; it is a relatively new environment and I don't have history to see if this is something that just started.

In our production environment where Jenkins itself is running on Windows without a slave (for 2 years), this same rev-list command takes only 3 seconds to complete.

I have seen several references to "git fetch hangs" but for me it is not the fetch that is hanging - given the size of the project it completes in a reasonable amount of time.

Timestamped log is below.

00:00:00.001 Started by user #####
00:00:00.002 Building remotely on JS-W-01 (msvs ishield win) in workspace c:\jenkins\workspace\MyProject\2.7.11_nightly@2
00:00:00.015 [TemplateProject] Using SCM from: MyProject � 2.7.11_Template
00:00:00.044 Cloning the remote Git repository
00:00:00.049 Cloning repository ssh://vdn-bitbucket:7999/myprojectcore.git
00:00:00.052  > git init c:\jenkins\workspace\MyProject\2.7.11_nightly@2 # timeout=10
00:00:00.153 Fetching upstream changes from ssh://vdn-bitbucket:7999/myprojectcore.git
00:00:00.154  > git --version # timeout=10
00:00:00.200 using GIT_SSH to set credentials Jenkins Builder vdnbuild-bb SSH access key
00:00:00.210  > git fetch --tags --progress ssh://vdn-bitbucket:7999/myprojectcore.git +refs/heads/*:refs/remotes/origin/*
00:00:22.857  > git config remote.origin.url ssh://vdn-bitbucket:7999/myprojectcore.git # timeout=10
00:00:22.911  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
00:00:22.967  > git config remote.origin.url ssh://vdn-bitbucket:7999/myprojectcore.git # timeout=10
00:00:23.023 Fetching upstream changes from ssh://vdn-bitbucket:7999/myprojectcore.git
00:00:23.023 using GIT_SSH to set credentials Jenkins Builder vdnbuild-bb SSH access key
00:00:23.028  > git fetch --tags --progress ssh://vdn-bitbucket:7999/myprojectcore.git +refs/heads/*:refs/remotes/origin/*
00:00:23.641  > git rev-parse "refs/remotes/origin/release/2.7.11^{commit}" # timeout=10
00:00:23.697  > git rev-parse "refs/remotes/origin/origin/release/2.7.11^{commit}" # timeout=10
00:00:23.748 Checking out Revision fb773fc64875f5d457bc255088fcec4df3028216 (refs/remotes/origin/release/2.7.11)
00:00:23.752  > git config core.sparsecheckout # timeout=10
00:00:23.799  > git checkout -f fb773fc64875f5d457bc255088fcec4df3028216
00:02:21.725 Commit message: "Merge pull request #30 in MYPROJECT/core from bugfix/MYPROJECT-19 to release/2.7.11"
00:02:21.730  > git rev-list fb773fc64875f5d457bc255088fcec4df3028216 # timeout=10
00:10:50.813 [TemplateProject] Starting builders from: MyProject � 2.7.11_Template
00:10:50.827 [2.7.11_nightly@2] $ cmd /c call C:\Windows\TEMP\jenkins7743297501210902154.bat
00:10:50.853

Execution environment:

  • Jenkins 2.86 running on Centos 7; Git Plugin 3.6.2 (happened with 3.6.0 as well)

  • Slave running slave.jar 3.7 on Windows Server 2012 R2 Standard.

  • Git version is 2.14.3 (happened with 2.10.0 as well)

John Elion
  • 1,323
  • 1
  • 16
  • 30
  • Tried connecting the stalling Jenkins to my "old" bitbucket outside the lab, and my "old" Jenkins to my "new" Bitbucket inside the lab. The problem stays with the stalling Jenkins - that is, it doesn't seem to matter where the slave is connecting, it stalls. Connecting via SSH; Could the stall could be caused by cert validation reaching outside our network - that would likely be blocked. I would expect earlier Git commands to also stall if that were the case. Anyway, suspecting problem lies in Jenkins Git Plugin executing on Jenkins slaves. – John Elion Oct 27 '17 at 11:35
  • (Possibly a local config problem; I do not claim this is a bug in the plugin.) – John Elion Oct 27 '17 at 11:41
  • I installed Jenkins locally on the slave machine, and configured the same jobs. The delay does not happen when the job is running via the Jenkins master. It is only occurring when the job is running in a Jenkins slave. I think this rules out network or Jenkins configuration and indicates a software bug. – John Elion Oct 30 '17 at 12:49
  • I have also confirmed that running the command "git rev-list " on the slave machine in a command window (or even in an at job) completes "instantly." – John Elion Oct 30 '17 at 18:49

2 Answers2

1

Beside making sure everything is updated (Jenkins or Git), you do not seem to have a network/firewall issue, since the checkout part is completed reasonnably fast.

If git rev-list, which, as I mentioned here had numerous fixes since 2017, is still slow, check first if it is because of Jenkins plugins.

That command has been reported slow, for instance, in issue JENKINS-60536, because of Jenkins pipeline JiraSearch and JiraIssueUpdater against Jira Cloud, fixed with jenkinsci/jira-plugin PR 230

Try and disable your Jenkins plugins, or use another Jenkins instance (one launched manually from a fresh installation), just to test if the same git rev-list remains slow when executed by a Jenkins without any plugin.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
0

I am not aware of your project scope, but it appears to me the issue should occur because git rev-list is a verbose command which lists all the commits from the requested commit in reverse order. Depending on your configuration, it might traverse all branches without detached HEAD. Therefore, if the output of the command should be transferred over the network for evaluation or redistribution to other nodes, it will be costly.

I am not aware of your specific use case, however, make sure that your slave is not shipping the output of the command to the master node or other nodes by ignoring the output of the command (Which might have been used in plugins) or configuring Jenkins to delegate all tasks to the same slave for each individual run.

Vahid Haratian
  • 708
  • 2
  • 6
  • 23