Why is Jenkins failing when fetching from git, while the command line isn't?

Question

All of my Jenkins builds are failing at the git fetch line.

It's failing at git fetch --tags --progress git@bitbucket.org:ethenwilson/whentoact.git

Started by user anonymous
Building in workspace /Users/ethen/.jenkins/workspace/Build NikNik
 > git rev-parse --is-inside-work-tree
Fetching changes from the remote Git repository
 > git config remote.origin.url git@bitbucket.org:ethenwilson/whentoact.git
Fetching upstream changes from git@bitbucket.org:ethenwilson/whentoact.git
 > git --version
using GIT_SSH to set credentials NikNik BitBucket SSH Key
 > git fetch --tags --progress git@bitbucket.org:ethenwilson/whentoact.git +refs/heads/*:refs/remotes/origin/*
FATAL: Failed to fetch from git@bitbucket.org:ethenwilson/whentoact.git
hudson.plugins.git.GitException: Failed to fetch from git@bitbucket.org:ethenwilson/whentoact.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:622)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:854)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:879)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
    at hudson.model.Run.execute(Run.java:1732)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:234)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git@bitbucket.org:ethenwilson/whentoact.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Counting objects: 2682, done.[K
remote: Compressing objects:   0% (1/1399)   [K
remote: Compressing objects:   1% (14/1399)   [K
...
remote: Compressing objects:  99% (1398/1399)   [K
remote: Compressing objects: 100% (1399/1399)   [K
remote: Compressing objects: 100% (1399/1399), done.[K
Receiving objects:   0% (1/2682)   
Receiving objects:   1% (27/2682)   
...  
Receiving objects:  78% (2092/2682), 4.07 MiB | 1.59 MiB/s   
Corrupted MAC on input.
Disconnecting: Packet corrupt
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1325)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1186)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:257)
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:620)
    ... 10 more

When I run git fetch --tags --progress git@bitbucket.org:ethenwilson/whentoact.git from the command line, it works fine, which means my SSH keys must be working.

I'm connecting to BitBucket with Jenkins with SSH verification. Jenkins gets the key from the file it's located (the default one), so I know that Jenkins is using the same key as I am when I run from the command line.

I'm using the latest build of the BitBucket and Git plugins for Jenkins. My installed Git on my Mac is version 1.8.5.2 (Apple Git-48).

My jenkins start command is nohup java -jar ~/jenkins.war --httpPort=8081 --ajp13Port=8010 > /tmp/jenkins.log 2>&1 &.

What's going wrong?

EDIT: I was wrong, I had accidentally hit an option to have the SSH Key be in the wrong place when I did that. Now, using @borrrden's suggestion, it still gives the same error. **EDIT: As @borrrden suggested, I changed my start command to nohup java -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true -jar ~/Downloads/jenkins.war --httpPort=8081 --ajp13Port=8010 > /tmp/jenkins.log 2>&1 &, and now I get a different crash:

Started by user anonymous
Building in workspace /Users/ethen/.jenkins/workspace/Build NikNik
 > git rev-parse --is-inside-work-tree
Fetching changes from the remote Git repository
 > git config remote.origin.url git@bitbucket.org:ethenwilson/whentoact.git
Fetching upstream changes from git@bitbucket.org:ethenwilson/whentoact.git
 > git --version
using GIT_SSH to set credentials NikNik BitBucket SSH Key
 > git fetch --tags --progress git@bitbucket.org:ethenwilson/whentoact.git +refs/heads/*:refs/remotes/origin/*
FATAL: Failed to fetch from git@bitbucket.org:ethenwilson/whentoact.git
hudson.plugins.git.GitException: Failed to fetch from git@bitbucket.org:ethenwilson/whentoact.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:622)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:854)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:879)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
    at hudson.model.Run.execute(Run.java:1732)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:234)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git@bitbucket.org:ethenwilson/whentoact.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1406)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1194)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:265)
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:620)
    ... 10 more

I have the same issue on my Jenkins instance on Docker. When I try to clone repo with the same SSH key from the same workspace in the terminal it works perfectly but running Jenkins job it trowing error `Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git@gitlab.domain.com:SomeName/repo-name.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:` All suggestions below doesn't work :( — Lyserty, Oct 11 '18 at 05:31

score 20 · Answer 1 · answered Feb 08 '18 at 16:43

20

1) Go to job configuration

2) Go to the "Source Code Management" section

3) Additional behaviors > add

4) Select "Wipe out repository and force clone"

This will delete and re-clone only the workspace which is for your job. If you'd like to confirm before deleting, then I suggest echoing out the $WORKSPACE variable via a batch/bash command build step.

Also, this makes the build much slower, so I suggest removing it after one build.

answered Feb 08 '18 at 16:43

Janac Meena

3,203
35
32

2

Yup! Depending on the version of the git plugin, the option might also be called "Lightweight checkout" - which you want to disable, then re-run, and then re-enable. – Max Nov 27 '19 at 10:32

score 16 · Answer 2 · answered Nov 22 '17 at 08:35

I had this problem as well and was only able to solve it by deleting the workspace of the problematic repository on our master Jenkins server.

I think the problem was that there was a connection-error (like @gbjbaanb said) in a few of the builds (our Bitbucket crashed). This left the workspace on master in a corrupt state, and because Jenkins tries to use cached workspaces where it can, this caused every following build to fail as well.

score 8 · Answer 3 · edited Jun 20 '20 at 09:12

For me, this was hitting the 10 minute default timeout for the git-client plugin.

Solved by setting an advanced clone behaviour on the job and upping the timeout.

In the job configuration page under the Git plugin section, there is a drop-down list "Add". Within that dropdown list there is a selection "Advanced clone behaviours". When you add the advanced clone behaviors, you'll see a field for "Timeout (in minutes) for clone and fetch operation".

If you add additional behaviours before the operation you can up the timeout for clone and checkout - which has translated to a higher timeout value in my console

Advanced Checkout behaviours

Advanced Clone behaviours

Putting any value in timeout overrides the default.

Knowledge gained from JENKINS-20445.

score 2 · Answer 4 · edited Jun 20 '20 at 09:12

2

Seems like a network error:

Receiving objects: 78% (2092/2682), 4.07 MiB | 1.59 MiB/s

Corrupted MAC on input.

Disconnecting: Packet corrupt

fatal: The remote end hung up unexpectedly

fatal: early EOF

fatal: index-pack failed

suggests the network broke at 78% of the way through.

Seems to be a common problem.

edited Jun 20 '20 at 09:12

Community

1
1

answered Jul 17 '14 at 21:36

gbjbaanb

51,617
12
104
148

1

What can I do to fix it then? I've tested this to happen about fifty times in a row and every build had this, so this is not a one off thing. – Eliza Wilson Jul 17 '14 at 21:39
@EthenA.Wilson see edit, and search for 'git fatal: early EOF' for some suggested ways to fix it. – gbjbaanb Jul 17 '14 at 21:40
If that's the case then why is he able to clone from the command line with no problem? – borrrden Jul 18 '14 at 02:27
@borrrden you'll have to ask him - sometimes users say "it always works for me" when they mean "I tried it once". Maybe its just fluke. Maybe his network is QoS based on user. I don't know - I do know what 'remote end hung up' means though. – gbjbaanb Jul 18 '14 at 07:27
@borrrden I tried to use the command line to do `git fetch` around a dozen times over the course of trying to figure out this problem, and it worked every time. – Eliza Wilson Jul 18 '14 at 13:00

score 1 · Answer 5 · answered Jul 23 '14 at 15:51

I was able to solve the problem by creating a BitBucket account exclusively for Jenkins, giving it admin permission to the repository.

I then had the repository URL be: https://JenkinsAccountUsername:JenkinsAccountPassword@bitbucket.org/OwnerOfRepositoryUsername/ProjectName.git.

score 1 · Answer 6 · answered May 23 '17 at 06:58

This issue is probably caused by a timeout check in place while fetching. You can increase it by following the advice mentioned below.

In the job configuration page under the Git plugin section, there is a drop-down list "Add". Within that dropdown list there is a selection "Advanced clone behaviours". When you add the advanced clone behaviors, you'll see a field for "Timeout (in minutes) for clone and fetch operation".

Rajaraman Subramanian · Answer 7 · 2019-07-22T12:29:43.540

I faced a similar timeout issue in my windows server where my remote GIT repository was huge and very slow to clone.

I have done the following to fix the timeout issue based on the suggestions from this post.

Manually clone the repository (not necessarily git clone --mirror git@github.com:my-user/my-repository.git as I had already cloned it in a folder before I could stumble on the second suggestion. Anyway if you are starting newly probably you can clone with mirror option). This will act as my reference repository.
Configure the Source Code Management in your jenkins job as follows

Repositories: Configure this as you would normally

Branches to build: Configure this as you would normally

Repository browser: (Auto) (Default value)

Additional Behaviours: Advanced clone behaviours

Fetch Tags - Unchecked

Honor refspec on initial clone - Unchecked

Shallow clone - Checked

Shallow clone depth - 1 (We are not bothered about the whole history, only latest is enough)

Path of the reference repo to use during clone - Folder path of the repo where the entire repo is cloned (refer Step 1 above)

Timeout (in minutes) for clone and fetch operations - Left blank in my case (If you need a different timeout (default is 10 mins) you can mention it here)

score 0 · Answer 8 · answered May 14 '19 at 18:52

0

I solved a similar issue by switching 'ssh' to 'https' when connecting to BitBucket. Remember on bitbucket UI, when clicking 'clone', there's dropdown options for ssh/https. After using https the git pulling works.

answered May 14 '19 at 18:52

Emily

306
4
8

score 0 · Answer 9 · answered Nov 16 '22 at 03:22

0

I"m tyring to connect to a bitbucket repo. I was facing this issue after I have added a new credential to the Jenkins Credential store. Deleting and Re-creating the username and password credentials to connect to the repo worked for me.

answered Nov 16 '22 at 03:22

Rajeshwar

565
1
5
13

Raship Saiyed · Answer 10 · 2023-08-07T14:53:03.090

This worked for me. Click on build which is failing, Open console output, Verify which config file it is referring to.

In my case, it was referring to following file. C:\ProgramData\Jenkins@.jenkins\caches\git-b39609380153414bd1f8dfb3ac7250de.git\config.

Go to the following path and rename this config file to any name and run Build Now on Jenkins. It will create the new config file and the new build will be successful.

It happens when config is locked or failed previously.

Why is Jenkins failing when fetching from git, while the command line isn't?

10 Answers10

Linked