How to track down a Jenkins failure to checkout git submodule over ssh?

Question

I'm using bitbucket and Jenkins to do my builds. My project has a submodule that needs to be checked out during the build process. The top level git repository that I'm using is working fine. It pulls that out of bitbucket using deployment keys with SSH access no problem. I have the same deployment key on both repositories. I also made my submodule defined relatively to my project:

[submodule "platform/web-app/WEB-INF/learning"]
    path = platform/web-app/WEB-INF/learning
    url = ./../xxxxxxxx.git

However, when Jenkins builds the project it pulls down the top level repository and fails during the submodule. Jenkins is building this using the EC2 plugin so it's built on a slave box that is started up on demand. I've tried to execute git manually on the box, but it fails the same way. I don't fully understand how the ssh key gets pushed onto the box, but I don't understand how to debug this. Why does this work for one module, but fails for the submodule?

Here is the output:

FATAL: Command "git submodule update --init --recursive" returned status code 1:
stdout: Cloning into 'platform/web-app/WEB-INF/learning'...

stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.
Clone of 'git@bitbucket.org:xxxxxxxx/xxxxxxx.git' into submodule path 'platform/web-app/WEB-INF/learning' failed

hudson.plugins.git.GitException: Command "git submodule update --init --recursive" returned status code 1:

stdout: Cloning into 'platform/web-app/WEB-INF/learning'...

stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.
Clone of 'git@bitbucket.org:xxxxxxxx/xxxxxx.git' into submodule path 'platform/web-app/WEB-INF/learning' failed

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1148)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1125)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1121)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:937)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.submoduleUpdate(CliGitAPIImpl.java:598)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.submoduleUpdate(CliGitAPIImpl.java:579)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:299)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:280)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:239)
    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:328)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

could it be because of http://stackoverflow.com/a/18912392/6309? Or http://stackoverflow.com/questions/19550590/hudson-jenkins-how-to-access-a-private-git-repository-on-bitbucket-com#comment29052778_19550590? (ie the dedicated acount used for Jankins doesn't have a `~/.ssh/` with the right private/public ssh key) — VonC, Mar 12 '14 at 07:32
Also this thread can help highlight the steps necessary for a Jenkins to connect to a Bitbucket repo: https://answers.atlassian.com/questions/85436/connecting-bitbucket-and-jenkins — VonC, Mar 12 '14 at 07:33
It could be, but this is a ec2 machine that is spun up to perform the build so the state of the box is dropped each time. If it didn't have a ~/.ssh, and it doesn't, how could it check out the initial repo without issue then fail on the submodule? How is it working now if there is no .ssh? — chubbsondubs, Mar 12 '14 at 14:05
Then this is your problem right there. Maybe the initial repo doesn't use an ssh url, but an https one, allowing for a public anonymous (not authenticated) access. While the submodule has an ssh url registered in the `.gitmodules` of the parent repo. — VonC, Mar 12 '14 at 14:23
I've confirmed the configuration is set to use ssh. From the logs: using GIT_SSH to set credentials Jenkins Bitbucket Access Key. And it says that twice which makes me think it's using the same key to access the submodule repo. — chubbsondubs, Mar 12 '14 at 21:40
I gave up and pulled it down as a separate project using the MultiSCM plugin and git. Then did a link from one project to the next. I think submodules are basically unusable wrt to authentication. The author never got back with me if it should or shouldn't work. — chubbsondubs, Mar 17 '14 at 16:22

score 2 · Accepted Answer · answered Mar 31 '14 at 14:32

2

I think you're encountering the problem reported here: https://issues.jenkins-ci.org/browse/JENKINS-20941

From the symptoms I'm seeing, the credentials you put into Jenkins are used for checking out the top level repo, but they're not used when trying to get the submodule updates.

There are two angles for a workaround:

Put the key file(s) onto all your build slaves
Update your job configurations to update explicitly

Or you can just wait for the maintainers to fix it, or fix it yourself :-)

Hope this helps

answered Mar 31 '14 at 14:32

randomsimon

471
4
11

1

I worked around this by checking out each module separately using MultiSCM plugin. I'm glad to know that this IS in fact a bug, and I wasn't going crazy. – chubbsondubs Apr 01 '14 at 15:54
What does "update explicitly" mean in the second angle? I tried the build step, and also some of the submodule options of the plugin, but none work. Thanks. – Efren Aug 26 '16 at 01:07
By "update explicitly" I meant doing what wierzbiks suggests below - adding a build step that runs the commands that Jenkins can't get right itself. Admitedly, I went for the option of putting the key files onto all the slaves, so I can't say precisely what commands would be needed. Sorry 'bout that. – randomsimon Sep 29 '16 at 15:49

score 1 · Answer 2 · answered Apr 24 '15 at 09:31

1

I'm using ssh-agent with specific credential in job configuration. Then in build step I have "Exec shell" component with:

git submodule init
git submodule sync
git submodule update --init --recursive

answered Apr 24 '15 at 09:31

wierzbiks

1,148
10
10

How to track down a Jenkins failure to checkout git submodule over ssh?

2 Answers2