Jenkins pipeline git command submodule update

Question

I want to update submodule on git clone.

Is there a way to do this with Jenkins pipeline Git command?

Currently I'm doing this...

git branch: 'master',
    credentialsId: 'bitbucket',
    url: 'ssh://bitbucket.org/hello.git'

It doesn't however update submodule once cloned

I find this answer https://stackoverflow.com/a/54758293/3061838 to a similar question easier than those below. — mrtnlrsn, Jul 03 '23 at 06:04

score 62 · Answer 1 · answered Feb 17 '17 at 09:03

62

The git command as a pipeline step is rather limited as it provides a default implementation of the more complex checkout command. For more advanced configuration, you should use checkout command, for which you can pass a whole lot of parameters, including the desired submodules configuration.

What you want to use is probably something like this :

checkout([$class: 'GitSCM',
          branches: [[name: '*/master']],
          doGenerateSubmoduleConfigurations: false,
          extensions: [[$class: 'SubmoduleOption',
                        disableSubmodules: false,
                        parentCredentials: false,
                        recursiveSubmodules: true,
                        reference: '',
                        trackingSubmodules: false]], 
          submoduleCfg: [], 
          userRemoteConfigs: [[url: 'your-git-server/your-git-repository']]])

From the documentation it is often cumbersome to write these kind of lines, I recommand you use instead Jenkins very good Snippet Generator (YourJenkins > yourProject > PipelineSyntax) to automatically generate the checkout line !

answered Feb 17 '17 at 09:03

Pom12

7,622
5
50
69

1

I just ran with above and got `hudson.plugins.git.GitException: Command "git submodule update --init --recursive common" returned status code 1:` – Passionate Engineer Feb 17 '17 at 10:13
Don't you have any other output in the console logs ? If you don't, you should maybe try the failed command directly on your Jenkins instance (if you have access) to make sure your submodule "common" can be updated manually – Pom12 Feb 17 '17 at 10:38
I've tried it manually on my Jenkins instance and it doesn't work either. Basically same error with `Permission denied (publickey).` – Passionate Engineer Feb 17 '17 at 10:40
1

Okay, so that's a different problem : you don't have permission to clone your submodule repository. The submodule repository should have the same public SSH key configure in Bitbucket so you can checkout it. – Pom12 Feb 17 '17 at 10:51
Nice command (+1) But as I mention in my answer, this is more about an ssh configuration issue. – VonC Feb 17 '17 at 11:57
3

Yes, but you are giving answers and code examples for declarative pipelines syntax whereas @PassionateDeveloper seems to be using jenkins pipelines standard syntax – Pom12 Feb 17 '17 at 12:22
By adding a "checkout" does it replace the default "General SCM" that pipelines does, or is it in addition to? – pingu Jan 09 '18 at 14:49
3

For those getting a `Permission denied (publickey)`. You must have the credentials on the same line as the URL, such as: `userRemoteConfigs: [[credentialsId: '45345trete-92eb-4eb0-8844-1fd7ghj76', url: 'ssh://git@bitbucket.....git']]])` . To get the _Credentials ID_, click on _Credentials_ in the right toolbar of your Jenkins homepage. You will see a table of your credentials and **ID** is a field of that table. – vincedjango Oct 19 '18 at 20:10
3

If you use credentialsId in userRemoteConfigs, you should also use **parentCredentials: true** in subModuleOption, if not permission denied will keep coming :) – BackToBasics Sep 12 '19 at 08:30
great answer and also nice tip with the "PipelineSyntax tool", thank you! – IceFire Oct 04 '19 at 08:02
Why re-specify the branch here, why not just use scm.branches? Similar for userRemoteConfigs, why not use scm.userRemoteConfigs? – chrisinmtown Apr 25 '22 at 13:25

VonC · Accepted Answer · 2022-03-07T19:19:02.347

34

With the current Git plugin, you don't even need that.

The Git plugin supports repositories with submodules which in turn have submodules themselves.
This must be turned on though:

in Job Configuration -> Section Source Code Management, Git -> Advanced Button (under Branches to build) -> Recursively update submodules

But the OP is using pipeline.

So a simple first build step is enough:

git submodule update --init --recursive

However, the OP adds:

Yes but if I'm using sh 'git submodule update --init --recursive', this will use $HOME/id_rsa right? I want to pass in my private key for this command if possible.

It is possible: In the Pipeline syntax, you can define environment variables.
Which means you can set GIT_SSH_COMMAND (with Git 2.10+).
That allows you to reference your own private key.

pipeline {
    agent any

    environment {
        GIT_SSH_COMMAND = 'ssh -i /path/to/my/private/key'
    }

    stages {
        stage('Build') {
            steps {
                sh 'printenv'
                sh 'git submodule update --init --recursive'
            }
        }
    }
}

If any clone involve an ssh url, that ssh clone will use the right private key.

Note that sancelot points out in the comments:

unfortunately this does not work: JENKINS-38860

The error reported above:
FATAL: Command "git config --get submodule.MySubModule.url" 
returned status code 1
Occurs for me whenever you have nested submodules.

Consider a scenario in which repo A contains submodule B, which contains submodule C.

If "Advanced submodule behaviour" with "Recursively update submodules" is not enabled, Jenkins will clone A, checkout/clone B, and fail to initialise/clone C.
This is probably expected behaviour.

If you enable "Recursively update submodules", you get the error:
FATAL: Command "git config --get submodule.C.url"
returned status code 1

edited Mar 07 '22 at 19:19

answered Feb 17 '17 at 05:39

VonC

1,262,500
529
4,410
5,250

2

This is with git pipeline though. How can I configure with Pipeline project? – Passionate Engineer Feb 17 '17 at 05:59
@PassionateDeveloper no pipeline required: the regular Git plugin is enough. – VonC Feb 17 '17 at 06:05
I'm using Pipeline script and it calls ```git``` command inside it. I don't have option to change Job Configuration -> Section Source Code Management. – Passionate Engineer Feb 17 '17 at 06:13
@PassionateDeveloper then a build step in your pipeline script is enough: git submodule update --ini- --recursive as mention"ed – VonC Feb 17 '17 at 06:17
1

Yes but if I'm using ```sh 'git submodule update --init --recursive'``` this will use $HOME/id_rsa right? I want to pass in my private key for this command if possible... – Passionate Engineer Feb 17 '17 at 06:49
@PassionateDeveloper OK, I have edited my answer to address that specific use case with Jenkins pipeline. – VonC Feb 17 '17 at 07:18
Thanks. Is there a way to use credentials id instead? – Passionate Engineer Feb 17 '17 at 07:46
@PassionateDeveloper what do you mean "credential id"? Credentials are for https url (username/password). Or do you refer to passphrase protexted private ssh keys? – VonC Feb 17 '17 at 07:47
yes. Credentials stored as "SSH Username and private key" in Jenkins can be retrieved via credentialsId when doing git clone in pipeline. I'm trying to find a clean way to do `sh 'git submodule update --init'` inside pipeline – Passionate Engineer Feb 17 '17 at 07:55
@PassionateDeveloper not that I know of: I see `credentialsId` in https://issues.jenkins-ci.org/browse/JENKINS-26085, so check if your build step used *after* the git clone still uses that. If not, maybe https://support.cloudbees.com/hc/en-us/articles/204897020-Fetch-a-userid-and-password-from-a-Credential-object-in-a-Pipeline-job- could help, using https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding+Plugin – VonC Feb 17 '17 at 08:04
what about adding id_rsa to ```$HOME/id_rsa``` inside Jenkins slave? I did that but it still errors with `permission denied` when I do ```git submodule update``` – Passionate Engineer Feb 17 '17 at 08:06
@PassionateDeveloper Then my answer remains the simplest way: make sure the private key is accessible from the agent where the git clone occurs, and GIT_SSH_COMMAND will make it work. – VonC Feb 17 '17 at 08:07
@PassionateDeveloper again, if you add `id_rsa` anywhere, simply reference it with its full path (not `~/...`, but `/full/complete/path/to/id_rsa`) with `GIT_SSH_COMMAND`, and this will work – VonC Feb 17 '17 at 08:08
Just tried this and it doesn't work. I'm getting the same error – Passionate Engineer Feb 17 '17 at 10:36
Do you have git 2.10 or more on your Jenkins agent? – VonC Feb 17 '17 at 10:41
It's 2.1.4 so that's why. I just ended up saving private key to home directory of the root user `/root/.ssh/id_rsa` interestingly `$HOME` of the root user was `/home/jenkins` which I was saving that private key to and wasn't working before. So I think it's ok for now. – Passionate Engineer Feb 17 '17 at 10:54
OK. But if you can, upgrade to Git 2.11.1. – VonC Feb 17 '17 at 11:02
@VonC IMPRESSIVE ANSWER - I like the way you handled all the post-question questions. Thank you! – gusgonnet Jul 24 '18 at 14:16
This works if you have once cloned git already. But with a fresh install, you will not have this luxury... why does git plugin not just offer an "recursive" option? I do not get it – IceFire Oct 04 '19 at 07:29
@IceFire At least Git, in its latest iteration, does: https://stackoverflow.com/a/3797061/6309 – VonC Oct 04 '19 at 07:33
@VonC Of course, Git has that. We talk about the Jenkins plugin here – IceFire Oct 04 '19 at 07:34
@IceFire I mean, Git has that... only since Git 2.23 (August 2019), so... the Jenkins Git plugin has some catching up to do. – VonC Oct 04 '19 at 07:35
@VonC Ah, for this specific command, you are right. But in the same answer, `git clone --recursive` is also around and this part is from 2010. I would also say that it is best practice to add submodules to Git and that, clearly, no project works if submodules are not checked out. So, it is quite awkward that this basic function is not embedded (or at least allowed for with custom parameters)... anyway, I use ' checkout' now instead of 'git' and this works fine – IceFire Oct 04 '19 at 08:00
I'm currently running Jenkins on a Windows box, and I'm trying to do something similar but also reference stored credentials in Jenkins. I use the ```withCredentials([sshUserPrivateKey...``` block to access the credentials and inside I use the ```withEnv()``` to set the GIT_SSH_COMMAND before running ```bat script: 'git submodule update --init --recursive'``` but I'm getting a failed login because the credentials aren't being used. Am I missing something? – daitienshi May 01 '20 at 13:06
1

@daitienshi Not sure: that would be best as a separate question, with the version of the OS, Git and Jenkins added to it. – VonC May 01 '20 at 13:24
unfortunately this does not work https://issues.jenkins.io/browse/JENKINS-38860 – sancelot Mar 07 '22 at 15:33
@sancelot Thank you for this feedback. I have edited the answer to make that issue visible. Do you confirm you are using the latest Git Jenkins plugin version? On which OS do you see the error? Are you using (on that Jenkins controller server) the latest Git 2.35.1.2 version? – VonC Mar 07 '22 at 19:20

score 29 · Answer 3 · answered Jul 21 '17 at 18:08

29

checkout([
    $class: 'GitSCM', 
    branches: scm.branches, 
    doGenerateSubmoduleConfigurations: false, 
    extensions: [[
      $class: 'SubmoduleOption', 
      disableSubmodules: false, 
      parentCredentials: true, 
      recursiveSubmodules: true, 
      reference: '', 
      trackingSubmodules: false
    ]], 
    submoduleCfg: [], 
    userRemoteConfigs: scm.userRemoteConfigs
  ])

answered Jul 21 '17 at 18:08

deepelement

2,457
1
25
25

1

I came back and tried to upvote ... only to notice I'd been here before ‍♂️ Anywho, thanks again. – mhvelplund Sep 21 '21 at 06:45
I prefer this answer for its use of scm.branches and scm.userRemoteConfigs, which I believe reuses the info entered in the Jenkins GUI, that avoids confusion & repetition. – chrisinmtown Apr 25 '22 at 13:29

score 2 · Answer 4 · answered Mar 25 '21 at 13:10

If you know the credential name that checkout scm is using, you can also explicitly pass that to a git command to update the submodules:

    stage ('Clone') {
        steps {
            checkout scm
            withCredentials([sshUserPrivateKey(credentialsId: 'bitbucket_ssh', keyFileVariable: 'SSH_KEY')]) {
                sh 'GIT_SSH_COMMAND="ssh -i $SSH_KEY" git submodule update --init'
            }
        }
    }

score 1 · Answer 5 · answered Sep 29 '21 at 17:10

This is a minor variation on the previous answers but I believe it to be the best solution.

I swapped the branches parameter to use env.GIT_COMMIT so that it checks out the specific commit that has been triggered via the job. Otherwise it will build the latest commit of the specific branch, in the other examples, when using scm.branches, it will build the tip of the branch instead of the expected commit.

checkout([
    $class: 'GitSCM',
    branches: [[name: "${env.GIT_COMMIT}"]],
    doGenerateSubmoduleConfigurations: false,
    extensions: [[
        $class: 'SubmoduleOption',
        disableSubmodules: false,
        parentCredentials: true,
        recursiveSubmodules: true,
        reference: '',
        trackingSubmodules: false
    ]],
    submoduleCfg: [],
    userRemoteConfigs: scm.userRemoteConfigs
    ])

Jenkins pipeline git command submodule update

5 Answers5

Linked