Using a Jenkins pipeline to checkout multiple git repos into same job

Question

I'm using the Jenkins Multiple SCM plugin to check out three git repositories into three sub directories in my Jenkins job. I then execute one set of commands to build a single set of artifacts with information and code drawn from all three repositories.

Multiple SCM is now depreciated, and the text recommends moving to pipelines. I tried, but I can't figure out how to make it work.

Here is the directory structure I'm interested in seeing from the top level of my Jenkins job directory:

$ ls
Combination
CombinationBuilder
CombinationResults

Each of those three sub-directories has a single git repo checked out. With the Multiple SCM, I used git, and then added the "checkout to a subdirectory" behavior. Here was my attempt with a pipeline script:

node('ATLAS && Linux') {
    sh('[ -e CalibrationResults ] || mkdir CalibrationResults')
    sh('cd CalibrationResults')
    git url: 'https://github.com/AtlasBID/CalibrationResults.git'
    sh('cd ..')
    sh('[ -e Combination ] || mkdir Combination')
    sh('cd Combination')
    git url: 'https://github.com/AtlasBID/Combination.git'
    sh('cd ..')
    sh('[ -e CombinationBuilder ] || mkdir CombinationBuilder')
    sh('cd CombinationBuilder')
    git url: 'https://github.com/AtlasBID/CombinationBuilder.git'
    sh 'cd ..'

    sh('ls')
    sh('. CombinationBuilder/build.sh')
}

However, the git command seems to execute at the top level directory of the workspace (which makes some sense), and according to the syntax too, there doesn't seem to be the checkout-to-sub-directory behavior.

Jenkins makes this so hard, it's like hammer and chisels compared to TeamCity which makes this feature so elegant. — C.J., Oct 04 '19 at 16:54
That is 2022 and I believe this plugin was deprecated. What is the current way of using multiple repos in a same job? — stackMeUp, May 24 '22 at 15:03
@stackMeUp multiple repos in the same job makes sense for example when using Gradle composite builds — Sebastian Zubrinic, Sep 14 '22 at 15:19

score 132 · Accepted Answer · edited May 31 '17 at 10:14

132

You can use the dir command to execute a pipeline step in a subdirectory:

node('ATLAS && Linux') {
    dir('CalibrationResults') {
        git url: 'https://github.com/AtlasBID/CalibrationResults.git'
    }
    dir('Combination') {
        git url: 'https://github.com/AtlasBID/Combination.git'
    }
    dir('CombinationBuilder') {
        git url: 'https://github.com/AtlasBID/CombinationBuilder.git'
    }

    sh('ls')
    sh('. CombinationBuilder/build.sh')
}

edited May 31 '17 at 10:14

Paul T. Rawkeen

3,994
3
35
51

answered Oct 24 '16 at 18:36

Wilco Greven

2,085
1
10
9

8

Perfect, that did the trick, thanks! I'm slowly learning the pipelines. I wish all of software would just decide on one DSL for doing workflow. I've had to learn so many. :-) – Gordon Oct 26 '16 at 18:19
3

how do you go back to the $WORKSPACE directory/the pipeline directory? I have to return to the global context to execute some scripts. The dir switches the context and leaves it there. The last 'ls' command would print contents of the CombinationBuilder repo. – Daniel Colceag Sep 14 '17 at 17:41
2

How do you use different credentials for the repos? Ie say my "main" repo (that Jenkins is monitoring to trigger the job) lives in Bitbucket, and I want to clone an existing Github repo inside the build, how do I specify one set of creds for Bitbucket & a different one for Github? – Adam Parkin Oct 10 '17 at 18:21
12

To use different credentials use the optional parameter 'credentialsId' on the git command. e.g. `git url: 'https://github.com/AtlasBID/CombinationBuilder.git', credentialsId: 'foo'` where 'foo' is a credential id from Jenkins. – Magneon Nov 07 '17 at 18:00
Where is the documentation for `git`? I want to see what all arguments can be passed to it. Via google, I am only seeing bits and pieces from other forum posts, but can't find the official documentation. – haridsv May 24 '18 at 15:46
If the pipeline polls scm and a commit is made to one of these repositories, how do you know later in the pipeline which one exactly was the trigger? – Nikolai Koudelia Jan 24 '19 at 19:13
How can I define the repositories as part of the Jenkins job instead of hard-coding them into the Jenkins file? – shlomiLan May 13 '19 at 17:52
2

@haridsv, documentation for `git` is here: https://jenkins.io/doc/pipeline/steps/git/ – Nick Jones Mar 05 '20 at 14:31

score 45 · Answer 2 · answered Aug 21 '18 at 14:36

You can checkout those three git repositories into three subdirectories by using the checkout SCM step three times like this:

stage('Checkout') {
 // Get CalibrationResults from GitHub
 checkout([  
            $class: 'GitSCM', 
            branches: [[name: 'refs/heads/master']], 
            doGenerateSubmoduleConfigurations: false, 
            extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'CalibrationResults']], 
            submoduleCfg: [], 
            userRemoteConfigs: [[credentialsId: '6463627-ab54-4e42-bc29-123458', url: 'https://github.com/AtlasBID/CalibrationResults.git']]
        ])
 // Get Combination from GitHub
 checkout([  
            $class: 'GitSCM', 
            branches: [[name: 'refs/heads/master']], 
            doGenerateSubmoduleConfigurations: false, 
            extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'Combination']], 
            submoduleCfg: [], 
            userRemoteConfigs: [[credentialsId: '6463627-ab54-4e42-bc29-123458', url: 'https://github.com/AtlasBID/Combination.git']]
        ])
 // Get CombinationBuilder from GitHub
 checkout([  
            $class: 'GitSCM', 
            branches: [[name: 'refs/heads/master']], 
            doGenerateSubmoduleConfigurations: false, 
            extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'CombinationBuilder']], 
            submoduleCfg: [], 
            userRemoteConfigs: [[credentialsId: '6463627-ab54-4e42-bc29-123458', url: 'https://github.com/AtlasBID/CombinationBuilder.git']]
        ])

}

For me this was the perfect solution as I have complex options. — Sean83, Sep 06 '18 at 14:18
fantastic extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'CalibrationResults']], was what i needed, just say the name of the folder were will put the cloned repo, thanks. — pelos, Jul 30 '21 at 21:08
The git plugin warns against using this extension in the pipeline: "This extension should not be used in Jenkins Pipeline (either declarative or scripted). Jenkins Pipeline already provides standard techniques for checkout to a subdirectory. Use ws and dir in Jenkins Pipeline rather than this extension." — Haydon, Feb 01 '23 at 13:00

score 21 · Answer 3 · answered Mar 07 '19 at 08:27

21

Here's Mine

    stage('CheckoutModule1') {
        steps {
            sh 'mkdir -p Module1'
            dir("Module1")
            {
                git branch: "develop",
                credentialsId: 'aaa',
                url: 'git@a.com:b/module1.git'
            }
        }
    }

    stage('CheckoutModule2') {
        steps {
            sh 'mkdir -p Module2'
            dir("Module2")
            {
                git branch: "develop",
                credentialsId: 'aaa',
                url: 'git@a.com:b/module2.git'
            }
        }
    }

answered Mar 07 '19 at 08:27

yan

1,382
14
11

Worked like a charm for me! Thanks. – MeowRude May 28 '19 at 20:05
very helpful response – Imran Aug 22 '19 at 11:27
3

This appears to require the [Git Parameter plugin](https://plugins.jenkins.io/git-parameter/) – SyntaxRules Mar 11 '20 at 16:52
you can simplify this, no need for the mkdir command since the dir() will create the directory if it doesn't exist – BarretV Feb 10 '23 at 19:51

score 6 · Answer 4 · answered Aug 08 '18 at 08:41

If your repository has submodules use git checkout

pipeline {
agent {label 'master'}
stages{
    stage("Demo"){
        steps{

            echo 'Hello World'
        }
    }
    stage("Source"){
        parallel{
            stage('CalibrationResults'){
                steps{
                    echo 'Checking out CalibrationResults'
                    checkout([$class: 'GitSCM', branches: [[name: '*/CI-CD-Demo']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: false, timeout: 60], [$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: false, recursiveSubmodules: true, reference: '', timeout: 60, trackingSubmodules: true], [$class: 'RelativeTargetDirectory', relativeTargetDir: 'server-core'],[$class: 'CheckoutOption', timeout: 60]], submoduleCfg: [], userRemoteConfigs: [[url: 'https://github.com/AtlasBID/CalibrationResults.git']]])
                }
            }
            stage('Combination'){

                steps{
                    echo 'Checking out server spoke'
                    checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: false, timeout: 60], [$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: false, recursiveSubmodules: true, reference: '', timeout: 60, trackingSubmodules: true], [$class: 'RelativeTargetDirectory', relativeTargetDir: 'server-spoke'], [$class: 'CheckoutOption', timeout: 60]], submoduleCfg: [], userRemoteConfigs: [[url: 'https://github.com/AtlasBID/CombinationBuilder.git']]])


                }
            }
        }

    }

}
}

Generated using Checkout git snippet generator

Any idea how to checkout submodules at a given branch? – Arnold Roa May 22 '21 at 10:27 — Arnold Roa, May 22 '21 at 10:27

user19751415 · Answer 5 · 2022-08-12T14:14:02.920

0

You can just add a shell step that does the cloning and moves it to a subdirectory of your choice.

git clone https://$bitbucketUsername:$bitbucketPassword@<yourbitbucketserver>/scm/projectname/reponame1.git
mv reponame1 new_subdir_name1

git clone https://$bitbucketUsername:$bitbucketPassword@<yourbitbucketserver>/scm/projectname/reponame2.git
mv reponame2 new_subdir_name2

edited Aug 12 '22 at 14:14

answered Aug 12 '22 at 14:12

user19751415

1
1

Using a Jenkins pipeline to checkout multiple git repos into same job

5 Answers5

Linked