Running same Jenkins job on multiple agents in parallel in declarative pipeline

Question

here's what I have:

#!/usr/bin/env groovy

pipeline {
    agent none
    stages {
        stage('Checkout SCM') {
            agent { label 'win' && 'apple' && 'rhel' }
            steps {
                echo "Cloning Repository"
                checkout([$class: 'GitSCM',
                    branches: [[name: "*/develop"]],
                    doGenerateSubmoduleConfigurations: false,
                    extensions: [[$class: 'WipeWorkspace']],
                    submoduleCfg: [],
                    userRemoteConfigs: [[credentialsId: 'UNAME', url: 'URL']],
                    browser: [$class: 'BitbucketWeb', repoUrl: 'URL'],
                ])}}

        stage('Building Win64, Linux764, MacOS') {
            agent { label 'win&&rhel&&apple' }
            steps {
                   script {
                        echo '************************'
                        echo '*****BUILDING JOBS******'
                        echo '************************'
                        sh 'python build.py'
                        sh 'cd ion-js && npm run prepublishOnly'
                      }}}
    }
}

However I get the There are no nodes with the label ‘win && rhel && apple’ error. Does anyone happen to know how to run a declarative jenkins pipeline where one of the stages is ran on multiple agent labels in parallel?

I want to checkout the same git repo to 3 different nodes at the same time. I've tried agent { label 'win' && 'apple' && 'rhel' } and agent { label 'win&&apple&&rhel' } but it just says it can't find that label.

Here they say you can use || and using && should work but I'm not sure what I'm missing. I could write 3 different checkout stages but I figured there was a better way

i have a similar problem, did you figure out how to resolve this? — mancocapac, Apr 26 '18 at 04:58

score 5 · Answer 1 · edited May 06 '22 at 14:09

To add more to answer from Micah, Instead of repeating stage with different labels, you can define a function which can create a stage for you and will be executing the generate stage on different agent nodes of your choice.

def agents  = ['win64', 'linux64', 'macos']
 
def generateStage(nodeLabel) {
    return {
        stage("Runs on ${nodeLabel}") {
            node(nodeLabel) {
               script {
                    echo "Running on ${nodeLabel}"
                    echo '************************'
                    echo '*****BUILDING JOBS******'
                    echo '************************'
                    sh 'python build.py'
                    sh 'cd ion-js && npm run prepublishOnly'
                }
            }
        }
    }
}
def parallelStagesMap = agents.collectEntries {
    ["${it}" : generateStage(it)]
}
pipeline {
    agent none
    stages {
        stage('non-parallel stage') {
            steps {
                echo 'This stage will be executed first.'
            }
        }
        stage('parallel stage') {
            steps {
                script {
                    parallel parallelStagesMap
                }
            }
        }       
    }
}

Since we have parallel keyword while calling parallelStageMap, the same stage will be executed in parallel on the agent nodes.

ProTip: You can define more steps inside the function which are common to be executed on all agents. If you wanted to defined label and stage name, you can added another argument named stagename and can parse into the function generateStage.

I got error `No such DSL method 'steps' found` while using your solution. In "generateStage" function, I replaced steps -> stage, and succeeded. — Narek Hambardzumyan, Mar 29 '22 at 11:14
Yep, the `steps` is a Declarative Pipeline syntax and because the `parallel` step is enclosed in a `script` block, it is expected that the `generateStage` will return Scripted Pipeline syntax in the closure. — edwardTew, May 05 '22 at 21:04

score 3 · Answer 2 · answered Jun 01 '18 at 13:47

I have tried the same things with no success. The only solution I'm aware of is to have a parallel block and define the stage multiple times, for each agent/node/label.

stage('Building Win64, Linux764, MacOS') {
  parallel {
    stage('Win64') {
      agent {
        label 'win-10-x64'
      }
      steps {
      ...
      }
    }
    stage('Linux64') {
      agent {
        label 'linux-x64'
      }
      steps {
      ...
      }
    }
    stage('MacOS') {
      agent {
        label 'macos'
      }
      steps {
      ...
      }
    }
  }
}

Running same Jenkins job on multiple agents in parallel in declarative pipeline

2 Answers2

Linked