Jenkins - Running instances of single build concurrently

Question

I'd like to be able to run several builds of the same Jenkins job simultaneously.

Example:

1. Build [*jenkins_job_1*]: calls an ant script with parameter 'A'
2. Build [*jenkins_job_1*]: calls an ant script with parameter 'B'
3. repeat as necessary
4. each instance of the job runs simultaneously, rather than through a queue.

The reason I'd like to do this is to avoid having to create several jobs that are nearly identical, all of which would need to be maintained.

Is there a way to do this, or maybe another solution (ie — dynamically create a job from a base job and remove it after it's finished)?

Answer 1

Jenkins has a check box: "Execute concurrent builds if necessary"

If you check this, then it'll start multiple builds for a job.

This works with the "This build is parameterized" checkbox.

You would still trigger the builds, passing your A or B as parameters. You can use another job to trigger them or you could do it manually via a script.

Answer 2

You can select Build a Multi-configuration project (Matrix build) when you create the job. Then, under the job's configuration, you can define the Configuration Matrix which lets you specify one or more parameters (axes) for different builds. Regarding running simultaneously, you should be able to run as many simultaneous builds as you have executors (with the appropriate label).

Unfortunately, the Jenkins wiki lacks documentation about this setup. There are a couple previous SO questions, here and here, that might provide a little guidance. There was a "recent" blog post about setting up a multi-configuration job to perform builds on various platforms.

Answer 3

A newer (and better) solution is the Jenkins Job DSL Plugin.

We've been using it with great success. Our job configurations are now disposable... we can set up a huge stack of complicated jobs from some groovy files and a couple template jobs. It's great.

I'm liking it a lot more than the matrix builds, which were complicated and harder to understand.

Answer 4

Nothing stopping you doing this using the Jenkins pipeline DSL.

We have the same pipeline running in parallel in order to model combined loads for an application that exposes web services, provides a database to several external applications, receives data via several work queues and has a GUI front end. The business gives us non-functional requirements (NFRs) which our application must meet that guarantees its responsiveness even at busy times.

The different instances of the pipeline are run with different parameters. The first instance might be WS_Load, the second GUI_Load and the third Daily_Update_Load, modelling a large data queue that needs processing within a certain time-frame. More can be added depending on which combination of loads we're wanting to test.

Other answers have talked about the checkboxes for concurrent builds, but I wanted to mention another issue: resource contention.

If your pipeline uses temporary files or stashes files between pipeline stages, the instances can end up pulling the rug from under each others' feet. For example you can end up overwriting a file in one concurrent instance while another instance expects to find the pre-overwritten version of the same stash. We use the following code to ensure stashes and temporary filenames are unique per concurrent instance:

    def concurrentStash(stashName, String includes) {
        /* make a stash unique to this pipeline and build
           that can be unstashed using concurrentUnstash() */
        echo "Safe stashing $includes in ${concurrentSafeName(stashName)}..."
        stash name: concurrentSafeName(stashName), includes: includes
    }
  
    def concurrentSafeName(name) {
        /* make a name or name component unique to this pipeline and build
         * guards against contention caused by two or more builds from the same
         * Jenkinsfile trying to:
         *   - read/write/delete the same file
         *   - stash/unstash under the same name
         */
        "${name}-${BUILD_NUMBER}-${JOB_NAME}"
    }
  
    def concurrentUnstash(stashName) {
        echo "Safe unstashing ${concurrentSafeName(stashName)}..."
        unstash name: concurrentSafeName(stashName)
    }

We can then use concurrentStash stashName and concurrentUnstash stashName and the concurrent instances will have no conflict.

If, say, the two pipelines both need to store stats, we can do something like this for filenames:

def statsDir = concurrentSafeName('stats')

and then the instances will each use a unique filename to store their output.

Answer 5

You can create a build and configure it with parameters. Click the This build is parameterized checkbox and add your desired param(s) in the Configuration of the build. You can then fire off simultaneous builds using different parameters.

Side note: The "Bulk Builder" in Jenkins might push it into a queue, but there's also a This bulk build is parameterized checkbox.

Answer 6

I was having a pretty large build queue and I performed below steps to run jobs in parallel in jenkins to reduce number of jobs waiting in queue

1. For each job you need to navigate to configure and select the checkbox stating "Execute concurrent builds if necessary"
2. Navigate to Manage -> Configure System -> look for "# of executors" and set the no of parallel executors you want (in my case it was set to 0 and I updated it to 2)