Jenkins task for remote hosts

Question

In deploy scenario i need to create and run jenkins task on list of hosts, i.e. create something like parametrized task (where ip address is a parameter) or a task on Multijob Plugin with HOST axis, but run by only 2 ones in parallel over multiple hosts.

One of the option could be to run ansible with the list of hosts, but i'd like to see a status per each host separately, and relaunch a jenkins job if needed.

The main option is to use Job DSL Plugin or Pipeline Plugin, but here i need help to understand what classes/methods of dsl groovy code should be used to achieve this.
Can anyone help with it?

Can't you use the [Matrix project](https://wiki.jenkins-ci.org/display/JENKINS/Building+a+matrix+project) with an axis for the IP addresses? — vlp, Oct 09 '16 at 19:59
@vlp Maybe yes, but how to run only 2 in parallel instead of all ? — yvs, Oct 11 '16 at 18:06
I've never used it, but [Throttle Concurrent Builds Plugin](https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin) might work. It should support throttling matrix builds (it is mentioned it in the changelog). Good luck! — vlp, Oct 11 '16 at 19:57

jil · Accepted Answer · 2016-10-11T18:15:06.717

2

Assume that the hosts have been configured as Jenkins slaves already. Assume that hosts are provided in pipeline job parameter HOSTS as whitespace separated list. Following example should get you started:

def hosts_pairs = HOSTS.split().collate(2)

for (pair in host_pairs) {
  def branches = [:]
  for (h in pair) {
    def host = h  // fresh variable per iteration; it will be mutated
    branches[host] = {
      stage(host) {
        node(host) {
          // do the actual job here, e.g. 
          // execute a shell script
          sh "echo hello world"
        }
      }
    }
  }
  parallel branches
}

edited Oct 11 '16 at 18:15

answered Oct 08 '16 at 19:37

jil

2,601
12
14

could you clarify please, one point, AFA i'm not fluent in groovy: should it be collapse instead of collate? – yvs Oct 11 '16 at 14:21
1

No, collate is correct. It will divide the list into sub lists of given size. E.g. here if you have string `HOSTS="foo bar baz qux"`, after `split()` you will have a flat list `["foo", "bar", "baz", "qux"]` and after `collate(2)` you will have a list of lists `[["foo", "bar"], ["baz", "quz"]]`. – jil Oct 11 '16 at 18:11

score 2 · Answer 2 · answered Oct 11 '16 at 22:32

A combination of Matrix project and Throttle Concurrent Builds Plugin is possible.

All you need is to setup a single user-defined axis (e.g. "targetHost") with all IP addresses as values and set the desired throttling under "Throttle Concurrent Builds" (please note that you have to enable the "Execute concurrent builds if necessary" option to tell jenkins to allow concurrent execution).

The axis values are available during every child build in the corresponding environment variable (e.g. targetHost).

Below is an example config.xml with simple ping&wait build step:

<?xml version='1.0' encoding='UTF-8'?>
<matrix-project plugin="matrix-project@1.7.1">
  <actions/>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties>
    <hudson.plugins.throttleconcurrents.ThrottleJobProperty plugin="throttle-concurrents@1.9.0">
      <maxConcurrentPerNode>2</maxConcurrentPerNode>
      <maxConcurrentTotal>2</maxConcurrentTotal>
      <categories class="java.util.concurrent.CopyOnWriteArrayList"/>
      <throttleEnabled>true</throttleEnabled>
      <throttleOption>project</throttleOption>
      <limitOneJobWithMatchingParams>false</limitOneJobWithMatchingParams>
      <matrixOptions>
        <throttleMatrixBuilds>true</throttleMatrixBuilds>
        <throttleMatrixConfigurations>true</throttleMatrixConfigurations>
      </matrixOptions>
      <paramsToUseForLimit></paramsToUseForLimit>
    </hudson.plugins.throttleconcurrents.ThrottleJobProperty>
  </properties>
  <scm class="hudson.scm.NullSCM"/>
  <canRoam>true</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers/>
  <concurrentBuild>true</concurrentBuild>
  <axes>
    <hudson.matrix.TextAxis>
      <name>targetHost</name>
      <values>
        <string>127.0.0.1</string>
        <string>127.0.0.2</string>
        <string>127.0.0.3</string>
        <string>127.0.0.4</string>
        <string>127.0.0.5</string>
      </values>
    </hudson.matrix.TextAxis>
  </axes>
  <builders>
    <hudson.tasks.Shell>
      <command>sleep 7
ping -c 7 $targetHost
sleep 7</command>
    </hudson.tasks.Shell>
  </builders>
  <publishers/>
  <buildWrappers/>
  <executionStrategy class="hudson.matrix.DefaultMatrixExecutionStrategyImpl">
    <runSequentially>false</runSequentially>
  </executionStrategy>
</matrix-project>

Good luck!

Jenkins task for remote hosts

2 Answers2