Access a Groovy variable inside a generated parallel pipeline in a shell

Question

I'm stuck on accessing a groovy variable inside a shell step in Jenkins pipeline.

I read Access a Groovy variable from within shell step in Jenkins pipeline and Pass groovy variable to shell script but cannot have what I want working as expected.

Here is my pipeline:

// While you can't use Groovy's .collect or similar methods currently, you can
// still transform a list into a set of actual build steps to be executed in
// parallel.

// Our initial list of strings we want to echo in parallel
def tempList = dusboard.split(',').collect{it as String}

def stringsToEcho = ["a", "b", "c", "d"]

// The map we'll store the parallel steps in before executing them.
def stepsForParallel = tempList.collectEntries {
  ["echoing ${it}" : transformIntoStep(it)]
}

// Actually run the steps in parallel - parallel takes a map as an argument,
// hence the above.
parallel stepsForParallel

// Take the string and echo it.
def transformIntoStep(tmpBoard) {
  // We need to wrap what we return in a Groovy closure, or else it's invoked
  // when this method is called, not when we pass it to parallel.
  // To do this, you need to wrap the code below in { }, and either return
// that explicitly, or use { -> } syntax.
  return {
    node('tb-tftp-server') {
      echo tmpBoard
      stage('install') {
        env.board = tmpBoard
        lock(resource: "$tmpBoard") {
          sh '''
                hwjs="hw-\${board}.js"
                echo "1 $hwjs"
                hwjs="hw-\${tmpBoard}.js"
                echo "2 $hwjs"
            '''
        }
        lock(resource: "$tmpBoard") {
          sh '''
                echo ''' + tmpBoard + ' abcd' '''
         '''
        }
      }
    }
  }
}

If I execute it with dus011,dus012 as input parameters (i.e. variable dusboard=dus011,dus012), here comes the output

[echoing dus011] Trying to acquire lock on [dus011]
[echoing dus011] Lock acquired on [dus011]
[Pipeline] [echoing dus011] {
[Pipeline] [echoing dus012] lock
[echoing dus012] Trying to acquire lock on [dus012]
[echoing dus012] Lock acquired on [dus012]
[Pipeline] [echoing dus012] {
[Pipeline] [echoing dus011] sh
[echoing dus011] [lionel_test3] Running shell script
[echoing dus011] + hwjs=hw-dus012.js
[echoing dus011] + echo '1 hw-dus012.js'
[echoing dus011] 1 hw-dus012.js
[echoing dus011] + hwjs=hw-.js
[echoing dus011] + echo '2 hw-.js'
[echoing dus011] 2 hw-.js
[Pipeline] [echoing dus012] sh
[echoing dus012] [lionel_test3@2] Running shell script
[Pipeline] [echoing dus011] }
[echoing dus011] Lock released on resource [dus011]
[Pipeline] [echoing dus011] // lock
[echoing dus012] + hwjs=hw-dus012.js
[echoing dus012] + echo '1 hw-dus012.js'
[echoing dus012] 1 hw-dus012.js
[echoing dus012] + hwjs=hw-.js
[echoing dus012] + echo '2 hw-.js'
[echoing dus012] 2 hw-.js
[Pipeline] [echoing dus011] lock
[echoing dus011] Trying to acquire lock on [dus011]
[echoing dus011] Lock acquired on [dus011]
[Pipeline] [echoing dus011] {
[Pipeline] [echoing dus011] sh
[echoing dus011] [lionel_test3] Running shell script
[Pipeline] [echoing dus012] }
[echoing dus012] Lock released on resource [dus012]
[Pipeline] [echoing dus012] // lock
[echoing dus011] + echo dus011 abcd
[echoing dus011] dus011 abcd
[Pipeline] [echoing dus012] lock
[echoing dus012] Trying to acquire lock on [dus012]
[echoing dus012] Lock acquired on [dus012]
[Pipeline] [echoing dus012] {
[Pipeline] [echoing dus012] sh
[echoing dus012] [lionel_test3@2] Running shell script
[Pipeline] [echoing dus011] }
[echoing dus011] Lock released on resource [dus011]
[Pipeline] [echoing dus011] // lock
[Pipeline] [echoing dus011] }
[echoing dus012] + echo dus012 abcd
[echoing dus012] dus012 abcd
[Pipeline] [echoing dus011] // stage
[Pipeline] [echoing dus011] }
[Pipeline] [echoing dus011] // node
[Pipeline] [echoing dus011] }
[echoing dus011] Failed in branch echoing dus011
[Pipeline] [echoing dus012] }
[echoing dus012] Lock released on resource [dus012]
[Pipeline] [echoing dus012] // lock
[Pipeline] [echoing dus012] }
[Pipeline] [echoing dus012] // stage
[Pipeline] [echoing dus012] }
[Pipeline] [echoing dus012] // node
[Pipeline] [echoing dus012] }
[echoing dus012] Failed in branch echoing dus012
[Pipeline] // parallel
[Pipeline] End of Pipeline
Also:   java.lang.NullPointerException: Cannot get property '
             ' on null object
java.lang.NullPointerException: Cannot get property '
             ' on null object

We can see the lock command has the correct board: one time dus011, another time dus012 ==> good

But this one is wrong [echoing dus011] + hwjs=hw-dus012.js it should be hw-dus011.js, not hw-dus012.js. I think it's because it's an environment variable and the second // stage overwrite the first one ==> bad

Other prints for board name are wrong.

So I tried with the '''++' syntax. Sounds good because this time board name matches [echoing dus011] dus011 abcd and [echoing dus012] dus012 abcd but then Java exception :(

Where is my mistake? how to solve that?

Thank you for your help

mkobit · Accepted Answer · 2018-04-04T22:53:57.333

1

I believe this could be happening because inside your parallel block you are modifying the global env variable by doing env.board = tmpBoard. You have a race condition between those blocks based on write/read access.

I think the best way to do this would be to instead use the withEnv inside of your blocks to isolate the context in which the environment is set.

Instead of

env.board = tmpBoard

you could do

withEnv(["board=$tmpBoard"]) {
    // block of code needing board as environment variable
}

edited Apr 04 '18 at 22:53

answered Apr 04 '18 at 14:26

mkobit

43,979
12
156
150

Except the fact I used withEnv(["board=$tmpBoard"]) to expand tmpBoard into board it worked. Thanks a lot – Lionelon Apr 04 '18 at 22:53
Good catch, thanks for pointing that out @Lionelon . Edited the answer. – mkobit Apr 04 '18 at 22:54

Access a Groovy variable inside a generated parallel pipeline in a shell

1 Answers1