Jenkins Pipeline: no permission to create directories inside script block with .execute()

Question

I want to run an external shell command (for example, git clone) inside a Jenkins pipeline.

I have found 2 ways of doing this.

This one works:

steps {
    sh "git clone --branch $BRANCH --depth 1 --no-single-branch $REMOTE $LOCAL
}

Downsides:

I only see the output when the complete command is finished. Which is annoying if the command takes a long time.
I need to do some Groovy scripting to look up values in a Map, based on parameters chosen by the user who starts the build. Haven't found a way to do that without a script {} block.
A variation is to run a Bash script that runs the git clone command, that also works. Which will get me into trouble when running on Windows nodes.

The next one errors on

fatal: could not create work tree dir 'localFolder'.: Permission denied

steps {
    script {
        def localFolder = new File(products[params.PRODUCT].local)
        if (!localFolder.exists()) {
            def gitCommand = 'git clone --branch ' + params.BRANCH + ' --depth 1 --no-single-branch ' + products[params.PRODUCT].remote + ' ' + localFolder
            runCommand(gitCommand)
        }
    }
}

This is the runCommand() wrapper:

def runCommand = { strList ->
    assert ( strList instanceof String ||
            ( strList instanceof List && strList.each{ it instanceof String } ) \
)
    def proc = strList.execute()
    proc.in.eachLine { line -> println line }
    proc.out.close()
    proc.waitFor()

    print "[INFO] ( "
    if(strList instanceof List) {
        strList.each { print "${it} " }
    } else {
        print strList
    }
    println " )"

    if (proc.exitValue()) {
        println "gave the following error: "
        println "[ERROR] ${proc.getErrorStream()}"
    }
    assert !proc.exitValue()
}

My question is: how come I have permission to create directories when running a sh command, and how come I don't have that permission when I do the same thing inside a script {} block with .execute()?

I'm intentionally using the example of the git clone command to avoid getting answers that don't read the question, like using a dir {} block. If I can't create the git directory, then I can also not create the files inside that directory.

If you want to safely create directories, files etc., you need to use pipeline steps like `writeFile`, `readFile`, `fileExists` etc. All code like `new File(...)` gets executed on the master node only, not on the node that executes current stage and stores the workspace. More information here -> https://stackoverflow.com/questions/51233919/create-a-file-with-some-content-using-groovy-in-jenkins-pipeline/51234067#51234067 — Szymon Stepniak, Dec 17 '19 at 12:01

Szymon Stepniak · Accepted Answer · 2019-12-17T14:25:30.017

1

If you want to run any shell commands, use sh step, not Groovy's process execution. There is one main reason for that - any Groovy code you execute inside the script block, gets executed on the master node. And this is (probably) the reason you see this permission denied issue. The sh step executes on the expected node and thus creates a workspace there. And when you execute a Groovy code that is designed to create a folder in your workspace, it fails, because there is no workspace on a master node.

"1. Except for the steps themselves, all of the Pipeline logic, the Groovy conditionals, loops, etc execute on the master. Whether simple or complex! Even inside a node block!"

^{Source: https://jenkins.io/blog/2017/02/01/pipeline-scalability-best-practice/#fundamentals}

However, there is a solution to that. You can easily combine the sh step with the script block. There is no issue with using any of the existing Jenkins pipeline steps inside the script block. Consider the following example:

steps {
    script {
        def localFolder = products[params.PRODUCT].local
        if (!fileExists(file: localFolder)) {
            sh 'git clone --branch ' + params.BRANCH + ' --depth 1 --no-single-branch ' + products[params.PRODUCT].remote + ' ' + localFolder
        }
    }
}

Keep in mind that this example uses fileExists and readFile steps to check if file exists in the workspace, as well as to read its content. Using new File(...) won't work correctly when your workspace is shared between master and slave nodes.

If you want to safely create files in the workspace(s), use writeFile step to make sure that the file is created on the node that executes your pipeline's current stage.

edited Dec 17 '19 at 14:25

answered Dec 17 '19 at 11:51

Szymon Stepniak

40,216
10
104
131

I'm slowly starting to understand it.... turns out that `readFile` won't work, because of `java.io.IOException: Is a directory`. And is it possible that `fileExists` returns false when it's a directory? I'm currently going through the entire list of Pipeline steps to see if one exists that checks if a directory, not a file, exists, but haven't found one yet. – Amedee Van Gasse Dec 17 '19 at 13:51
The `fileExists` step works for any type of files, including directories. Just double-checked that in one of my pipelines - it returns `true` for the folder that exists in the workspace. – Szymon Stepniak Dec 17 '19 at 14:05
Okay. But `readFile` definitely gives an `IOException` when used on a directory. – Amedee Van Gasse Dec 17 '19 at 14:06
For this very specific use case, I decided to just use the `checkout scm` step wrapped inside a `script` block (to get the correct variables). It works, and I don't have to bother if the directory already exists or not. I'll tackle the problem of checking if a directory exists in a workspace when I encounter it again. – Amedee Van Gasse Dec 17 '19 at 14:16
Upvoted and accepted, even though it wasn't the solution that I chose, it still explains my initial problem. – Amedee Van Gasse Dec 17 '19 at 14:16
It was impossible to deduct from your question that you are looking for the equivalent of the `checkout scm` step. The `fileExists` step throws an exception for the file directories because it is designed to read contents from files. The directory has no content in that matter. Now I see more clearly what you tried to achieve, and in this case using `readFile` was not needed. Will update my answer to not confuse anyone. – Szymon Stepniak Dec 17 '19 at 14:24
Thanks to your answer, I was able to write the following code in another stage: def doxyFile = 'doxyfile' def doxyConfig = "OUTPUT_DIRECTORY = ../doxydocs_${PRODUCT}_${GIT_REFERENCE}" if (fileExists(file: doxyFile)) { doxyConfig = readFile(doxyFile).replaceAll(/OUTPUT_DIRECTORY *= .*/, doxyConfig) } writeFile file: doxyFile, text: doxyConfig As you can see, I used `fileExists`, `readFile` and `writeFile`. – Amedee Van Gasse Dec 18 '19 at 13:00

score 0 · Answer 2 · answered Dec 17 '19 at 11:46

A solution to my problem:

Don't bother with showing output as a command progresses, just deal with it that I will only see it at the end.
Compose the entire command inside a script {} block.
put a sh statement inside the script {} block.

Like this:

steps {
    script {
        def localFolder = new File(products[params.PRODUCT].local)
        if (!localFolder.exists()) {
            def gitCommand = 'git clone --branch ' + params.BRANCH + ' --depth 1 --no-single-branch ' + products[params.PRODUCT].remote + ' ' + localFolder
            sh gitCommand
        }
    }
}

This still doesn't answer my question about the permission issue, I would still like to know the root cause.

Jenkins Pipeline: no permission to create directories inside script block with .execute()

2 Answers2