How to configure Jenkins GitHubPullRequestBuilder plugin using Job DSL

Question

I am setting up webhook integration between a private GitHub repository and a Jenkins build. I configure jobs exclusively using Job DSL groovy scripts (I'm open to switching to another programmatic job configuration mechanism, but I will not accept any answer that requires me to configure jobs manually). II would like to set up a commit status context and a set of custom messages based on build status.

The Job DSL API documentation embedded in Jenkins is not helpful, only giving me this signature: githubPullRequest(Closure closure), but not telling me how to construct a suitable closure.

Here are the relevant sections of my job DSL:

triggers {
    githubPush()
    githubPullRequest {
        useGitHubHooks()
        buildStatus {
            completedStatus('SUCCESS', 'Build succeeded!')
            completedStatus('FAILURE', 'Build failed. ')
            completedStatus('ERROR', 'Build errored. This is probably a problem with Jenkins or related infrastructure and not an issue with your code changes.')
        }
    }
}

(...)

scm {
    git {
        remote {
            github('privateorg/myrepo', 'ssh')
            credentials('my-credential-id')
            refspec('+refs/pull/*:refs/remotes/origin/pr/*')
        }
        branch('${sha1}')
    }
}

This errors as follows:

ERROR: (build.groovy, line 8) No signature of method: javaposse.jobdsl.dsl.helpers.triggers.TriggerContext.buildStatus() is applicable for argument types: 
(build$_run_closure1$_closure2$_closure10$_closure11) values: 
[build$_run_closure1$_closure2$_closure10$_closure11@602572cb]

Line 8 is:

buildStatus {

If I remove the entire buildStatus block, then Jenkins accepts the script and creates the job successfully. My push hooks work, but my pull request hooks don't.

I'm not a Groovy programmer, nor am I deeply familiar with any aspect of Jenkins. I understand that there is no method compatible with the DSL I've written, but I don't know where to look to find valid method signatures. I don't understand how the DSL maps to method calls well enough to find or even recognize an appropriate method and build DSL that is compatible.

Googling the error message led me to some people who had similar problems in 2016-2017: 1, 2, 3. Their issue seemed to stem from the deprecation of the Github Pull Request Builder plugin as a core, bundled plugin, and a corresponding change in syntax. That led me to discover a new syntax, given here:

triggers {
    githubPush()
    githubPullRequest {
        useGitHubHooks()
        extensions {
            'org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus' {
                buildStatus {
                    'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
                        message 'Build in progress...'
                        result 'PENDING'
                    }
                    'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
                        message 'Build succeeded! It is safe to merge ${ghprbSourceBranch} into ${ghprbTargetBranch}.'
                        result 'SUCCESS'
                    }
                    'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
                        message 'Build failed.'
                        result 'FAILURE'
                    }
                    'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
                        message 'Build errored. This is probably a problem with Jenkins or related infrastructure and not an issue with your code changes.'
                        result 'ERROR'
                    }
                }
            }
        }
    }
}

But that did not help either; the failure is essentially the same:

ERROR: (build.groovy, line 9) No signature of method: javaposse.jobdsl.dsl.helpers.triggers.TriggerContext.org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus() is applicable for argument types: 
(build$_run_closure1$_closure2$_closure10$_closure11$_closure12) values: 
[build$_run_closure1$_closure2$_closure10$_closure11$_closure12@707221f0]

Line 9 is:

'org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus' {

Amidst all of this, I'm struggling to understand the differences between a buildStatus, commitStatus, completedStatus, etc. What do these things mean?

Meanwhile, I reverted the DSL to the version without any buildStatus and tried creating a PR to see if it would trigger a build. It did not. I checked the "GitHub Hook Log":

Started on Aug 4, 2020 6:16:47 PM
Started by event from 10.101.32.177 ⇒ https://my-jenkins-host.com/github-webhook/ on Tue Aug 04 18:16:47 UTC 2020
Using strategy: Default
[poll] Last Built Revision: Revision 91170fb44c40737a6410acfba820d6555a0475bb (refs/remotes/origin/dev)
using credential my-credential-id
 > git --version # timeout=10
using GIT_ASKPASS to set credentials 
 > git ls-remote -h -- git@github.com:privateorg/myrepo.git # timeout=10
Found 64 remote heads on git@github.com:privateorg/myrepo.git
Ignoring refs/heads/branch1 as it doesn't match any of the configured refspecs
Ignoring refs/heads/branch2 as it doesn't match any of the configured refspecs
...
Ignoring refs/heads/branch64 as it doesn't match any of the configured refspecs
Done. Took 0.71 sec
No changes

Maybe the Hook Log isn't the right place to look, but the use of -h in the call to git ls-remote caused it to only list branches -- not PRs. If I use the same command locally but without -h, PRs are listed that I am confident would match my refspec.

I originally encountered these problems using CloudBees Core Client Master version 2.204.3.7, revision 3. Upgrading to latest (2.235.2.3) did not help.

Plugin versions in use:

• Job DSL: 1.77
• GHPRB: 1.42.1

If there are other plugins in play that are relevant here, let me know and I'll add them.

Summary of my questions:

1. What is the correct syntax to configure custom status messages that will display in GitHub?
2. What's wrong with my otherwise-valid config such that polling for remotes ignores PRs, and that opening a new PR doesn't trigger a build?
3. Is there another place I should be looking for documentation for these things? Or other resources that would help me learn what I'm doing?

Answer 1

Got it figured. There were several issues, but the crux of the problem was authentication: the various plugins and components accept and require different types of credentials. The setup I have working for me now uses a combination of personal access tokens and SSH keypairs to authenticate to GitHub.

Here's how to set up authentication:

1. Generate a new public-private keypair. I did this on my local machine, but you can do it anywhere. The private key file's contents will provide access to your GitHub account, so use appropriate caution when you decide where to keep the file, and clean up after yourself accordingly.
2. Go to GitHub and log in as the user you want Jenkins to use to authenticate to GitHub.
3. Navigate to Settings -> SSH and GPG keys. (Note: that's the user's settings, not a repo's settings)
4. Create a new SSH key. Give it a name (I named mine after the Jenkins instance) and paste in the contents of the public key generated in step 1.
5. Navigate to Settings -> Developer settings -> Personal access tokens
6. Generate a new token. Grant it access to "repo". Make sure you capture the token value somewhere safe -- I saved mine in my password manager. If you lose it, you'll have to go through all of these steps again to make a new one and configure Jenkins to use it.
   • My company uses a private GitHub org with SAML-based auth and SSO. If this is true for you as well, make sure you enable SSO on the token for the appropriate org(s).
7. In Jenkins, go to Manage Jenkins -> Manage Credentials.
8. Create a new "SSH Username with private key" credential in the System / Global domain. In the "Username" field, enter the GitHub username. For the private key, choose "Enter directly", choose "Add", and paste in the text contents of the private key file generated in step 1.
   • You'll give the credential an ID and a description as part of creating it. The ID is used in XML config files, and the description is used in the Jenkins configuration UI. I like to use the same value for both so that the value stored in the XML config files is the same as the value I see in the user interface. IDs can be uppercase and lowercase letters plus separator characters. For best readability, i-like-to-use-kebab-case.
9. Create a new "Secret Text" credential in the System / Global domain. In the "Secret" field, enter the token value GitHub generated in step 6. Again, give it a description and an ID.
10. In Manage Jenkins -> Configure System -> GitHub Pull Request Builder -> Credentials, choose the token-based credential you created in step 9.

Here's the Job DSL that worked for PRs, using the jenkins-ghprb plugin:

scm {
    git {
        remote {
            github('privateorg/myrepo', 'ssh')
            credentials('ssh-credential-id')
            refspec('+refs/pull/*:refs/remotes/origin/pr/*')
        }
        branch('${sha1}')
    }
}

triggers {
    githubPullRequest {
        useGitHubHooks()
        orgWhitelist('privateorg')
        allowMembersOfWhitelistedOrgsAsAdmin()
        extensions {
            commitStatus {
                context('Jenkins')
                completedStatus('SUCCESS', 'Build succeeded!')
                completedStatus('FAILURE', 'Build failed. ')
                completedStatus('ERROR', 'Build errored. This is probably a problem with Jenkins or related infrastructure and not an issue with your code changes.')
            }
        }
    }
}

Notes:

• Everyone in our private org is allowed to submit PRs that trigger builds automatically. Your situation might be different, in which case you'll want to configure a different whitelist (people whose PRs trigger builds automatically) and/or a different set of admins (people who can trigger builds for non-whitelisted contributors).

The webhook on the GitHub side is configured as follows:

Notes:

• Payload URL: https://your-jenkins-host/ghprbhook/
  • Note that the host URL must be publicly accessible. Mine isn't, but we have a public-facing proxy. I used the proxy's hostname here. I also had to configure the proxy hostname in Manage Jenkins -> Configure System -> GitHub Pull Request Builder -> Jenkins URL override.
• Content-type must be application/json.
• The secret used here is a random string I generated with my password manager. This is optional. If supplied, you need to enter the same secret in Manage Jenkins -> Configure System -> GitHub Pull Request Builder -> Shared secret.
• The webhook should trigger on pull requests and issue comments.
  • I've trimmed the screenshot to hide unimportant events.

The end result:

And this for pushes:

scm {
    git {
        remote {
            github('privateorg/myrepo', 'ssh')
            credentials('ssh-credential-id')
        }
        branch('refs/heads/*')
    }
}

triggers {
    githubPush()
}

Webhook:

Notes:

• Payload URL: https://your-jenkins-host/github-webhook/
  • Note that the host URL must be publicly accessible. Mine isn't, but we have a public-facing proxy. I used the proxy's hostname here.
• Content-type must be application/x-www-form-urlencoded.
• I did not configure a secret. If there is a way to configure one on the Jenkins side, I didn't find it.
• The webhook should trigger on pull requests and pushes.
  • I've trimmed the screenshot to hide unimportant events.
• This configuration results in a build for every push, regardless of branch. That's what I wanted; it might not be what you want. If you want something else, just change the branch specifier.

I was not able to have a single job that handled both PRs and pushes, due to differences in both branch and refspec params. I found some evidence that Git supports multiple refspecs, and was able to get that feature working with git on the CLI, but was not successful in my attempts to configure Jenkins to do the same. I had no in creating a branch specifier that worked for both. I might be able to set up a single, parameterized build, and then have mini-jobs that use these triggers and then call the parameterized build, but at this time I don't see that being worth adding yet another job. Tangentially, I also set up a third job that runs nightly against our main development branch. We'll be building out an extensive (long-running) test suite for this build, while keeping the PR and push builders fast.

As for where I should have been looking for docs: I googled and googled, and pieced this together through trial and error with hints and bits and pieces of config found in dozens of places. I have gotten a bit better at reading the Job DSL plugin's API docs, but that was not enough by itself. Also useful: for the push-triggered job, the GitHub Hook Log, available on the Jenkins job summary page. For the PR-triggered job, the Jenkins System Log, available from Manage Jenkins -> System Log.