How to make Jenkins to build automatically after polling from bitbucket instead to manually build a job with specific parameters

Question

I have my Jenkinsfile working in place, which I can manually execute with two parameters that have to be specified depending if I'm building for dev, staging or prod.

What I would like to do now is that everytime Jenkins finds out if there's new content in the bitbucket repository, it builds a new job with specific parameters (build dev first with database update) and then if that is successful, make another build with the staging parameter on and also with the parameter of database build as well.

I would like the process to deploy (or rather, create a new job) for production to be a manual task, until I check that everything looks good in staging.

Just as a side note, my jenkins pipeline works as expected and polling works as expected as well, I just need to make this process automated.

NOTE

I'm not using a webhook from Bitbucket even I tried to do it, because my jenkins server is running on a host which is not allowed to have external connections and when bitbucket is sending the signal to my jenkins server it just times out.

Thank you for the suggestions in advance.

My Jenkinsfile is as follows:

// Deployment template for CMS-based websites (Drupal or Wordpress)
// 
//
pipeline {
agent any

parameters {
    choice(choices: "Dev\nStaging\nProduction", description: "Choose which environment to push changes to.", name: "DEPLOY_TO")
    choice choices: "No\nYes", description: "Choose whether to deploy the database as well.", name: "DEPLOY_DB"
}

environment {
    SITEID = "ge"
    NOFLAGS = "0"
    DBNAME = "wpress_website"
    DBSERVER = "dbserver"
    DBUSER = "geWordpress"
    DBPASS = "akjh23kas"
    EXCLUDE = "comp_commentmeta,comp_comments"  // separate multiple tables with commas
    DEPLOY_TO = "${params.DEPLOY_TO}"
    DEPLOY_DB = "${params.DEPLOY_DB}"
}

stages {
    stage("deploy-db-dev") {
        when {
            allOf { 
                environment ignoreCase: true, name: "DEPLOY_TO", value: "dev"; 
                environment ignoreCase: true, name: "DEPLOY_DB", value: "yes"; 
            }
        }
        steps {
            // this stage only required until we make our dev the master DB
            // copy full dev database from appserv1
            // import latest database dump to dev server
            script {
                FILENM = sh(script: 'ls -t goewp-s-dump* | head -1', returnStdout: true)
            }
            //Fixing the problem with the collation existing in the sql dump file, refer to: https://stackoverflow.com/questions/42385099/1273-unknown-collation-utf8mb4-unicode-520-ci 
            //apparently, this is due to a version of mysql issue. Once the problem is fixed from the server side we can then remove the following lines. 

            sh "sed -i s/utf8mb4_unicode_520_ci/utf8mb4_unicode_ci/g ${FILENM}"
            //The following line was added because the site is pointing to a staging server which we don't have control over, again, once this is fixed we can delete the following line of code. 
            sh "sed -i s/edit.staging.websites.3pth.com/stage.goewpfoods.hcgweb.net/g ${FILENM}"

            sh "mysql -h appserver -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_dev < ${WORKSPACE}/${FILENM}"
        }
    }
    stage("deploy-dev") {
        when {
            environment ignoreCase: true, name: "DEPLOY_TO", value: "dev"
        }
        steps {
            // copy files to appserv2
            // NOTE: if we move the repo to SVN, we should change httpdocs/ to ${env.SITEID}docs/
            sh "sudo chown jenkins:jenkins *"

            //Replace the wp-config.php file with our comp file with our information. 
            sh "/bin/cp httpdocs/wp-config-comp.php httpdocs/wp-config.php"

            // prepare the dev server to receive files by changing the owner
            sh "ssh webadmin@appserv2 \"sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs/\""
            // copy files from control server to dev
            sh "rsync --exclude=Jenkinsfile -rav -e ssh --delete ${WORKSPACE}/httpdocs/ webadmin@appserv2:/var/opt/httpd/${env.SITEID}docs/"
            // fix the owner/permissions on the dev server
            sh "ssh webadmin@appserv2 \"sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/\""
            sh "ssh webadmin@appserv2 \"sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/\""
            sh "ssh webadmin@appserv2 \"sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;\""
        }
    }
    stage("deploy-db-staging") {
        when {
            allOf { 
                environment ignoreCase: true, name: "DEPLOY_TO", value: "staging"; 
                environment ignoreCase: true, name: "DEPLOY_DB", value: "yes"; 
            }
        }
        steps {
            script {
                def myexcludes = env.EXCLUDE.split(',').toList()
                MYFLAGS = "-Q -K -c -e --default-character-set=utf8 "
                if (env.NOFLAGS == "0") {
                    myexcludes.each {
                        MYFLAGS = "${MYFLAGS} --ignore-table=${env.DBNAME}_dev.${it}"
                    }
                }
            }
            sh "cd ${WORKSPACE}"
            // pull a backup of the current dev database (may exclude some tables)
            sh "mysqldump -h appserv2 -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_dev ${MYFLAGS} > ${env.DBNAME}_dev.sql"
            // create a backup copy of the current staging database (full backup)
            sh "mysqldump -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_stage > ${env.DBNAME}_stage_bak.sql"
            // upload the dev database dump to the staging database
            sh "mysql -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_stage < ${WORKSPACE}/${env.DBNAME}_dev.sql"
        }
    }
    stage("deploy-staging") {
        when {
            environment ignoreCase: true, name: "DEPLOY_TO", value: "staging"
        }
        steps {
            // copy files from dev to control server
            sh "rsync --exclude=.svn --exclude=.git -rav -e ssh webadmin@appserv2:/var/opt/httpd/${env.SITEID}docs/ /tmp/${env.SITEID}docs/"

            //Replace the wp-config.php file with our comp file with our information. 
            sh "/bin/cp httpdocs/wp-config-comp.php httpdocs/wp-config.php"

            // prepare the staging server to receive files by changing the owner
            sh "ssh webadmin@stageserv \"sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs/\""
            // copy files from control server to staging
            sh "rsync --exclude=.svn --exclude=.git -rav -e ssh --delete /tmp/${env.SITEID}docs/ webadmin@stageserv:/var/opt/httpd/${env.SITEID}docs/"
            // fix the owner/permissions on the staging server
            sh "ssh webadmin@stageserv \"sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/\""
            sh "ssh webadmin@stageserv \"sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/\""
            sh "ssh webadmin@stageserv \"sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;\""

            // delete the temporary files on the control server
            sh "rm -Rf /tmp/${env.SITEID}docs/"
            // clear the caches
            sh "wget -O - \"http://www.web.net/incacache.php?api_key=yoiVbjgtL&site_id=088\""
        }
    }
    stage("deploy-db-production") {
        when {
            allOf { 
                environment ignoreCase: true, name: "DEPLOY_TO", value: "production"; 
                environment ignoreCase: true, name: "DEPLOY_DB", value: "yes"; 
            }
        }
        steps {
            script {
                def myexcludes = env.EXCLUDE.split(',').toList()
                MYFLAGS = "-Q -K -c -e --default-character-set=utf8 "
                if (env.NOFLAGS == "0") {
                    myexcludes.each {
                        MYFLAGS = "${MYFLAGS} --ignore-table=${env.DBNAME}_stage.${it}"
                    }
                }
            }
            sh "cd ${WORKSPACE}"
            // pull a backup of the current staging database (may exclude some tables)
            sh "mysqldump -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_stage ${MYFLAGS} > ${env.DBNAME}_stage.sql"
            // create a backup copy of the current production database (full backup)
            sh "mysqldump -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_prod > ${env.DBNAME}_prod_bak.sql"
            // upload the staging database dump to the production database
            sh "mysql -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_prod < ${WORKSPACE}/${env.DBNAME}_stage.sql"
        }
    }
    stage("deploy-production") {
        when {
            environment ignoreCase: true, name: "DEPLOY_TO", value: "production"
        }
        steps {
            // copy files from staging to control server
            sh "rsync --exclude=.svn --exclude=.git -rav -e ssh webadmin@stageserv:/var/opt/httpd/${env.SITEID}docs/ /tmp/${env.SITEID}docs/"

            // prepare the production server to receive files by changing the owner
            sh "ssh webadmin@appserv3 \"sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs/\""
            sh "ssh webadmin@appserv4 \"sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs/\""
            // copy files from control server to production
            sh "rsync --exclude=.svn --exclude=.git -rav -e ssh --delete /tmp/${env.SITEID}docs/ webadmin@appserv3:/var/opt/httpd/${env.SITEID}docs/"
            sh "rsync --exclude=.svn --exclude=.git -rav -e ssh --delete /tmp/${env.SITEID}docs/ webadmin@appserv4:/var/opt/httpd/${env.SITEID}docs/"
            // fix the owner/permissions on the production server
            sh "ssh webadmin@appserv3 \"sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/\""
            sh "ssh webadmin@appserv4 \"sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/\""
            sh "ssh webadmin@appserv3 \"sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/\""
            sh "ssh webadmin@appserv4 \"sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/\""
            sh "ssh webadmin@appserv3 \"sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;\""
            sh "ssh webadmin@appserv4 \"sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;\""

            // delete the temporary files on the control server
            sh "rm -Rf /tmp/${env.SITEID}docs/"
             // clear the caches
            sh "wget -O - \"http://www.web.net/incacache.php?api_key=yoiVbjgtL&site_id=088\""
        }
    }
}

score 2 · Answer 1 · edited Dec 12 '18 at 08:57

2

You can create function which can be considered as stages, And then call those in a sequence you wanted with certain parameters.

For example :

def deployDBDev(param1, param2){
  //some steps
}

def deployDev(diffParam1, diffParam2){
  //some steps
}

//then call it in the sequence you want

deployDBDev(param1, param2)
deployDev(diffParam1, diffParam2)

edited Dec 12 '18 at 08:57

Yash Jagdale

1,446
14
17

answered Dec 12 '18 at 06:24

Snehalkumar Mahale

186
4

Thank you for your answer, Don't I already have that sequence you are talking about in my Jenkinsfile? – VaTo Dec 12 '18 at 17:23
In your code you are using conditional stages, by wrapping stages into the function will allow you call the next step at the end of first step with different params. – Snehalkumar Mahale Dec 13 '18 at 04:36
I understand that, However, when Jenkins starts to build automatically after polling, how Jenkins will know which functions to call? Or how can I tell Jenkins which function/sequence to follow? – VaTo Jan 03 '19 at 16:40
and also, would I then need to midify the stages I have for the functions you are recommending? – VaTo Jan 03 '19 at 17:43

score 1 · Answer 2 · answered Dec 12 '18 at 08:43

I suggest to create 3 branches in bitbucket

dev
Stage and
production(or master).

At jenkins side you can create 3 jobs for each environment and add the polling on particular branch for auto trigger.

Your developers will commit in dev which will trigger the build for dev once you find that all things are good you can merge code in stage branch which will trigger stage job.

And finally when you confirms all good on staging just merge the staging to the production branch which will again trigger the production job.

score 0 · Answer 3 · answered Dec 12 '18 at 01:21

0

for triggering build when either code is pushed or Pull Request is created, you can follow below article.

https://support.cloudbees.com/hc/en-us/articles/115000051132-How-to-Trigger-Multibranch-Jobs-from-BitBucket-Cloud-

answered Dec 12 '18 at 01:21

Gaurang Shah

11,764
9
74
137

Thank you for your response, however I tried that approach already and I can't use that approach since my jenkins server is behind a firewall that doesn't allow any incoming connections or requests, so I can't use webhooks like the article suggests, and that's the reason I'm using polling every 30 minutes. But in any case, my question here is more about the Jenkinsfile configuration I need to add in order for me to build a job automatically with specific parameters and how to make sure it goes all the way to staging if dev built was successful only. – VaTo Dec 12 '18 at 06:01

How to make Jenkins to build automatically after polling from bitbucket instead to manually build a job with specific parameters

3 Answers3