I have around 30 Wordpress websites so the way Jenkins is configured is I have a Job for every website. The process of development is as follows (I don't know if it's optimal but this is how we have it):
- Since we have outsourcing developers, they have their own repository hosted in their own repository hosting provider. Whenever the code is ready for QA, they commit all their changes to our repository in the master branch.
- I then run the Jenkins job manually with the jenkinsfile like the one below.
- The workflow must be: deploy to dev environment, if and only if the previous deployment is successful, deploy to stage. Here we have to stop and let the QA people review the website for any broken links, errors, etc.
- If everything looks like we expected and no errors have been found, then deploy finally to production environment.
NOTE: Some people have suggested me to have only staging and production. The reason why we don't have that configuration is because the dev environment is not accessible online, and the reason for that is because I use this environment to test back-end configuration (e.g. apache conf, etc.).
Also, some other people have suggested to have a branch for each environment, which in theory makes sense but I think it will change the way that our outsourcing devs are committing code to the repository, I mean, they will always have to commit the code to the dev branch and then merge to the stage branch to get deployed to stage, which I don't think is pretty good.
Now, the steps 2-4 look like the following: In order to give you an example on how that process looks we are going to have an example website and job called "Bearitos":
Inside of that Job called "Bearitos" there is a project called "Bearitos to any"
which basically means inside of that project I have a pipeline configured with three stages: dev, staging and prod which are parameterized with the following parameters: DEPLOY_TO: Dev/staging/prod and DEPLOY_DB: Yes/No . So depending of what the user chooses, Jenkins will deploy to that specific environment which I don't think it's even necessary to have those options since the correct deployment flow should be dev -> staging -> prod, there shouldn't be a scenario where dev or staging would be skipped and then deploy right next to production, so in my opinion this should be updated better
Inside of the Jenkinsfile I have defined the three stages Dev, Staging or Prod and also the options if it was chosen to build a DB or not, following is the example of how my Jenkinsfile looks like:
// Deployment template for CMS-based websites (Drupal or Wordpress)
//
//
pipeline {
agent any
parameters {
choice choices: ['Dev', 'Staging', 'Production'], description: "Choose which environment to push changes to.", name: "DEPLOY_TO"
booleanParam defaultValue: true, "Choose whether to deploy the database.", name: "DEPLOY_DB"
}
environment {
SITEID = "lb"
NOFLAGS = "0"
DBNAME = "wpress_myproject"
DBSERVER = "dbserver"
DBUSER = "WordpressUser"
DBPASS = "hiddenpassword"
EXCLUDE = "domain_commentmeta,domain_comments" // separate multiple tables with commas
DEPLOY_TO = "${params.DEPLOY_TO}"
DEPLOY_DB = "${params.DEPLOY_DB}"
}
stages {
stage("deploy-db-dev") {
when {
allOf {
environment ignoreCase: true, name: "DEPLOY_TO", value: "dev";
environment ignoreCase: true, name: "DEPLOY_DB", value: "true";
}
}
steps {
// this stage only required until we make our dev the master DB
// copy full dev database from bolwebdev1
// import latest database dump to dev server
script {
FILENM = sh(script: 'ls -t myproject-s-dump* | head -1', returnStdout: true)
}
//Fixing the problem with the collation existing in the sql dump file, refer to: https://stackoverflow.com/questions/42385099/1273-unknown-collation-utf8mb4-unicode-520-ci
//apparently, this is due to a version of mysql issue. Once the problem is fixed from the server side we can then remove the following lines.
sh """sed -i s/utf8mb4_unicode_520_ci/utf8mb4_unicode_ci/g ${FILENM}
# The following line was added because the site is pointing to a staging server which we don't have control over, again, once this is fixed we can delete the following line of code.
sed -i s/myproject.staging.websites.3pth.com/myproject.example.net/g ${FILENM}
mysql -h devserver2 -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_dev < ${WORKSPACE}/${FILENM}
rm -f ${WORKSPACE}/${FILENM}"""
}
}
stage("deploy-dev") {
when {
environment ignoreCase: true, name: "DEPLOY_TO", value: "dev"
}
steps {
// copy files to devserver2
// NOTE: if we move the repo to SVN, we should change httpdocs/ to ${env.SITEID}docs/
sh """sudo chown jenkins:jenkins *
#Replace the wp-config.php file with our domain file with our information.
/bin/cp httpdocs/wp-config-domain.php httpdocs/wp-config.php
# prepare the dev server to receive files by changing the owner
ssh webadmin@devserver2 'sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs/'
# copy files from control server to dev
rsync --exclude=Jenkinsfile -rav -e ssh --delete ${WORKSPACE}/httpdocs/ webadmin@devserver2:/var/opt/httpd/${env.SITEID}docs/
# fix the owner/permissions on the dev server
ssh webadmin@devserver2 'sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/ && sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/ && sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;'"""
}
}
stage("deploy-db-staging") {
when {
allOf {
environment ignoreCase: true, name: "DEPLOY_TO", value: "staging";
environment ignoreCase: true, name: "DEPLOY_DB", value: "true";
}
}
steps {
script {
def myexcludes = env.EXCLUDE.split(',').toList()
MYFLAGS = "-Q -K -c -e --default-character-set=utf8 "
if (env.NOFLAGS == "0") {
myexcludes.each {
MYFLAGS = "${MYFLAGS} --ignore-table=${env.DBNAME}_dev.${it}"
}
}
}
// pull a backup of the current dev database (may exclude some tables)
sh """mysqldump -h devserver2 -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_dev ${MYFLAGS} > ${env.DBNAME}_dev.sql
#Searching and replace for the URL to change from the dev sever to the staging server
sed -i s/myproject.example.net/stage-myproject.example.net/g ${env.DBNAME}_dev.sql
# create a backup copy of the current staging database (full backup)
mysqldump -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_stage > ${env.DBNAME}_stage_bak.sql
# upload the dev database dump to the staging database
mysql -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_stage < ${WORKSPACE}/${env.DBNAME}_dev.sql
rm -f ${WORKSPACE}/${env.DBNAME}_dev.sql"""
}
}
stage("deploy-staging") {
when {
environment ignoreCase: true, name: "DEPLOY_TO", value: "staging"
}
steps {
// copy files from dev to control server
sh """rsync --exclude=.svn --exclude=.git -rav -e ssh webadmin@devserver2:/var/opt/httpd/${env.SITEID}docs/ /tmp/${env.SITEID}docs/
#Replace the wp-config.php file with our domain file with our information.
/bin/cp httpdocs/wp-config-domain.php httpdocs/wp-config.php
#prepare the staging server to receive files by changing the owner
ssh webadmin@stageserver 'sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs/'
# copy files from control server to staging
rsync --exclude=.svn --exclude=.git -rav -e ssh --delete /tmp/${env.SITEID}docs/ webadmin@stageserver:/var/opt/httpd/${env.SITEID}docs/
# fix the owner/permissions on the staging server
ssh webadmin@stageserver 'sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/ && sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/ && sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;'
#delete the temporary files on the control server
rm -Rf /tmp/${env.SITEID}docs/
# clear the Incapsula caches
if [[ \$( curl -sS -X POST \"http://www.example.net/incapcache.php?api_key=asdaswwGR)feasdsdda&site_id=stage&resource_url=stage-myproject.example.net\" | jq -r .debug_info.id_info) != \"incapsula cache cleared successfuly\" ]]; then exit 255; fi"""
}
}
stage("deploy-db-production") {
when {
allOf {
environment ignoreCase: true, name: "DEPLOY_TO", value: "production";
environment ignoreCase: true, name: "DEPLOY_DB", value: "true";
}
}
steps {
script {
def myexcludes = env.EXCLUDE.split(',').toList()
MYFLAGS = "-Q -K -c -e --default-character-set=utf8 "
if (env.NOFLAGS == "0") {
myexcludes.each {
MYFLAGS = "${MYFLAGS} --ignore-table=${env.DBNAME}_stage.${it}"
}
}
}
sh """cd ${WORKSPACE}
# pull a backup of the current staging database (may exclude some tables)
mysqldump -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_stage ${MYFLAGS} > ${env.DBNAME}_stage.sql
#Searching and replace for the URL to change from the stage sever to the prod server
sed -i s/stage-myproject.example.net/www.myproject.com/g ${env.DBNAME}_stage.sql
# create a backup copy of the current production database (full backup)
mysqldump -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_prod > ${env.DBNAME}_prod_bak.sql
# upload the staging database dump to the production database
mysql -h ${env.DBSERVER} -u ${env.DBUSER} --password='${env.DBPASS}' ${env.DBNAME}_prod < ${WORKSPACE}/${env.DBNAME}_stage.sql
rm -f ${WORKSPACE}/${env.DBNAME}_stage.sql"""
}
}
stage("deploy-production") {
when {
environment ignoreCase: true, name: "DEPLOY_TO", value: "production"
}
steps {
// copy files from staging to control server
sh """rsync --exclude=.svn --exclude=.git -rav -e ssh webadmin@stageserver:/var/opt/httpd/${env.SITEID}docs/ /tmp/${env.SITEID}docs/
# prepare the production server to receive files by changing the owner
ssh webadmin@prodserver1 'sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs'
ssh webadmin@prodserver2 'sudo chown -R webadmin:webadmin /var/opt/httpd/${env.SITEID}docs'
# copy files from control server to production
rsync --exclude=.svn --exclude=.git -rav -e ssh --delete /tmp/${env.SITEID}docs/ webadmin@prodserver1:/var/opt/httpd/${env.SITEID}docs/
rsync --exclude=.svn --exclude=.git -rav -e ssh --delete /tmp/${env.SITEID}docs/ webadmin@prodserver2:/var/opt/httpd/${env.SITEID}docs/
# fix the owner/permissions on the production server
ssh webadmin@prodserver1 'sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/'
ssh webadmin@prodserver2 'sudo chown -R apache:${env.SITEID}-web /var/opt/httpd/${env.SITEID}docs/'
ssh webadmin@prodserver1 'sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/'
ssh webadmin@prodserver2 'sudo chmod -R g+w /var/opt/httpd/${env.SITEID}docs/'
ssh webadmin@prodserver1 'sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;'
ssh webadmin@prodserver2 'sudo find /var/opt/httpd/${env.SITEID}docs/ -type d -exec chmod g+s {} \\;'
# delete the temporary files on the control server
rm -Rf /tmp/${env.SITEID}docs/
# clear the Incapsula caches
if [[ \$( curl -sS -X POST \"http://www.example.net/incapcache.php?api_key=asdaswwGR)feasdsdda&site_id=088&resource_url=www.myproject.com\" | jq -r .debug_info.id_info) != \"incapsula cache cleared successfuly\" ]]; then exit 255; fi"""
}
}
}
}
The problems that I'm currently facing with this approach are:
I can't figure out how make deployments automated since it's a parameterized pipeline so I'm not sure how to make it automated. The desired process would be to make the deployment automated once Jenkins polling every X amount of minutes on the git repository, deploy to Dev > Stage (only if Dev deployment was successful) automatically and then stop there until we manually deploy to Prod after we do QA on Staging.
The current Git configuration has configured only one branch (master) which is where the developers push the changes once they want to make a deployment to Dev -> Stage -> Prod. But I think the ideal scenario would have a dev branch for the dev deployments then stage branch for deploying to Stage environment and then master for deployment once we merge those dev and staging branches to the master branch. I'm not sure if this would be optimal, so I would appreciate any suggestions or ideas on this.
The desired approach will be to have mentioned problems resolved and also have an automated way to deploy and notify once the dev -> staging deployment was successful. As well as having an option to do the mentioned workflow manually, like we are doing right now (this is not that important, but would be a nice to have feature).
Thank you in advance for your help!
