Dynamically change branches on AWS CodePipeline

Question

I am looking for a good solution on how to run parametrized (customized) builds in CodePipeline where branch can be changed dynamically?

A little background on the problem: I need an on-demand environment that will be started on certain branch. We already use Bamboo CI server for part of the infrastructure and this is easily achievable with customized build also in Jenkins.

So basically I need a way to trigger a build with branch as a variable on CodePipeline in AWS.

Nimin Unnikrishnan · Accepted Answer · 2019-04-26T04:55:52.530

3

We can very well have dynamic branching support with the following approach.

We follow this approach in our organisation and it works very well.

One of the limitations in AWS code-pipeline is that we have to specify branch names while creating the pipeline. We can however overcome this issue using the architecture shown below.

flow diagram

Create a Lambda function which takes the GitHub web-hook data as input, using boto3 integrate it with AWS pipeline(pull the pipeline and update), have an API gateway to make the call to the Lambda function as a rest call and at last create a web-hook to the GitHub repository.

External links:

edited Apr 26 '19 at 04:55

answered Feb 27 '19 at 12:49

Nimin Unnikrishnan

353
2
9

I'm using this workflow, but currently I have Codepipelines for each release branch (we use different release branches for different accounts). The only way I see to make this work with one Codepipeline, is to change the Codepipeline configuration every time time it's run. I don't like this approach because hitting the release button in the GUI is going to have unpredictable results if you don't verify the branch in the config. In my case I'm just doing away with Codepipeline and triggering Codebuild directly with "source version" supplied by my lambda function that processes the hooks. – Andy Fraley Apr 24 '19 at 18:11
2

Hi, you can use the AWS lambda to update the branch details in the pipeline, the webhook will have the necessary branch details , you can use python boto3 library in AWS Lambda to update the branch details in the pipeline automatically whenever a new branch is created. – Nimin Unnikrishnan Apr 26 '19 at 04:54
Just use naming convention to identify a release branch, we use 'release/*' where * represent the release name, this helps in identifying only release branches from webhook response in AWS Lambda. – Nimin Unnikrishnan Apr 26 '19 at 05:02
Hey I like this idea. I'm wondering if you can update the pipeline's branch while another build is running. Any experience with that? Will that create race conditions? – Gambo Aug 27 '21 at 16:40
1

@Gambo It will cancel the existing execution if you update it – systemdebt Aug 22 '22 at 23:38
@NiminUnnikrishnan What happens when multiple pushes are made at about the same time? The code pipeline seems to supersede code build executions. Did you find a fix for it? – systemdebt Aug 22 '22 at 23:38

score 2 · Answer 2 · answered Feb 17 '18 at 00:03

2

Currently CodePipeline does not support branch based builds. Typically CodePipeline works best for running validations and automating the release of your "release" branch.

One option for pre merge validation is to use CodeBuild pull request support to validate pull requests then use CodePipeline to validate the merged code: https://aws.amazon.com/about-aws/whats-new/2017/09/aws-codebuild-now-supports-building-github-pull-requests/

answered Feb 17 '18 at 00:03

TimB

1,457
8
10

CodePipeline now supports specifying a branch for Github and CodeCommit sources. I'm not sure when the feature was added. – kaliatech May 10 '18 at 19:12
3

@kaliatech CodePipeline has always supported specifying a branch. The question is about CodePipeline running dynamically on any branch rather than a specific one. This isn't something CodePipeline currently supports. – TimB Aug 10 '18 at 15:48
@TimB I am already doing what you suggest but this is not ideal, its good because I have a full pipeline per repository in github that runs after a merge to master\main but it lacks on many fronts. I cannot have a pipeline that can dynamically build and destroy an ephemeral environment when PR is raised (for dev), and it is not easy to coordinate a deployment across multiple repos pipelines. I have opted for a strategy where I fix forwards and deploy to a test environment from master and try to organise my code so any one repo can be deployed on its own rather than dependent on other repos. – berimbolo Mar 25 '21 at 07:23

score 1 · Answer 3 · answered Apr 21 '19 at 08:47

For your use case it is best to create a pipeline specifically for each branch, as it sounds like your branch will have a fixed name for a given environment.

This works well where branches represent environments, where CodePipeline struggles is performing Continuous Delivery for more dynamic branches, such as feature branches/pull requests.

For the latter scenario I use CodeBuild to process pull requests, and then publish build artefacts in an S3 archive that I then use to trigger CodePipeline to run integration tests and staging deployments. There are a few traps along the way, but it allows you to leverage some of the more powerful features of CodePipeline (e.g. ability to only have a single stage execution running at a time, which is important for environments with shared resources).

score 1 · Answer 4 · answered Sep 18 '19 at 12:24

I've got the same requirement after doing some R&D got the answer, in the AWS codebuild section didn't have a conditioning base trigger, There is a trigger for scheduling cron base job which only runs after a minute.. So if need to run trigger or dynamically run the codebuild after checking your condition then you have to use aws sdk

I use aws cli and nodejs

For command (cli):

aws codebuild start-build --project-name project1

--environment-variables-override name=variablename,value=variablevalue

for node js

     var AWS = require("aws-sdk");
      var codebuild = new AWS.CodeBuild();

     var params = {
    projectName: machineName, /* required */
    environmentVariablesOverride: [
        {
          name: 'APPID', /* required */
          value: appid, /* required */
        },
         {
          name: 'BUILDID', /* required */
          value: buildidnew, /* required */
        },{
          name: 'REBUILD', /* required */
          value: rebuildid, /* required */
        }

      ],
    };

    codebuild.startBuild(params, function(err, data) {

      if (err) {
       callback(err, null);

      }
      else {

     callback(null, data);
      }  



    });

Remember the variable value should be a string

Dynamically change branches on AWS CodePipeline

4 Answers4

Linked