Deploy from multiple sources with CodePipeline

Question

I am trying to set up a project on AWS. I am using CodePipeline to deploy my code to Elastic Beanstalk, and the source is coming from a git repository. This works fine.

The project has some configuration files (passwords and settings and such) that I don't want to include in the git repository. Since they are not in the git repository, they are not deployed by CodePipeline.

How can I include the configuration files in the CodePipeline without including them in the git repository?

Idea: I have tried adding an extra S3 source in the CodePipeline, containing the configuration files. I then had to add an extra deployment action to deploy the new S3 source. But then the two deployment processes get in conflict with each other, and only one of them succeeds. If I retry the one that fails, whatever was deployed by the one that succeeded is removed again. It doesn't seem to be possible to add two input artifacts (sources) to a single deployment action.

score 1 · Accepted Answer · answered Jul 23 '17 at 08:47

It is possible to use .ebextensions to copy files from an S3 bucket or another source during deployment. Amazon describes it well in their documentation.

Here is an example:

Resources:
  AWSEBAutoScalingGroup:
    Metadata:
      AWS::CloudFormation::Authentication:
        S3Auth:
          type: "s3"
          buckets: ["elasticbeanstalk-us-west-2-123456789012"]
          roleName:
            "Fn::GetOptionSetting":
              Namespace: "aws:autoscaling:launchconfiguration"
              OptionName: "IamInstanceProfile"
              DefaultValue: "aws-elasticbeanstalk-ec2-role"

files:
  "/tmp/data.json" :
    mode: "000755"
    owner: root
    group: root
    authentication: "S3Auth"
    source: https://s3-us-west-2.amazonaws.com/elasticbeanstalk-us-west-2-123456789012/data.json

are we supposed to keep the file in S3 publicly available in order for this to work? (like accessible directly from the url to anyone) ? — saibbyweb, Jan 23 '21 at 09:33

score 0 · Answer 2 · answered Jul 13 '17 at 17:04

Rather than storing the configuration files in a repository I'd recommend using the Software Configuration feature that Elastic Beanstalk has.

Here's a related answer explaining how to do that: https://stackoverflow.com/a/17878600/7433105

If you want to model your config as a separate source action then you would either have to have a build step that merges the source artifacts into one deployable artifact, or have some independent deployment process for the configuration that won't interfere with your application deployment (eg. copy to S3 in a Lambda function, then pull down the configuration when your application starts).

Deploy from multiple sources with CodePipeline

2 Answers2