AWS CLI create RDS with elasticbeanstalk create-environment

Question

How can I create an RDS instance with the create-environment or another subcommand of aws elasticbeanstalk? I've tried several combinations of parameters to no avail. Below is an example.

APP_NAME="randall-railsapp"
aws s3api create-bucket --bucket "$APP_NAME"
APP_VERSION="$(git describe --always)"
APP_FILE="deploy-$APP_NAME-$APP_VERSION.zip"
git archive -o "$APP_FILE" HEAD
aws s3 cp "$APP_FILE" "s3://$APP_NAME/$APP_FILE"

aws --region us-east-1 elasticbeanstalk create-application-version \
--auto-create-application \
--application-name "$APP_NAME" \
--version-label "$APP_VERSION" \
--source-bundle S3Bucket="$APP_NAME",S3Key="$APP_FILE"

aws --region us-east-1 elasticbeanstalk create-environment \
--application-name "$APP_NAME" \
--version-label "$APP_VERSION" \
--environment-name "$APP_NAME-env" \
--description "randall's rails app environment" \
--solution-stack-name "64bit Amazon Linux 2014.03 v1.0.0 running Ruby 2.1 (Puma)" \
--cname-prefix "$APP_NAME-test" \
--option-settings file://test.json

And the contents of test.json:

[
{
    "OptionName": "EC2KeyName",
    "Namespace": "aws:autoscaling:launchconfiguration",
    "Value": "a-key-is-here"
},
{
    "OptionName": "EnvironmentType",
    "Namespace": "aws:elasticbeanstalk:environment",
    "Value": "SingleInstance"
},
{
    "OptionName": "SECRET_KEY_BASE",
    "Namespace": "aws:elasticbeanstalk:application:environment",
    "Value": "HAHAHAHAHAHA"
},
{
    "OptionName": "DBPassword",
    "Namespace": "aws:rds:dbinstance",
    "Value": "hunter2"
},
{
    "OptionName": "DBUser",
    "Namespace": "aws:rds:dbinstance",
    "Value": "random"
},
{
    "OptionName": "DBEngineVersion",
    "Namespace": "aws:rds:dbinstance",
    "Value": "9.3"
},
{
    "OptionName": "DBEngine",
    "Namespace": "aws:rds:dbinstance",
    "Value": "postgres"
}
]

Anyone know why this is failing? Anything I specify with a aws:rds:dbinstance namespace seems to get removed from the configuration.

score 8 · Accepted Answer · edited Sep 22 '14 at 04:08

8

Just setting the aws:rds:dbinstance options does not create an RDS database. Currently you can create an RDS instance using one of the following techniques:

Create using AWS Console
Use eb cli
Use Resources section of ebextensions to create an RDS resource

The first two approaches are the most convenient as they do all the heavy lifting for you but for the third one you have to do some extra work. The third approach is what you would want to use if you are not using the console or eb CLI.

You can create an RDS resource for your beanstalk environment using the following ebextension snippet. Create a file called 01-rds.config in the .ebextensions directory of your app source.

Resources:
    AWSEBRDSDatabase:
        Type: AWS::RDS::DBInstance
        Properties:
            AllocatedStorage: 5
            DBInstanceClass: db.t2.micro
            DBName: myawesomeapp
            Engine: postgres
            EngineVersion: 9.3
            MasterUsername: myAwesomeUsername
            MasterUserPassword: myCrazyPassword

This file is in YAML format so indentation is important. You could also use JSON if you like. These are not option settings so you cannot pass it as --option-settings test.json. You just need to bundle this file with your app source.

Read more about what properties you can configure on your RDS database here. On this page you can also find what properties are required and what properties are optional.

Let me know if the above does not work for you or if you have any further questions.

edited Sep 22 '14 at 04:08

Randall Hunt

12,132
6
32
42

answered Sep 21 '14 at 20:58

Rohit Banga

18,458
31
113
191

So is the `01-rds.config` how the EB command line tools do it? – Randall Hunt Sep 22 '14 at 03:07
Also did you try this? When I try it my environments instantly terminate and I can't seem to get any logs from them. – Randall Hunt Sep 22 '14 at 03:49
SWEET I figured it out. The ecstatic joy I'm experiencing right now is fantastic. You guys rock. I've made a small edit to your answer correcting the parameter names for 01-rds.config I've figured out this is how the eb command line tools do it. – Randall Hunt Sep 22 '14 at 03:59
I'm now having issues getting the environment variables to go through. – Randall Hunt Sep 22 '14 at 04:24
What specific issues? You can see how CUSTOM_ENV is set here using a config file containing option settings. http://stackoverflow.com/a/14491294/161628 Since it is an option setting you can either specify it in `.ebextensions` or pass it through the command line option `--option-settings` – Rohit Banga Sep 22 '14 at 04:30
specifically I can't get the `RDS_PORT`, `RDS_HOSTNAME`, `RDS_PASSWORD`, etc. variables to be set. – Randall Hunt Sep 22 '14 at 04:39
I can't think why those other variables are not working for you but those variables have special significance for RDS created via console, eb cli and your RDS is created via an ebextension (although I cannot confirm that will cause a problem until I test it out). You might want to try a different custom variable name like `DB_PORT`, `DB_HOSTNAME` etc. That should unblock you. Also, protip - if a variable is set both via API (e.g. using --option-settings) and via ebextensions, value set with API takes precedence. – Rohit Banga Sep 22 '14 at 04:45
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/61648/discussion-between-ranman-and-rohit-banga). – Randall Hunt Sep 22 '14 at 04:47

score 3 · Answer 2 · answered Dec 12 '17 at 22:51

As of December 2017 we use the following ebextensions

$ cat .ebextensions/rds.config
Resources:
    AWSEBRDSDBSecurityGroup:
        Type: AWS::RDS::DBSecurityGroup
        Properties:
            EC2VpcId:
                Fn::GetOptionSetting:
                    OptionName: "VpcId"
            GroupDescription: RDS DB VPC Security Group
            DBSecurityGroupIngress:
                - EC2SecurityGroupId:
                    Ref: AWSEBSecurityGroup

    AWSEBRDSDBSubnetGroup:
        Type: AWS::RDS::DBSubnetGroup
        Properties:
            DBSubnetGroupDescription: RDS DB Subnet Group
            SubnetIds:
                Fn::Split:
                    - ","
                    - Fn::GetOptionSetting:
                        OptionName: DBSubnets

    AWSEBRDSDatabase:
        Type: AWS::RDS::DBInstance
        DeletionPolicy: Delete
        Properties:
            PubliclyAccessible: true
            MultiAZ: false
            Engine: mysql
            EngineVersion: 5.7
            BackupRetentionPeriod: 0
            DBName: test
            MasterUsername: toor
            MasterUserPassword: 123456789
            AllocatedStorage: 10
            DBInstanceClass: db.t2.micro
            DBSecurityGroups:
                - Ref: AWSEBRDSDBSecurityGroup
            DBSubnetGroupName:
                Ref: AWSEBRDSDBSubnetGroup

Outputs:
    RDSId:
        Description: "RDS instance identifier"
        Value:
            Ref: "AWSEBRDSDatabase"

    RDSEndpointAddress:
        Description: "RDS endpoint address"
        Value:
            Fn::GetAtt: ["AWSEBRDSDatabase", "Endpoint.Address"]

    RDSEndpointPort:
        Description: "RDS endpoint port"
        Value:
            Fn::GetAtt: ["AWSEBRDSDatabase", "Endpoint.Port"]

    AWSEBRDSDatabaseProperties:
        Description: Properties associated with the RDS database instance
        Value:
            Fn::Join:
                - ","
                - - Ref: AWSEBRDSDatabase
                  - Fn::GetAtt: ["AWSEBRDSDatabase", "Endpoint.Address"]
                  - Fn::GetAtt: ["AWSEBRDSDatabase", "Endpoint.Port"]

With such custom options

$ cat .ebextensions/custom-options.config
option_settings:
    "aws:elasticbeanstalk:customoption":
        DBSubnets: subnet-1234567,subnet-7654321
        VpcId: vpc-1234567

The only things - you have to explicitly pass RDS_* env variables to your application.

score 2 · Answer 3 · answered Sep 11 '15 at 22:12

The other answers did not work in my environment as of Sept 2015. After much trial and error, the following worked for me:

config template snippet (YAML):

  aws:rds:dbinstance:
    DBAllocatedStorage: '5'
    DBDeletionPolicy: Delete
    DBEngine: postgres
    DBEngineVersion: 9.3.9
    DBInstanceClass: db.t2.micro
    DBPassword: PASSWORD_HERE
    DBUser: USERNAME_HERE
    MultiAZDatabase: false

.ebextensions/rds.config file (JSON):

{
    "Parameters": {
    "AWSEBDBUser": {
        "NoEcho": "true",
        "Description": "The name of master user for the client DB Instance.",
        "Default": "ebroot",
        "Type": "String",
        "ConstraintDescription": "must begin with a letter and contain only alphanumeric characters"
    },
    "AWSEBDBPassword": {
        "NoEcho": "true",
        "Description": "The master password for the DB instance.",
        "Type": "String",
        "ConstraintDescription": "must contain only alphanumeric characters"
    },
    "AWSEBDBName": {
        "NoEcho": "true",
        "Description": "The DB Name of the RDS instance",
        "Default": "ebdb",
        "Type": "String",
        "ConstraintDescription": "must contain only alphanumeric characters"
    }
    },
    "Resources": {
    "AWSEBAutoScalingGroup": {
        "Metadata": {
        "AWS::ElasticBeanstalk::Ext": {
            "_ParameterTriggers": {
            "_TriggerConfigDeployment": {
                "CmpFn::Insert": {
                "values": [
                    {
                    "CmpFn::Ref": "Parameter.AWSEBDBUser"
                    },
                    {
                    "CmpFn::Ref": "Parameter.AWSEBDBPassword"
                    },
                    {
                    "CmpFn::Ref": "Parameter.AWSEBDBName"
                    }
                ]
                }
            }
            },
            "_ContainerConfigFileContent": {
            "plugins": {
                "rds": {
                "Description": "RDS Environment variables",
                "env": {
                    "RDS_USERNAME": {
                    "Ref": {
                        "CmpFn::Ref": "Parameter.AWSEBDBUser"
                    }
                    },
                    "RDS_PASSWORD": {
                    "Ref": {
                        "CmpFn::Ref": "Parameter.AWSEBDBPassword"
                    }
                    },
                    "RDS_DB_NAME": {
                    "Ref": {
                        "CmpFn::Ref": "Parameter.AWSEBDBName"
                    }
                    },
                    "RDS_HOSTNAME": {
                    "Fn::GetAtt": [
                        "AWSEBRDSDatabase",
                        "Endpoint.Address"
                    ]
                    },
                    "RDS_PORT": {
                    "Fn::GetAtt": [
                        "AWSEBRDSDatabase",
                        "Endpoint.Port"
                    ]
                    }
                }
                }
            }
            }
        }
        }
    },
    "AWSEBRDSDatabase": {
        "Type": "AWS::RDS::DBInstance",
        "DeletionPolicy": "Delete",
        "Properties": {
        "DBName": {
            "Ref": {
            "CmpFn::Ref": "Parameter.AWSEBDBName"
            }
        },
        "AllocatedStorage": "5",
        "DBInstanceClass": "db.t2.micro",
        "Engine": "postgres",
        "DBSecurityGroups": [
            {
            "Ref": "AWSEBRDSDBSecurityGroup"
            }
        ],
        "MasterUsername": {
            "Ref": {
            "CmpFn::Ref": "Parameter.AWSEBDBUser"
            }
        },
        "MasterUserPassword": {
            "Ref": {
            "CmpFn::Ref": "Parameter.AWSEBDBPassword"
            }
        },
        "MultiAZ": false
        }
    },
    "AWSEBRDSDBSecurityGroup": {
        "Type": "AWS::RDS::DBSecurityGroup",
        "Properties": {
        "DBSecurityGroupIngress": {
            "EC2SecurityGroupName": {
            "Ref": "AWSEBSecurityGroup"
            }
        },
        "GroupDescription": "Enable database access to Beanstalk application"
        }
    }
    }
}

Update that as of Jan 2016 the above approach no longer works. — Adam Bloomston, Jan 26 '16 at 21:31

score 1 · Answer 4 · answered Dec 04 '14 at 01:11

I had the same problem, couldn't get it to work via .ebextensions, and I don't like the EB CLI tool.

EB CLI uses an undocumented API feature, and a customized version of the botocore library ('eb_botocore') to make this work. :(

So I went ahead and forked botocore, and merged in the API data file used by eb_botocore: https://github.com/boto/botocore/pull/396

Then I ran 'python setup.py install' on both my modified botocore and aws-cli (both at master), and aws-cli now accepts a --template-specification option on the 'aws elasticbeanstalk create-environment' command. Hooray!

Example usage:

aws elasticbeanstalk create-environment\
  ...various options...\
  --option-settings file://option-settings.json
  --template-specification file://rds.us-west-2.json

where rds.us-west-2.json is:

{
  "TemplateSnippets": [{
    "SnippetName": "RdsExtensionEB",
    "Order": 10000,
    "SourceUrl":
"https://s3.amazonaws.com/elasticbeanstalk-env-resources-us-west-2/eb_snippets/rds/rds.json"
    }]
}

(it appears you must select a snippet specific to your EB region).

and option-settings.json contains RDS-related settings similar to ones listed in the question (DBEngine, DBInstanceClass, DBAllocatedStorage, DBPassword).

It works great. I hope the AWS CLI team allows us to use this feature in the official tool in the future. I'm guessing it's not a trivial change or they would have done it already, but it's a pretty major omission functionality-wise from the Elastic Beanstalk API and AWS CLI tool, so hopefully they take a crack at it.

Just an update that this answer isn't up to date as of 9/2015 the CLI is pretty different now. — Randall Hunt, Sep 11 '15 at 22:28

AWS CLI create RDS with elasticbeanstalk create-environment

4 Answers4

Linked