8

I've created a worker environment for my eb application in order to take advantage of its "periodic tasks" capabilities using cron.yaml (located in the root of my application). It's a simple sinatra app (for now) that I would like to use to use to issue requests to my corresponding web server environment.

However, I'm having trouble deploying via the eb cli. Below is what happens I run eb deploy.

╰─➤  eb deploy
Creating application version archive "4882".
Uploading myapp/4882.zip to S3. This may take a while.
Upload Complete.
INFO: Environment update is starting.
ERROR: Service:AmazonCloudFormation, Message:Stack named 'awseb-e-1a2b3c4d5e-stack'
aborted operation. Current state: 'UPDATE_ROLLBACK_IN_PROGRESS'
Reason: The following resource(s) failed to create: [AWSEBWorkerCronLeaderRegistry].

I've looked around the CloudFormation dashboard to see to check for possible errors. After reading a bit of about what I could find regarding AWSEBWorkerCronLeaderRegistry, I found it that it's most likely a DynamoDB table that gets updated/created. However, when I look in the DynamoDB dashboard, there are no tables listed.

As always, any help, feedback, or guidance is appreciated.

Kurt Mueller
  • 3,173
  • 2
  • 29
  • 50
  • Does the instance profile have dynamodb permissions? – Nick Humrich May 10 '15 at 05:30
  • I created the environment via EB web console. I assumed (maybe wrongly?) that the install process would create the permissions necessary to access dynamodb. How do I go about granting access to the envrionment's security group/role? – Kurt Mueller May 10 '15 at 17:15
  • 2
    http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.iam.roles.apps.html – Nick Humrich May 10 '15 at 17:35

4 Answers4

3

If you are reluctant to add full DynamoDB access (like I was), Beanstalk now provides a Managed Policy for Worker environment permissions (AWSElasticBeanstalkWorkerTier). You can try adding one of these to your instance profile role instead.

See http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/iam-instanceprofile.html

bryantrobbins
  • 41
  • 7
1

We had the same issue and fixed it by attaching AmazonDynamoDBFullAccess to Elastic Beanstalk role (which was named aws-elasticbeanstalk-ec2-role in our case).

ali gurbuz
  • 190
  • 1
  • 3
  • 14
1

I was using Codepipeline to deploy my worker and was getting the same error. Eventually I tried giving AWS-CodePipeline-Service the AmazonDynamoDBFullAccess policy and that seemed to resolve the issue.

Anthony Manning-Franklin
  • 4,408
  • 1
  • 18
  • 23
1

As Anthony suggested, when triggering the deploy from other services such as CodePileline, its service role needs the dynamodb:CreateTable permission to create the Leader Registry table (more info below) in DynamoDB.

Adding Full Access permission is a bad practice and should be avoided. Also, the managed policy AWSElasticBeanstalkWorkerTier does not have the appropriate permissions since it is for the worker to access DynamoDB and check if they are the current leader.

1. Find the Role that is trying to create the table:

  • Go to CloudTrail > Event History
  • Filter Event Name: CreateTable
  • Make sure the error code is AccessDenied
  • Locate the role name (i.e. AWSCodePipelineServiceRole-us-east-1-dev):

Cloud Trail Screenshot CloudTrail Error Details

2. Add the permissions:

  • Go to IAM > Roles
  • Find the role in the list
  • Attach a policy with: 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CreateCronLeaderTable",
            "Effect": "Allow",
            "Action": "dynamodb:CreateTable",
            "Resource": "arn:aws:dynamodb:*:*:table/*-stack-AWSEBWorkerCronLeaderRegistry*"
        }
    ]
}

3. Check results:

  1. Redeploy by triggering the pipeline
  2. Check Elasticbeanstalk for errors
  3. Optionally go to CloudTrail and make sure the request succeded this time.

You may use this technique any time you are not sure of what permission should be attached to what.

About the Cron Leader Table

From the Periodic Tasks Documentation:

Elastic Beanstalk uses leader election to determine which instance in your worker environment queues the periodic task. Each instance attempts to become leader by writing to an Amazon DynamoDB table. The first instance that succeeds is the leader, and must continue to write to the table to maintain leader status. If the leader goes out of service, another instance quickly takes its place.

For those wondering, this DynamoDB table uses 10 RCU and 5 WCU which covered by the always free tier.

Arian Acosta
  • 6,491
  • 1
  • 35
  • 32