8

I have created an ECS Fargate Task, which I can manually run. It updates a Dynomodb and I get logs.

Now I want this to run on a schedule. I have setup a scheduled ECS task through EventBridge. However, this does not run.

My looking at the EventBridge logs I can see that the container has been stopped for the following stopped reason:

ResourceInitializationError: unable to pull secrets or registry auth: execution resource 
retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3
time(s): RequestError: send request failed caused by: Post https://api.ecr....

I thought this might be a problem with permissions. However, I tested giving the Task Execution Role full power user permissions and I still get the same error. Could the problem be something else?

Mhairi McNeill
  • 1,951
  • 11
  • 20
  • 1
    Have you configured a role for the Eventbridge rule with enough permissions? See https://stackoverflow.com/questions/71625037/amazon-ecs-how-to-schedule-a-container/71630839#71630839 – Paolo Mar 29 '22 at 09:22
  • I'll check out cloudtrail, thanks! – Mhairi McNeill Mar 29 '22 at 12:11
  • Nothing in CloudTrail! Only one RunTask, which seems fine. None of the other events have errors either and they all look correct (as far as I understand anyway). – Mhairi McNeill Mar 29 '22 at 16:29
  • Does your ECS task/service have access to the intrnet? Maybe it can't connect to ECR endpoint at all, due being in private subnets without NAT, for example. – Marcin Apr 01 '22 at 07:23

3 Answers3

7

This is due to a connectivity issue.

The docs say the following:

For tasks on Fargate, in order for the task to pull the container image it must either use a public subnet and be assigned a public IP address or a private subnet that has a route to the internet or a NAT gateway that can route requests to the internet.

So you need to make sure your task has a route to an internet gateway (i.e. it's in a Public subnet) or a NAT gateway.

Alternatively, if your service is in an isolated subnet, you need to create VPC endpoints for ECR and other services you need to call, as described in the docs:

To allow your tasks to pull private images from Amazon ECR, you must create the interface VPC endpoints for Amazon ECR.

When you create a scheduled task, you also specify the networking options. The docs mention this step:

(Optional) Expand Configure network configuration to specify a network configuration. This is required for tasks hosted on Fargate and for tasks using the awsvpc network mode. For Subnets, specify one or more subnet IDs. For Security groups, specify one or more security group IDs. For Auto-assign public IP, specify whether to assign a public IP address from your subnet to the task.

So the networking configuration changed between the manually run task and the scheduled task. Refer to the above to figure out the needed settings for your case.

gshpychka
  • 8,523
  • 1
  • 11
  • 31
  • She mentioned that she can run the task manually though – Paolo Apr 01 '22 at 14:18
  • Right, but you also specify networking options when creating a scheduled task. I will update the answer to reflect this. – gshpychka Apr 01 '22 at 14:57
  • yeah, good point – Paolo Apr 01 '22 at 17:16
  • From my reading of the documentation my networking is set up correctly. I only have the default VPC, with its default subnets. When I configure the scheduled task I've added the three subnets. The security group was automatically configured. Looking at those, they allow all outbound traffic. For auto-assign public IP the only option is 'disabled' whether editing an existing rule or creating a new one. – Mhairi McNeill Apr 04 '22 at 13:40
  • It's not about security groups. You have to enable public IP when creating the scheduled task. Your networking is not set up correctly, since your task is not getting a public IP. – gshpychka Apr 04 '22 at 14:32
  • So, it was a networking error, thank you @gshpychka, but not something covered in the documentation - I think it may be a bug. I'll explain in my answer. – Mhairi McNeill Apr 05 '22 at 10:36
2

I fixed this by enabling auto-assign public IP.

However, to do this, I had to first change from "Capacity provider strategy" - "Use cluster default", to "Launch type" - "FARGATE". Then the option to enable auto-assign public IP became available in the dropdown in the EventBridge UI.

This seems odd to me, because my default capacity provider strategy for my cluster is Fargate. But it is working now.

Mhairi McNeill
  • 1,951
  • 11
  • 20
-1

Need to use a gateway to follow the traffic from ECS to ECR. It can either Internet Gateway or NAT Gateway eventually which would be effecting cost factor.

But where we can resolve this scenario, by creating VPC Endpoints. Which maintains the traffic within the AWS Resources.

Endpoints Required for this would be : S3 Gateway ECR ECS

Pavan
  • 1
  • 1
    As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Aug 27 '22 at 15:46