ECS unable to pull secrets or registry auth, related to the api.ecr endpoint, Resourceinitializationerror

Question

I have tried everything here, here, here, and virtually every SO article from a Google of the error.

I have a private ECR image that I am trying to pull with an ECS service in a public subnet.

Error when trying to create an ECS service:

Resourceinitializationerror: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "https://api.ecr.us-west-2.amazonaws.com/": dial tcp: lookup api.ecr.us-west-2.amazonaws.com: i/o timeout

Task definition:

{
    "family": "chat-app-frontend",
    "containerDefinitions": [
        {
            "name": "frontend",
            "image": "576765093341.dkr.ecr.us-west-2.amazonaws.com/frontend:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "frontend-80-tcp",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/chat-app-frontend",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "ecs"
                }
            }
        }
    ],
    "taskRoleArn": "arn:aws:iam::576765093341:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::576765093341:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "3072",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
    "tags": [
        {
            "key": "ecs:taskDefinition:createdFrom",
            "value": "ecs-console-v2"
        }
    ]
}

ECS task execution Role.

ecs-extra-services-access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:CreateLogGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "ssm:GetParameters",
                "secretsmanager:GetSecretValue",
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds",
                "secretsmanager:ListSecrets"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-west-2:576765093341:secret:prod/ecr-private-registry",
                "arn:aws:kms:us-west-2:576765093341:key/807cbd08-a0ce-4948-b681-a49c7553003a"
            ]
        }
    ]
}

VPC Endpoints

These are attached to the public subnets.

Endpoint Policy for com.amazonaws.us-west-2.secretsmanager

{
    "Statement": [
        {
            "Sid": "AccessSpecificAccount",
            "Principal": {
                "AWS": "*"
            },
            "Action": "secretsmanager:*",
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Security Group

Applies to ECS task and all endpoints

Other configs

Task definition public IP create enabled
Subnets public IP create enabled
VPC DNS resolution and hostnames enabled
Created repositories docker images with the flag --endpoint-url https://api.ecr.us-west-2.amazonaws.com
Tried using ECR private registry permissions but they are "Not allowed" with no specified reason.

I also want to note that there are no logs being generated which makes me suspicious the entire taskExecutionRole (my extra permissions policy) is not being applied somehow.

You make a lot of mention about an ECR private registry, but the error is with fetching secrets from SecretsManager. The error message is unrelated to ECR. Also your secrets are named `secret:prod/ecr-private-registry` which is really strange. What are you storing in secrets manager and why is it named `ecr-private-registry`? You have your task definition configured to inject two secrets from SecretsManager as environment variables into the container. You are pulling both values from the same secret though, so the two values will be the same. — Mark B, Feb 01 '23 at 20:02
"I also want to note that there are no logs being generated which makes me suspicious the entire taskExecutionRole (my extra permissions policy) is not being applied somehow.:" Logs won't show up until the container is started, and ECS hasn't gotten to the point of starting the container yet. — Mark B, Feb 01 '23 at 20:03
@MarkB It looks like I was following the guide for ECR private registry permissions without explicitly enabling that option. I dont need secrets at all. I removed the secrets from the task def and its now failing to fetch the repo from `ecr.api`. I have tried everything [here](https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html) for fargate 1.4.0 and still getting the error — clinzy25, Feb 01 '23 at 20:29
Can you please edit your question to show the updated code, and the updated error? Specifically, need to know if it is a permissions error, or a network connection error. Also, since this is in a public subnet with public IPs enabled, have you tried simply removing all those VPC Endpoints? — Mark B, Feb 01 '23 at 20:31
Do you have any outbound restrictions on the security group attached to the ECS task? Or is it wide open? Also, you don't have any Network ACL rules on the subnet do you? — Mark B, Feb 01 '23 at 20:52
Security group is wide open outbound and I haven't made any changes to the ACL — clinzy25, Feb 01 '23 at 20:58
It honestly doesn't sound like the subnet is really public. Does it actually have a route to an Internet Gateway? — Mark B, Feb 01 '23 at 20:59
The subnet has a route 0.0.0.0/0 to the internet gateway. Im not sure how else to verify the subnet is public — clinzy25, Feb 01 '23 at 21:03
Yes that means it is public. If you removed all the VPC Endpoints and the routes to them, and you are still getting a timeout, I'm out of ideas. I would start with a new VPC. — Mark B, Feb 01 '23 at 21:14
I don't see public IP configuration in your task definition. — Azize, Feb 02 '23 at 01:25

score 0 · Answer 1 · answered Feb 02 '23 at 03:35

0

I launched the task in the default VPC with zero config (default security group) and it worked. Something wrong with my VPC config.

answered Feb 02 '23 at 03:35

clinzy25

91
8

score 0 · Answer 2 · answered Feb 02 '23 at 20:11

0

I want to add that I was able to create a custom VPC and get it working but it would always fail if I created the VPC with this module.

https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest

answered Feb 02 '23 at 20:11

clinzy25

91
8

Ayodele · Answer 3 · 2023-02-26T17:03:34.670

0

I had a similar issue and found out that ECS requires that your service has a public_ip assigned if it's in a public subnet.

If you don’t want your service to have a public IP, you can create your service in a private subnet instead

edited Feb 26 '23 at 17:03

answered Feb 17 '23 at 11:35

Ayodele

1
1

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Feb 25 '23 at 16:05