Pushing an image to ECR, getting "Retrying in ... seconds"

Question

I recently created a new repository in AWS ECR, and I'm attempting to push an image. I'm copy/pasting the directions provided via the "View push commands" button on the repository page. I'll copy those here for reference:

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 123456789.dkr.ecr.us-west-2.amazonaws.com

("Login succeeded")

docker build -t myorg/myapp .
docker tag myorg/myapp:latest 123456789.dkr.ecr.us-west-2.amazonaws.com/myorg/myapp:latest
docker push 123456789.dkr.ecr.us-west-2.amazonaws.com/myorg/myapp:latest

However, when I get to the docker push step, I see:

> docker push 123456789.dkr.ecr.us-west-2.amazonaws.com/myorg/myapp:latest
The push refers to repository [123456789.dkr.ecr.us-west-2.amazonaws.com/myorg/myapp]

a53c8ed5f326: Retrying in 1 second 
78e16537476e: Retrying in 1 second 
b7e38d172e62: Retrying in 1 second 
f1ff72b2b1ca: Retrying in 1 second 
33b67aceeff0: Retrying in 1 second 
c3a550784113: Waiting 
83fc4b4db427: Waiting 
e8ade0d39f19: Waiting 
487d5f9ec63f: Waiting 
b24e42eb9639: Waiting 
9262398ff7bf: Waiting 
804aae047b71: Waiting 
5d33f5d87bf5: Waiting 
4e38024e7e09: Waiting
EOF

I'm wondering if this has something to do with the permissions/policies associated with this repository. Right now there are no statements attached to this repository. Is that the missing part? If so, what would that statement look like? I've tried this, but it had no effect:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "AllowPutImage",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:root"
      },
      "Action": "ecr:PutImage"
    }
  ]
}

Bonus Points: I eventually want to use this in a CDK CodeBuildAction. I was getting the same error as above, so I check to see if I was getting the same result in my local terminal, which I am. So if the policy statement needs to be different for use in the CDK CodeBuildAction those details would be appreciated as well.

Thank you in advance for and advice.

score 41 · Answer 1 · answered May 24 '22 at 11:05

41

I had this issue when the repository didn't exist in ECR - I assumed that pushing would create it, but it didn't.

Creating it before pushing solved the problem.

answered May 24 '22 at 11:05

Duncan McGregor

17,665
12
64
118

score 40 · Accepted Answer · answered Jan 25 '22 at 07:02

I was having the same problem when trying to upload the image manually using the AWS and Docker CLI. I was able to fix it by going into ECR -> Repositories -> Permissions then adding a new policy statement with principal:* and the following actions:

"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetDownloadUrlForLayer",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart"

Be sure to add more restrictive principals. I was just trying to see if permissions were the problem in this case and sure enough they were.

Thanks, @badr it worked. The error is misleading and should provide the exact error instead of Retrying... — Kunal D., Jul 15 '22 at 05:23

Vishwas M.R · Answer 3 · 2022-06-14T17:25:06.560

The accepted answer works correctly in resolving the issue. However, as has been mentioned in the answer, allowing principal:* is risky and can get your ECR compromised.

Be sure to add specific principal(s) i.e. IAM Users/Roles such that only those Users/Roles will be allowed to execute the mentioned "Actions". Following JSON policy can be added in Amazon ECR >> Repositories >> Select Required Repository >> Permissions >> Edit policy JSON to get this resolved quickly:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<AccountNumber>:role/<RoleName>"
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:CompleteLayerUpload",
        "ecr:GetDownloadUrlForLayer",
        "ecr:InitiateLayerUpload",
        "ecr:PutImage",
        "ecr:UploadLayerPart"
      ]
    }
  ]
}

thanks for adding the full policy doc – Clintm Apr 21 '23 at 16:28 — Clintm, Apr 21 '23 at 16:28

score 7 · Answer 4 · answered Oct 05 '22 at 04:52

7

In my case, the repo was not created on ECR. Creating it fixed it.

answered Oct 05 '22 at 04:52

HelloThere

840
10
18

1

Also, you have to create the REPO as a PRIVATE REPOSITORY so the desired URI `[123456789.dkr.ecr.us-west-2.amazonaws.com/myorg/myapp]` is mapped accordingly – Abhiram Satputé Nov 07 '22 at 13:23

score 6 · Answer 5 · answered Aug 02 '22 at 19:16

6

The same message ("Retrying in ... seconds" in loop) may be seen when running "docker push" without first creating the corresponding repo in ECR ("myorg/myapp" in your example). Run:

aws ecr create-repository --repository-name myorg/myapp --region us-west-2

answered Aug 02 '22 at 19:16

Paul Robu

77
1
3

Ah, I deleted the repository thinking that I deleted the image so I can refresh it. Then trying to push did not work so I had to re-create the repository. Like in this answer – Jorj Sep 14 '22 at 04:55
1

Thank you. I managed. In my case multiple AWS account was messed up – Ruslan Novikov Feb 08 '23 at 12:58
1

Thanks for adding the actual command. Made it easy to fix. – Under4Mhz Jun 14 '23 at 00:24

e-e · Answer 6 · 2022-01-24T04:14:29.443

It turns out it was a missing/misconfigured policy. I was able to get it working within CodeBuild by adding a role with the AmazonEC2ContainerRegistryPowerUser managed policy:

new CodeBuildAction({
  actionName: "ApplicationBuildAction",
  input: this.applicationSourceOutput,
  outputs: [this.applicationBuildOutput],
  project: new PipelineProject(this, "ApplicationBuildProject", {
      vpc: this.codeBuildVpc,
      securityGroups: [this.codeBuildSecurityGroup],
      environment: {
        buildImage: LinuxBuildImage.STANDARD_5_0,
        privileged: true,
      },
      environmentVariables: {
        ECR_REPO_URI: {
          value: ECR_REPO_URI,
        },
        ECR_REPO_NAME: {
          value: ECR_REPO_NAME,
        },
        AWS_REGION: {
          value: this.region,
        }
      },
      buildSpec: BuildSpec.fromObject({
        version: "0.2",
        phases: {
          pre_build: {
            commands: [
              "echo 'Logging into Amazon ECR...'",
              "aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REPO_URI",
              "COMMIT_HASH=$(echo \"$CODEBUILD_RESOLVED_SOURCE_VERSION\" | head -c 8)"
            ]
          },
          build: {
            commands: [
              "docker build -t $ECR_REPO_NAME:latest ."
            ]
          },
          post_build: {
            commands: [
              "docker tag $ECR_REPO_NAME:latest $ECR_REPO_URI/$ECR_REPO_NAME:latest",
              "docker tag $ECR_REPO_NAME:latest $ECR_REPO_URI/$ECR_REPO_NAME:$COMMIT_HASH",
              "docker push $ECR_REPO_URI/$ECR_REPO_NAME:latest",
              "docker push $ECR_REPO_URI/$ECR_REPO_NAME:$COMMIT_HASH",
            ]
          }
        }
      }),
      // * * ADDED THIS ROLE HERE * *
      role: new Role(this, "application-build-project-role", {
        assumedBy: new ServicePrincipal("codebuild.amazonaws.com"),
        managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName("AmazonEC2ContainerRegistryPowerUser")]
      })
    }),
});

score 1 · Answer 7 · answered May 10 '22 at 04:03

1

The problem is your iam-user have not permission to full access of ecr so attach below policy to your iam-user.

follow photo for policy attachment

answered May 10 '22 at 04:03

Jay Harnesha

31
2

score 1 · Answer 8 · answered Sep 09 '22 at 08:51

1

For me, the problem was that the repository name on ECR had to be the same as the name of the app/repository I was pushing. Tried all fixes here, didn't work. This did!

answered Sep 09 '22 at 08:51

Sohaib Furqan

291
2
10

score 1 · Answer 9 · answered Nov 22 '22 at 07:11

Browse ECR -> Repositories -> Permissions

Edit JSON Policy.

Add these actions.

"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetDownloadUrlForLayer",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart"

And Add "*" in Resources.

Save it.

You're good to go, Now you can push the image to ECR.

score 1 · Answer 10 · answered Dec 06 '22 at 08:28

1

If you have MFA enforcement policy on your account that might be the problem because you have to have a token for getting action. Take a look at this AWS document to get a token on CLI.

answered Dec 06 '22 at 08:28

bkloppg

21
3

score 1 · Answer 11 · answered Apr 25 '23 at 16:25

In my case, it was causing this error due to the wrong syntax. This is the correct syntax that solved the issue in my case:

docker tag my-image:latest [AWS_ACCOUNT_ID].dkr.ecr.[REGION].amazonaws.com/my-ecr-repo:latest
docker push [AWS_ACCOUNT_ID].dkr.ecr.[REGION].amazonaws.com/my-ecr-repo:latest

I had the correct image name written in the command but did not write the right repository name. And because of that, it couldn't find the right repository which caused the failure in pushing the image.

score 1 · Answer 12 · answered May 21 '23 at 17:34

An approach you could take to concretely identify what's behind such error messages is to look at CloudTrail logs.

I was in the same boat, trying to deploy a CodeBuild action (via CDK) and hit this error when the CodePipeline pipeline ran. Looking at the CloudTrail logs, I noticed the InitiateLayerUpload event, and the source being ecr.amazonaws.com. The error code for this event was logged as RepositoryNotFoundException. Furthermore, the trail payload (given below) had all the details I needed.

{
...
    "awsRegion": "eu-central-1",
    "sourceIPAddress": "AWS Internal",
    "userAgent": "AWS Internal",
    "errorCode": "RepositoryNotFoundException",
    "errorMessage": "The repository with name '<masked>/<masked>' does not exist in the registry with id '<masked>'",
    "requestParameters": {
...
}

The CDK code snippet is also added below for your reference.

...
     /* Create an ECR repository */
    
     const ecr_repo = new ecr.Repository(this, 'ECRRepository', {
       repositoryName: "python-app-repository"
     });

     /* Define a CodeBuild project */

     const dockerImageBuildCodebuildProject = new codebuild.PipelineProject(this, 'DockerImageBuild',{
      buildSpec: codebuild.BuildSpec.fromSourceFilename('buildspecs/dockerimagebuild.yml'),
      environment: {
        buildImage: codebuild.LinuxBuildImage.STANDARD_2_0,
        privileged: true,
        environmentVariables: 
        {
          "ECR_REPO_URL": { value: ecr_repo.repositoryUri },
          "AWS_REGION": { value: process.env.CDK_DEFAULT_REGION },
        }
      },
    });

    /* Give CodeBuild permissions to login (into ECR) and push an image */

    const buildRolePolicy =  new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      resources: ['*'],
      actions: [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:PutImage"
            ]
    });

    /* Add role policy to the CodeBuild project */
    dockerImageBuildCodebuildProject.addToRolePolicy(buildRolePolicy);

    /* Create a CodeBuild action to add into the pipeline stage */

    const dockerImageBuildAction =  new CodeBuildAction({
      actionName: 'DockerImageBuild',
      input: sourceOutput,
      project: dockerImageBuildCodebuildProject
    })
...

score 0 · Answer 13 · answered Jul 14 '22 at 03:00

For anyone running into this issue, my problem was having the wrong AWS profile/account configured in my AWS cli.

run aws configure and add the keys of the account having access to ECR repository.

If you have multiple AWS accounts using the cli, then check out this solution.

score 0 · Answer 14 · answered Aug 30 '22 at 13:09

Just had this problem. It was permission related. In my case I was using CDKv2, which assumes a specific role in order to upload assets. Because the user I was deploying as did not have permission to assume that role, it failed. The hint was these warning messages that appeared during the deploy:

current credentials could not be used to assume 'arn:aws:iam::12345:role/cdk-abcde1234-image-publishing-role-12345-ap-southeast-2', but are for the right account. Proceeding anyway.
current credentials could not be used to assume 'arn:aws:iam::12345:role/cdk-abcde1234-file-publishing-role-12345-ap-southeast-2', but are for the right account. Proceeding anyway.

Yes, updating the permissions on your ECR repo would fix it, but since CDK is supposed to maintain this for you, the proper solution is to allow your user to assume the CDK role so you don't need to mess with ECR permissions yourself.

In my case I did this by granting the sts:AssumeRole permission for the resource arn:aws:iam::*:role/cdk-*. This allowed my user to assume both the file upload role and the image upload role.

After granting this permission, the CDK errors about being unable to assume the role went away, and I was able to deploy successfully.

score 0 · Answer 15 · answered Dec 07 '22 at 13:23

I was uploading from EC2 instance and I was missing to specify the region to my awscli, the login was successful but the docker push command was Retrying all the time, I have set the correct permissions on the ECR repo side

This line fix the issue for me and

aws configure set default.region us-west-1

score 0 · Answer 16 · answered Dec 21 '22 at 10:17

0

In my case I used wrong AWS credentials and aws configure with correct credentials resolved the issue.

answered Dec 21 '22 at 10:17

vishal soni

1
1

score 0 · Answer 17 · answered May 08 '23 at 09:12

For future Googlers,

Please double-check the access key and secret you're using.

In my case, I was using a credential that seemed like working (no error message, nothing when I was running aws cli commands). But when I finally checked it on AWS → IAM → Users → Me → Security Credentials → Access keys, it was freaking empty! So I created a new access key and used that. Problem solved.

I wonder why aws couldn't show something useful instead of ERROR: EOF ‍♂️

score 0 · Answer 18 · edited May 18 '23 at 08:53

0

For such cases, try to start debugging by using commands under the "view push commands" option on ECR Console w.r.t. the repository. This issue mostly occurs in the scenarios of hitting incorrect location references among push commands.

edited May 18 '23 at 08:53

starball

20,030
7
43
238

answered May 18 '23 at 08:46

Akshat Mukhriya

13
2

Pushing an image to ECR, getting "Retrying in ... seconds"

18 Answers18

Linked