Docker Watchtower cannot pull new image from AWS ECR

Question

I've installed Watchtower, and followed the docs for setting up private registry auth with auth helpers. In debug mode, I see it logging a message that auth value was obtained, but then it fails to pull image with "no basic auth credentials." Inspecting auth value, it's just host name from my config, and no credential. I verified that on the host system (Raspbian) I'm able to pull new version using the same docker config, without having to do any custom auth, everything works out of the box, using the same binary.

Here's my docker config:

{
    "auths" : {
          "0000000000.dkr.ecr.us-east-1.amazonaws.com" : {}
    },
    "credHelpers": {
        "0000000000.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
    }
}

Here's my docker compose:

version: "3"
services:
  cavo:
    image: 0000000000.dkr.ecr.us-east-1.amazonaws.com/test:1
    ports:
      - "8080:80"
    restart: always
  watchtower:
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json
      - /usr/bin/docker-credential-ecr-login:/bin/docker-credential-ecr-login
    environment:
      - AWS_REGION=us-east-1
      - AWS_ACCESS_KEY_ID=AAAAAAAAAAAAA
      - AWS_SECRET_ACCESS_KEY=aaaaaaaaaaaaaaa
    command: --debug --interval 30
    restart: always

And when watchtower attempts to check for new image, here is the log file:

watchtower_1  | time="2019-12-25T22:49:34Z" level=debug msg="Pulling 0000000000.dkr.ecr.us-east-1.amazonaws.com/test:1 for /root_test_1"
watchtower_1  | time="2019-12-25T22:49:34Z" level=debug msg="Loaded auth credentials {    0000000000.dkr.ecr.us-east-1.amazonaws.com  } from /config.json"
watchtower_1  | time="2019-12-25T22:49:34Z" level=debug msg="Got auth value: eyJzZXJ2ZXJhZGRyZXNzIjoiMDAwMDAwMDAwMC5ka3IuZWNyLnVzLWVhc3QtMS5hbWF6b25hd3MuY29tIn0="
watchtower_1  | time="2019-12-25T22:49:34Z" level=debug msg="Got image name: 0000000000.dkr.ecr.us-east-1.amazonaws.com/test:1"
watchtower_1  | time="2019-12-25T22:49:35Z" level=debug msg="Error pulling image 0000000000.dkr.ecr.us-east-1.amazonaws.com/sump-pump-v2:1, Error response from daemon: Get https://0000000000.dkr.ecr.us-east-1.amazonaws.com/v2/test/manifests/1: no basic auth credentials"
watchtower_1  | time="2019-12-25T22:49:35Z" level=info msg="Unable to update container /root_test_1. Proceeding to next."
watchtower_1  | time="2019-12-25T22:49:35Z" level=debug msg="Error response from daemon: Get https://0000000000.dkr.ecr.us-east-1.amazonaws.com/v2/test/manifests/1: no basic auth credentials"

Unpacking the auth value, it just has the hostname. No repository credential.

I was trying to follow "Credential helpers" documentation, but I'm not sure I understand where the aforementioned Dockerfile belongs.

Any pointers in the right direction would be appreciated. Thanks!

I ran into the same issue, and had to add `{ "credsStore": "ecr-login" }` to the docker config in order to get it working. Removing it causes it to stop working for some reason I couldn't find — mati, Sep 02 '20 at 18:16

score 3 · Answer 1 · answered Apr 16 '20 at 20:27

Try the folowing:

Create a docker volume named helper

docker volume create helper
Build de image from de Dockerfile in the docs

docker build -t aws-ecr-dock-cred-helper .
Run the container

docker run  -d --rm --name aws-cred-helper --volume helper:/go/bin aws-ecr-dock-cred-helper

The container will start and mount helper volume into /go/bin where is the docker-credential-ecr-login and then stop. You can check the content of helper volume with

docker run --rm --it -v helper:/go/bin  alpine

then do ls /go/bin

you should see the docker-credential-ecr-login binary. I didn't use docker compose, but you have to mount the helper volume into the watchtower container on /go/bin and for some reason export your $PATH with /go/bin like in the docs:

environment:
      - HOME=/
      - PATH=$PATH:/go/bin

Docker Watchtower cannot pull new image from AWS ECR

1 Answers1