Can't create local kubernetes secret from docker for AWS ECR when using Okta SSO

Question

I did a very simple Kubernetes walkthrough on Windows using Docker Desktop running the local Kubernetes cluster and was successful in a few minutes.

I am now trying to do the same thing, except using my own image from AWS ECR. I cannot get Kubernetes authorization to access my private repo.

I have an AWS ECR repository created. I use Okta SSO, have AWS CLI installed, and can execute commands fine. I pushed my image to my private repo and can view it on the CLI. I have verified docker can get my private image and run the container: > docker run MY_ECR_ADDRESS.dkr.ecr.MY_REGION.amazonaws.com/MY_REPO_NAME:latest

I then tried to authorize Kubernetes to access my private repo using the kubernetes documentation, but this is where I have issues.

After trying this with and without the > docker login step, I then

> aws ecr get-login-password --region MY_REGION | docker login --username AWS --password-stdin MY_ECR_ADDRESS.dkr.ecr.MY_REGION.amazonaws.com
> Login Succeeded
> kubectl create secret generic regcred --from-file=.dockerconfigjson=<path/to/.docker/config.json> --type=kubernetes.io/dockerconfigjson
> secret/regcred created

I can view the .docker/config.json file on both windows and wsl2, but it does not have the assumed base64 encoded auth the Kubernetes documentation says should be there. The page says I should see something like this:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "c3R...zE2"
        }
    }
}

But I see something like this:

{
    "auths": {
        "MYECRADDRESS.dkr.ecr.MYREGION.amazonaws.com": {}
    },
    "credsStore": "desktop",
    "stackOrchestrator": "swarm",
    "currentContext": "default"
}

I already figured I needed that base64 encoded auth string in the file and that everything hinges on that, but I tried to deploy anyway:

> kubectl create deployment test --image=MY_ECR_ADDRESS.dkr.ecr.MY_CORRECT_REGION.amazonaws.com/MY_REPO_NAME:latest
> kubectl get deploy
> NAME       READY   UP-TO-DATE   AVAILABLE   AGE
> test   0/1     1            0           2m10s
> > kubectl get pods -l app=test
NAME                       READY   STATUS             RESTARTS   AGE
test-b4d8dccfc-9ttzb   0/1     ImagePullBackOff   0          2m41s

As expected, you'll see that I get the ImagePullBackOff, meaning my local cluster cannot access my private repo.

Can someone please help me get the auth string from the docker creds store and authorize Kubernetes to pull from my private repo?

Just to make it clear `ImagePullBackOff` doesn't necessarily mean that only because it couldn't access the private repo, did you check the exact error via `kubectl describe pod pods-name`? However I think answer below should help you to resolve it. — moonkotte, Jan 07 '22 at 13:13
It doesn't know what auth to use: Failed to pull image "....dkr.ecr.....amazonaws.com/...:latest": rpc error: code = Unknown desc = Error response from daemon: Head "https://....dkr.ecr.....amazonaws.com/v2/.../manifests/latest": no basic auth credentials — Mario, Jan 07 '22 at 18:36

rzlvmp · Answer 1 · 2022-01-11T02:46:43.790

-1

but it does not have the assumed base64 encoded auth the kubernetes documentation says should be there

Docker desktop using own credential store: "credsStore": "desktop". So here is no "auth" key:

Note: If you use a Docker credentials store, you won't see that auth entry but a credsStore entry with the name of the store as value.

Anyway, you may try to use second way to create credentials directly without generating .docker/config.json:

kubectl create secret docker-registry regcred --docker-server=MY_ECR_ADDRESS.dkr.ecr.MY_REGION.amazonaws.com --docker-username=AWS --docker-password="$(aws ecr get-login-password --region MY_REGION)"

Also don't forget to set imagePullSecrets parameter inside deployment description file. In other way you may send secret name parameter directly by using command line override parameter.

The third way how to solve problem is to generate auth block manually.

According to this and this articles auth is just the username:password base64 encoded line. You may generate it by using THE_AUTH="$(echo 'AWS:'$(aws ecr get-login-password --region aws_region) | base64)"; echo $THE_AUTH

edited Jan 11 '22 at 02:46

answered Jan 07 '22 at 02:28

rzlvmp

7,512
5
16
45

thanks - I read what you pointed out - it's why my question is "Can someone please help me get the auth string from the docker creds store". I can't use your other method, because like I said my company uses SSO and we don't have usernames/passwords – Mario Jan 07 '22 at 15:17
@Mario `auth` is a `user:password` base64 encoded pair. So you need a password in any way. Also `aws ecr get-login-password` command return a password. So you using password authentication when logging in docker (`docker login`). I don't understand why you able to login into docker by using password and don't able to do same stuff with Kubernetes. – rzlvmp Jan 11 '22 at 02:49
All the walkthroughs I read tell me to get creds from docker. Since I realized it has nothing to do with docker and just convenient (as of a few years ago when things were stored in plain text, not now), I just used the aws cli - I put in a solution. TBH I think this should 'just work'. IMO, especially with it now being a paid product, Docker Desktop should throw all your secrets into each node and use them automatically so you don't have to do any of this locally. – Mario Jan 11 '22 at 21:29

score -1 · Accepted Answer · answered Jan 11 '22 at 17:07

The quick explanation to this is:

You don't need to get your access out of docker, you can get it from the aws cli
AFAIK, you can't use a deployment on the kubectl cli, you must use a chart to specify the added secret

Details for those that want the solution:

Make sure you have auth in kubernetes by naming the credential whatever you want and using the get-login-password:
> kubectl create secret docker-registry myAwsCred --docker-server=MY_ECR_ADDRESS.dkr.ecr.MY_REGION.amazonaws.com --docker-username=AWS --docker-password=$(aws ecr get-login-password --region MY_REGION)

Create a chart for deployment that specifies whatever you named your secret:

spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - name: app
          image: MY_ECR_ADDRESS.dkr.ecr.MY_REGION.amazonaws.com/MY_REPO:latest
          ports:
          - containerPort: 80
      imagePullSecrets:
        - name: myAwsCred

Then, create a deployment based on your chart that uses the properly named secret:
> kubectl apply -f c:\mylocaltestdeploy.yaml

Can't create local kubernetes secret from docker for AWS ECR when using Okta SSO

2 Answers2