k3s image pull from private registries

Question

I've been looking at different references on how to enable k3s (running on my pi) to pull docker images from a private registry on my home network (server laptop on my network). If someone can please point my head in the right direction? This is my approach:

1. Created the docker registry on my server (and making accessible via port 10000):

docker run -d -p 10000:5000 --restart=always --local-docker-registry registry:2

This worked, and was able to push-pull images to it from the "server pc". I didn't add authentication TLS etc. yet...

(viewing the images via docker plugin on VS Code).

2. Added the inbound firewall rule on my laptop server, and tested that the registry can be 'seen' from my pi (so this also works):

$ curl -ks http://<server IP>:10000/v2/_catalog
{"repositories":["tcpserialpassthrough"]}

3. Added the registry link to k3s (k3s running on my pi) in registries.yaml file, and restarted k3s and the pi

$ cat /etc/rancher/k3s/registries.yaml
mirrors:
  pwlaptopregistry:
    endpoint:
      - "http://<host IP here>:10000"

4. Putting the registry prefix to my image endpoint on a deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tcpserialpassthrough
spec:
  selector:
    matchLabels:
      app: tcpserialpassthrough
  replicas: 1
  template:
    metadata:
      labels:
        app: tcpserialpassthrough
    spec:
      containers:
      - name: tcpserialpassthrough
        image: pwlaptopregistry/tcpserialpassthrough:vers1.3-arm
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 8001
          hostPort: 8001
          protocol: TCP
        command: ["dotnet", "/app/TcpConnector.dll"]

However, when I check the deployment startup sequence, it's still not able to pull the image (and possibly also still referencing docker hub?):

kubectl get events -w
LAST SEEN   TYPE      REASON             OBJECT                                      MESSAGE
8m24s       Normal    SuccessfulCreate   replicaset/tcpserialpassthrough-88fb974d9   Created pod: tcpserialpassthrough-88fb974d9-b88fc
8m23s       Warning   FailedScheduling   pod/tcpserialpassthrough-88fb974d9-b88fc    0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
8m23s       Warning   FailedScheduling   pod/tcpserialpassthrough-88fb974d9-b88fc    0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
8m21s       Normal    Scheduled          pod/tcpserialpassthrough-88fb974d9-b88fc    Successfully assigned default/tcpserialpassthrough-88fb974d9-b88fc to raspberrypi
6m52s       Normal    Pulling            pod/tcpserialpassthrough-88fb974d9-b88fc    Pulling image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm"
6m50s       Warning   Failed             pod/tcpserialpassthrough-88fb974d9-b88fc    Error: ErrImagePull
6m50s       Warning   Failed             pod/tcpserialpassthrough-88fb974d9-b88fc    Failed to pull image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": failed to resolve reference "docker.io/pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
6m3s        Normal    BackOff            pod/tcpserialpassthrough-88fb974d9-b88fc    Back-off pulling image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm"
3m15s       Warning   Failed             pod/tcpserialpassthrough-88fb974d9-b88fc    Error: ImagePullBackOff

Wondered if the issue is with authorization, and added based on basic auth, following this youtube guide, but the same issue persists. Also noted that that /etc/docker/daemon.json must be edited to allow unauthorized, non-TLS connections, via:

{
  "Insecure-registries": [ "<host IP>:10000" ]
}

but seemed that this needs to be done on node side, whereas nodes don't have docker cli installed??

Answer 1

... this is so stupid, have no idea why a domain name and port needs to be specified as the "name" of your referred registry, but anyway this solved my issue (for reference):

$cat /etc/rancher/k3s/registries.yaml
mirrors:
  "<host IP>:10000":
    endpoint:
      - "http://<host IP>:10000"

and restarting k3s:

systemctl restart k3s

Then in your deployment, referring to that in your image path as:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tcpserialpassthrough
spec:
  selector:
    matchLabels:
      app: tcpserialpassthrough
  replicas: 1
  template:
    metadata:
      labels:
        app: tcpserialpassthrough
    spec:
      containers:
      - name: tcpserialpassthrough
        image: <host IP>:10000/tcpserialpassthrough:vers1.3-arm
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 8001
          hostPort: 8001
          protocol: TCP
        command: ["dotnet", "/app/TcpConnector.dll"]
      imagePullSecrets:
      - name: mydockercredentials

referring to registry's basic auth details saved as a secret:

$ kubectl create secret docker-registry mydockercredentials --docker-server host IP:10000 --docker-username username --docker-password password

You'll be able to verify the pull process via

$ kubectl get events -w