1

I've been working on a setup for Kubernetes (while simultaneously learning Kubernetes) and I've hit a snag that I thing suggests what I want to do is impossible. My setup is described in full detail here: Can a self-signed cert secure multiple CNs / FQDNs?

After generated a self-signed cert with an SAN and configuring my registry and setting up a DaemonSet to trust the self-signed certificate on all nodes in my cluster, I ran into a new issue: Kubernetes coudn't find the host "docker-registry"

There is a ClusterIP service running with the name "docker-registry" which points to the Docker registry. When I kubectl exec a shell on a pod in the cluster for debug purposes I can see that the KubeDNS is working and locating this service:

root@debug:/# ping docker-registry
PING docker-registry.default.svc.cluster.local (10.245.46.24) 56(84) bytes of data.
^C
--- docker-registry.default.svc.cluster.local ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2039ms

I assume the packet loss is because the service only responds to traffic on port 443 and doesn't respond to pings. What's important is that the hostname was resolved to an IP address.

Here's my problem, though: This hostname only resolves from pods within the cluster. Kubernetes, itself, can't hit this hostname. I've defined a pod like so:

apiVersion: v1
kind: Pod
metadata:
  name: test-docker
  labels:
    name: test
spec:
  containers:
  - name: test
    image: docker-registry/my-ubuntu

When I try to create this pod (kubectl create -f pods/test.yml) I get the following error:

...
Failed to pull image "docker-registry/my-ubuntu": rpc error: code = Unknown desc = Error response from daemon: pull access denied for docker-registry/my-ubuntu, repository does not exist or may require 'docker login'
...

This makes some sense, intuitively. While other pods can access my registry at the host docker-registry, the Kubernetes nodes can't because this isn't a valid host or defined in the /etc/hosts file.

I tried doing some preliminary Googling and found the following Kubernetes document: https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-services/

This suggests that the only ways to access a service running inside of a cluster (as is the case with my Docker registry) are:

  1. Use a NodePort (this is no good, as it would expose my Docker registry to the internet)
  2. Use the Proxy Verb (it's not clear to me exactly how this method works or how authentication is handled, but it sounds like I may be able to replace docker-registry with a ridiculously long and hard-to-remember URL like https://<node-ip>/api/v1/namespaces/default/services/docker-registry)
  3. Access from a pod within the cluster (this is no good, as I need Kubernetes to perform the image pull and I'm not doing it manually)

It may not be possible to accomplish what I want (have a Docker registry that's available to any node in my cluster but not available to the public internet) but I wanted to ask on StackOverflow if there's anything I'm missing.

If what I want isn't possible then my last resort (before paying for Docker Hub) will be to try just exposing my registry to the public internet but utilize Docker authentication and a pull secret (plus possibly putting my registry behind a firewall to block traffic from outside of my cluster)

stevendesu
  • 15,753
  • 22
  • 105
  • 182
  • you may want to check this https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ – Nirav Nov 03 '20 at 14:52
  • @Nirav that document describes how to pull from a registry that's accessible via the internet but utilizes Docker auth to make it "private". I'm interested in running a registry that isn't accessible via the internet. Essentially the same as running a local registry except it needs to be accessible to all nodes in my cluster and not just the master node. – stevendesu Nov 03 '20 at 14:55
  • Why would a NodePort service expose your registry to the Internet? Where are you running your Cluster? – Howard_Roark Nov 03 '20 at 17:14
  • @Howard_Roark the cluster is running on DigitalOcean, using their managed Kubernetes services. If you know a node IP address then you can access any NodePort service at `:`. There's also a load balancer in front of the cluster with a domain name which is manually configured to only forward certain ports, and the load balancer allows ports below 30000. So I generally run a web server on nodeport 30080 then configure the load balancer to redirect traffic from port 80 to cluster port 30080. Then I can see the website at `http://example.com` – stevendesu Nov 03 '20 at 18:10
  • @Howard_Roark Per Kubernetes' own documentation on using NodePort: "Depending on your cluster environment, this may just expose the service to your corporate network, or it may expose it to the internet. Think about whether the service being exposed is secure." -- I definitely fall into the situation of "it may expose it to the internet" – stevendesu Nov 03 '20 at 18:11
  • I see-- well at least in the AWS world it is easy to ensure NodePort services are private because you ensure that the Nodes themselves are in private subnets. I guess your setup runs in public subnets? – Howard_Roark Nov 03 '20 at 18:14
  • @Howard_Roark ¯\\_(ツ)_/¯ I wish I could provide a better answer, but my experience with devops up until now has been root access on a single virtual machine. I've never dealt with subnets, clusters, load balancers, or any of this stuff before. It's all new to me. I'm using this personal project as a learning experience. I think DigitalOcean has a "private networking" option when creating new VMs, so I can look into that. Maybe it's possible to create clusters such that NodePorts aren't exposed to the internet. But as it's configured now, that's not the case. – stevendesu Nov 03 '20 at 18:16
  • Have you tried with kubernetes documentation [Configuring nodes to authenticate to a private registry](https://kubernetes.io/docs/concepts/containers/images/#configuring-nodes-to-authenticate-to-a-private-registry)? – Jakub Nov 04 '20 at 09:55

0 Answers0