How to operate the k8s cluster in the docker container

Question

How does a docker container running on a docker machine instead of a k8s pod operate the k8s cluster. For example, if i need to do something like this inside a container:

kubectl get pods

In my dockerfile, I installed kubectl

RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN sudo mv ./kubectl /usr/local/bin/kubectl

when i run kubectl get pods, the result is as follows:

kubectl get pod
error: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

So I mounted the config into the docker container at docker runcommand

docker run -v /root/.kube/config:/root/.kube/config my-images

the result is as follows:

kubectl get pod
Error in configuration: 
* unable to read client-cert /root/.minikube/profiles/minikube/client.crt for minikube due to open /root/.minikube/profiles/minikube/client.crt: no such file or directory
* unable to read client-key /root/.minikube/profiles/minikube/client.key for minikube due to open /root/.minikube/profiles/minikube/client.key: no such file or directory
* unable to read certificate-authority /root/.minikube/ca.crt for minikube due to open /root/.minikube/ca.crt: no such file or directory

This seems to be due to the current-context: minikube in the k8s config file

Then mount the authentication file again, it run success.

Now, I can call the kubectl get pods command or otherwise manipulate a cluster outside the container when I mount -v /root/.kube/config:/root/.kube/config -v /root/.minikube/:/root/.minikube/, however, this does not apply to cluster mounts created by kubeadm or otherwise.

But I want to be able to mount the required configuration files and so on to the container in a uniform way so that I can use the same command to manipulate the k8s cluster, which may be created by minikube or rancher k3s or kubeadm

In summary, I want to mount a uniform set of files or directories for all cases of the k8s cluster, such as -v file: file -v dir:dir, to implement operations on the k8s cluster created in any way, such as getting the pod status, creating, deleting various types of resources, and so on

I need to have the maximum permission to operate on k8s

Can someone please tell me what is it that I need to do?

Answer 1

I think you can set the Docker user when running your container

You can run (in this example - ubuntu image) with an explicit user id and group id.

$ docker run -it --rm \
  --mount "type=bind,src=$(pwd)/shared,dst=/opt/shared" \
  --workdir /opt/shared \
  --user "$(id -u):$(id -g)" \
  ubuntu bash

The difference is ‘–user “$(id -u):$(id -g)“’ - they tell the container to run with the current user id and group id which are obtained dynamically through bash command substitution by running the “id -u” and “id -g” and passing on their values.

This can be good enough already. The problem here is, that the user and group don’t really exist in the container. This approach works for the terminal command, but the session looks broken and you’ll see some ugly error messages like:

"groups: cannot find name for group ID"
"I have no name!"
  - your container, complaining

While bash works, some apps might refuse to run if those configs look fishy.

Next you have to configure and run your Docker containers correctly, so you don’t have to fight permission errors and access your files easily.

As you should create a non-root user in your Dockerfile in any case, this is a nice thing to do. You might as well set the user id and group id explicitly.

Below is a minimal Dockerfile which expects to receive build-time arguments, and creates a new user called “user”:

FROM ubuntu

ARG USER_ID
ARG GROUP_ID

RUN addgroup --gid $GROUP_ID user
RUN adduser --disabled-password --gecos '' --uid $USER_ID --gid $GROUP_ID user
USER user

Take a look: add-user-to-container.

You can use this Dockerfile, to build a fresh image with the host uid and gid. This image, needs to be built specifically for each machine it will run on to make sure everything is in order.

Then, you can run use this image for our command. The user id and group id are correct without having to specify them when running the container.

$ docker build -t your-image \
  --build-arg USER_ID=$(id -u) \
  --build-arg GROUP_ID=$(id -g) .
$ docker run -it --rm \
  --mount "type=bind,src=$(pwd)/shared,dst=/opt/shared" \
  --workdir /opt/shared \
  your-image bash

There is no need to use “chown”, and you will get rid of annoying permission errors anymore. Please take a look on this very interesting article: kubernetes-management-docker, docker-shared-permissions.