How to properly access multiple kubernetes cluster using kubectl

Question

I have two clusters and the config files are stored in .kube. I am exporting KUBECONFIG as below

export KUBECONFIG=/home/vagrant/.kube/config-cluster1:/home/vagrant/.kube/config-cluster2

checking the contexts

kubectl config get-contexts
CURRENT   NAME        CLUSTER     AUTHINFO           NAMESPACE
*         cluster-1   cluster-1   kubernetes-admin   
          cluster-2   cluster-2   kubernetes-admin

But when I choose cluster-2 as my current context I get an error

kubectl config get-contexts
CURRENT   NAME        CLUSTER     AUTHINFO           NAMESPACE
*         cluster-1   cluster-1   kubernetes-admin   
          cluster-2   cluster-2   kubernetes-admin   

kubectl config use-context cluster-2
Switched to context "cluster-2".


kubectl get pods -A
error: You must be logged in to the server (Unauthorized)

If I export only the config for cluster-2 and try running kubectl it works fine.

My question is whether I am exporting the config files properly or should I be doing something more.

you see `cluster-2` in `get-contexts` so that has to come from somewhere. Either you're loading both files, or you have details of both clusters in one file. My question would be if you have a stray section in `.kube/config-cluster1` that conflicts with `.kube/config-cluster2` — Paul Becotte, Nov 04 '20 at 16:31
if possible attach both the kubeconfig files to this question . we just need to stich them together into a single file.which is different to what you are doing ("export KUBECONFIG=/home/vagrant/.kube/config-cluster1:/home/vagrant/.kube/config-cluster2") — confused genius, Nov 07 '20 at 08:44

score 8 · Accepted Answer · answered Nov 04 '20 at 17:01

You need to separate the AUTHINFO (context.user on config file) for each cluster with the respective credentials.

For example:

apiVersion: v1
clusters:
- cluster:
    server: https://192.168.10.190:6443
  name: cluster-1
- cluster:
    server: https://192.168.99.101:8443
  name: cluster-2
contexts:
- context:
    cluster: cluster-1
    user: kubernetes-admin-1
  name: cluster-1
- context:
    cluster: cluster-2
    user: kubernetes-admin-2
  name: cluster-2
kind: Config
preferences: {}
users:
- name: kubernetes-admin-1
  user:
    client-certificate: /home/user/.minikube/credential-for-cluster-1.crt
    client-key: /home/user/.minikube/credential-for-cluster-1.key
- name: kubernetes-admin-2
  user:
    client-certificate: /home/user/.minikube/credential-for-cluster-2.crt
    client-key: /home/user/.minikube/credential-for-cluster-2.key

You can find more useful tips in the following article:

Using different kubectl versions with multiple Kubernetes clusters:

When you are working with multiple Kubernetes clusters, it’s easy to mess up with contexts and run kubectl in the wrong cluster. Beyond that, Kubernetes has restrictions for versioning mismatch between the client (kubectl) and server (kubernetes master), so running commands in the right context does not mean running the right client version.

To overcome this:

Use asdf to manage multiple kubectl versions
Set the KUBECONFIG env var to change between multiple kubeconfig files
Use kube-ps1 to keep track of your current context/namespace
Use kubectx and kubens to change fast between clusters/namespaces
Use aliases to combine them all together

I also recommend the following reads:

Mastering the KUBECONFIG file by Ahmet Alp Balkan (Google Engineer)
How Zalando Manages 140+ Kubernetes Clusters by Henning Jacobs (Zalando Tech)

score 0 · Answer 2 · answered Aug 06 '22 at 08:37

0

I wrote a script to switch kubeconfig and namespace easily. Hope it can help you.

. k-use -k <kubeconfig> -n <namespace>

https://github.com/kingonion/k-use

answered Aug 06 '22 at 08:37

Kingonion

1
1

R3m1n0X · Answer 3 · 2023-03-22T15:58:58.867

The post from Eduardo Baitello is already very awesome, but I would still like to add some things that might help to handle serveral kubeconfigs.

If you work in large environments, you usually have to handle several kubeconfigs.

Add to the .bashrc/.zshrc (etc):

export KUBECONFIG="~/.kube/config:$(find ~/.kube/kubeconfig* -type f -maxdepth 1 | tr '\n' ':')"

This will load ALL kubeconfigs which can be found under ~/.kube. With the increase of -maxdepth you can index subfoldes also. They are available in every SHELL and can be accessed by kubectl or kubectx without having to make a new export from another kubeconfig.

Alternatively you can adapt the following alias:

alias kubeconfig-reload="export KUBECONFIG="~/.kube/config:$(find ~/.kube/kubeconfig* -type f -maxdepth 1 | tr '\n' ':')""

This will - if you have a new kubeconfig - reload it without having to restart your SHELL.

Now you can either display the configs with "kubectl" - "kubectl config get-contexts" and switch them with "kubectl config set-context" or use kubctx (https://github.com/ahmetb/kubectx). kubectx also gives you the advantage to use kubens, which allows you to switch the namespace permanently.

How to properly access multiple kubernetes cluster using kubectl

3 Answers3