22

I'm using kubernetes on-prem

While I build gitlab using kubernetes has some problem. I think it's related with serviceaccount or role-binding. but couldn't find correct way

I found these posts

Kubernetes log, User "system:serviceaccount:default:default" cannot get services in the namespace

https://github.com/kubernetes/kops/issues/3551

my error logs

==> /var/log/gitlab/prometheus/current <==
2018-12-24_03:06:08.88786 level=error ts=2018-12-24T03:06:08.887812767Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:372: Failed to list *v1.Node: nodes is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"
2018-12-24_03:06:08.89075 level=error ts=2018-12-24T03:06:08.890719525Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:320: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
Community
  • 1
  • 1
Siner
  • 511
  • 2
  • 5
  • 18

1 Answers1

41

The issue is due to your default service account doesn't have the permission to get the nodes or pods at the cluster scope. The minimum cluster role and cluster role binding to resolve that is:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prom-admin
rules:
# Just an example, feel free to change it
- apiGroups: [""]
  resources: ["pods", "nodes"]
  verbs: ["get", "watch", "list"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prom-rbac
subjects:
- kind: ServiceAccount
  name: default
roleRef:
  kind: ClusterRole
  name: prom-admin
  apiGroup: rbac.authorization.k8s.io

The above cluster role provide permission to default service account to access any pods or nodes in any namespace.

You can change the cluster role to provide more permission to service account, if you want to grant access all permission to default service account then, replace resources: ["*"] in prom-admin

Hope this helps.

Prafull Ladha
  • 12,341
  • 2
  • 37
  • 58
  • 1
    Hi @Siner if this or any answer has solved your question please consider [accepting it](https://meta.stackexchange.com/q/5234/179419) by clicking the check-mark. This indicates to the wider community that you've found a solution and gives some reputation to both the answerer and yourself. There is no obligation to do this – Prafull Ladha Dec 24 '18 at 04:32
  • I forgot about this. Your answer will spread out to kubernetes users – Siner Dec 26 '18 at 05:32
  • 2
    Just to add, sometimes it can be **Role** and **RoleBinding** also. – Ivan Aracki Jun 06 '19 at 14:37
  • 1
    This answer worked for me but I did have to add `namespace: default` to the subjects section at the bottom. – timsegraves Mar 27 '20 at 19:03
  • This solved for me: https://kubecloud.io/kubernetes-dashboard-on-arm-with-rbac-61309310a640 – Subha Chandra May 14 '20 at 15:08
  • 2
    For folks landing here, the official documentation on this subject is located here: https://kubernetes.io/docs/reference/access-authn-authz/rbac/ – speedplane Dec 19 '20 at 15:20
  • 1
    The differentiator for me was the `kind: ClusterRole`. I was following docs that used `kubectl create role`, I needed `kubectl create clusterrole` to run node-related commands – E. Moffat Jul 29 '21 at 22:45