Forbidden to access Kubernetes API Server

Question

I have defined a ClusterRole for Prometheus:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
  labels:
    k8s-app: prometheus
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - endpoints
  - services
  - nodes
  - pods
  verbs:
  - get
  - watch
  - list
- nonResourceURLs:
  - /metrics
  - /api/*
  verbs:
  - get

Prometheus is able to access the API-Servers /metrics route:

https://10.0.1.104:443/metrics
https://10.0.2.112:443/metrics

But I get "server returned HTTP status 403 Forbidden" on

https://kubernetes.default.svc:443/api/v1/nodes/ip-10-0-0-219.eu-west-1.compute.internal/proxy/metrics

and

https://kubernetes.default.svc:443/api/v1/nodes/ip-10-0-0-219.eu-west-1.compute.internal/proxy/metrics/cadvisor

I thought I had this covered by

- nonResourceURLs:
  - /api/*

What am I missing?

Do you have a `ClusterRoleBinding` to bind this to Prometheus? — Arian Motamedi, Nov 09 '18 at 17:32
sure - else access to https://10.0.2.112:443/metrics would not work — Ronald, Nov 09 '18 at 17:40
try to clusterrolebinding prometheus with cluster-admin for testing — sure ruan, May 15 '19 at 06:17

score 9 · Accepted Answer · answered Nov 09 '18 at 23:57

I tried this myself and yes nodes/proxy is missing. (it works for me after adding it)

rules:
- apiGroups: [""]
  resources:
  - namespaces
  - endpoints
  - services
  - nodes
  - nodes/proxy <===
  - pods

# From my K8s master
$ curl -k -H 'Authorization: Bearer <redacted>' https://localhost:6443/api/v1/nodes/ip-x-x-x-x.us-west-2.compute.internal/proxy/stats/summary
{
  "node": {
   "nodeName": "ip-x-x-x-x.us-west-2.compute.internal",
   "systemContainers": [
    {
     "name": "kubelet",
     "startTime": "2018-10-19T21:02:19Z",
     "cpu": {
      "time": "2018-11-09T23:51:15Z",
      "usageNanoCores": 30779949,
      "usageCoreNanoSeconds": 59446529195638
     },
....

Removing it:

$ curl -k -H 'Authorization: Bearer <redacted>'  https://localhost:6443/api/v1/nodes/ip-x-x-x-x.us-west-2.compute.internal/proxy/stats/summary
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "nodes \"ip-x-x-x-x.us-west-2.compute.internal\" is forbidden: User \"system:serviceaccount:default:prometheus-k8s\" cannot get resource \"nodes/proxy\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "name": "ip-x-x-x-x.us-west-2.compute.internal",
    "kind": "nodes"
  },
  "code": 403
}

For anyone who could not find "nodes/proxy" in the documentation like myself: https://stackoverflow.com/questions/49396607/where-can-i-get-a-list-of-kubernetes-api-resources-and-subresources — Ronald, Nov 12 '18 at 07:40
I've been looking for this answer for 5 continous working days! I appreciate it! — Lethargos, Aug 30 '22 at 21:32

apisim · Answer 2 · 2018-11-09T22:26:31.217

2

For those two endpoints, the rules may be missing nodes/metrics and nodes/proxy for (sub)resources, and possibly the proxy verb.

If acceptable from security standpoint, it will be much easier to assign the cluster-reader role to the prometheus' service account.

edited Nov 09 '18 at 22:26

answered Nov 09 '18 at 19:06

apisim

4,036
1
10
16

There is no cluster role "cluster-reader" in my kubernetes instance. I have a cluster role "view" but this does not work as it does not have enough rights. – Ronald Nov 12 '18 at 07:34
@Ronald - my bad. `cluster-reader` is a standard cluster role in OpenShift, not "plain" Kubernetes. I couldn't try any of my suggestions when I provided the answer and was working off of some vague memory about how to reference subresources in roles rules. Today I found it in the documentation - https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources Not great but it can be useful, I think. Cheers! – apisim Nov 13 '18 at 00:35

Forbidden to access Kubernetes API Server

2 Answers2

Linked