Forbidden after enabling Google Cloud Groups RBAC in GKE

Question

We are enabling Google Cloud Groups RBAC in our existing GKE clusters.

For that, we first created all the groups in Workspace, and also the required "gke-security-groups@ourdomain.com" according to documentation.

Those groups are created in Workspace with an integration with Active Directory for Single Sign On.

All groups are members of "gke-security-groups@ourdomain" as stated by documentation. And all groups can View members.

The cluster was updated to enabled the flag for Google Cloud Groups RBAC and we specify the value to be "gke-security-groups@ourdomain.com".

We then Added one of the groups (let's called it group_a@ourdomain.com) to IAM and assigned a custom role which only gives access to:

"container.apiServices.get",
"container.apiServices.list",
"container.clusters.getCredentials",
"container.clusters.get",
"container.clusters.list",

This is just the minimum for the user to be able to log into the Kubernetes cluster and from there being able to apply Kubernetes RBACs.

In Kubernetes, we applied a Role, which provides list of pods in a specific namespace, and a role binding that specifies the group we just added to IAM.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-role
  namespace: custom-namespace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: test-rolebinding
  namespace: custom-namespace
roleRef:
  kind: Role
  name: test-role
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: Group
    name: group_a@ourdomain.com

Everything looks good until now. But when trying to list the pods of this namespace with the user that belongs to the group "group_a@ourdomain.com", we get:

Error from server (Forbidden): pods is forbidden: User "my-user@ourdomain.com" cannot list resource "pods" in API group "" in the namespace "custom-namespace": requires one of ["container.pods.list"] permission(s).

Of course if I give container.pods.list to the group_a@ourdomain assigned role, I can list pods, but it opens for all namespaces, as this permission in GCloud is global.

What am I missing here?

Not sure if this is relevant, but our organisation in gcloud is called for example "my-company.io", while the groups for SSO are named "...@groups.my-company.io", and the gke-security-groups group was also created with the "groups.my-company.io" domain.

Also, if instead of a Group in the RoleBinding, I specify the user directly, it works.

Answer 1

It turned out to be an issue about case-sensitive strings and nothing related with the actual rules defined in the RBACs, which were working as expected.

The names of the groups were created in Azure AD with a camel case model. These group names where then showed in Google Workspace all lowercase.

Example in Azure AD: thisIsOneGroup@groups.mycompany.com

Example configured in the RBACs as shown in Google Workspace: thisisonegroup@groups.mycompany.com

We copied the names from the Google Workspace UI all lowercase and we put them in the bindings and that caused the issue. Kubernetes GKE is case sensitive and it didn't match the name configured in the binding with the email configured in Google Workspace.

After changing the RBAC bindings to have the same format, everything worked as expected.

Answer 2

Looks like you are trying to grant access to deployments in the extensions and apps API groups. That requires the user to specify the extensions and apps api group in your role rules:

rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - '*'
- apiGroups:
  - extensions
  - apps
  resources:
  - deployments
  - replicasets
  verbs:
  - '*'

I can recommend you to recreate role and role bindings too. You can visit the following thread as a reference too RBAC issue : Error from server (Forbidden):

Edited 012622:

Can you please confirm that you provided the credentials or configuration file (manifest, YAML)? As you may know, this information is provided by Kubernetes and the default service account. You can verify it by running:

$ kubectl auth can-i get pods

Let me tell you that the account type you need to use for your accounts is “service account”. To create a new service account with a wider set of permissions, the following is a YAML example:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-read-role
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
 
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: pod-read-sa
 
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pod-read-rolebinding
namespace: default
subjects:
- kind: ServiceAccount
name: pod-read-sa
apiGroup: ""
roleRef:
kind: Role
name: pod-read-role
apiGroup: ""

Please use the following thread as a reference.