"services is forbidden: User \"system:serviceaccount:tick:external-dns\" cannot list resource \"services\" in API group \"\" at the cluster scope"

Question

I've been following the walkthrough to create an AWS ALB Ingress Controller for my app which is also deployed at an EKS cluster.
Everything seems okay , similar answers with the walkthrough but when it comes to the setting up of an external DNS I get the error :

kubectl logs -f $(kubectl get po | egrep -o 'external-dns[A-Za-z0-9-]+')

time="2020-02-20T16:21:57Z" level=error msg="services is forbidden: User \"system:serviceaccount:tick:external-dns\" cannot list resource \"services\" in API group \"\" at the cluster scope" time="2020-02-20T16:22:58Z" level=error msg="services is forbidden: User \"system:serviceaccount:tick:external-dns\" cannot list resource \"services\" in API group \"\" at the cluster scope"

every one minute . I made sure that all the permissions are the needed ones so it should not be because of that.

I tried the solutions from here , but nothing helped and I couldn't find any other solutions.

What does this error practically means? What should I do to fix it?

UPDATE EDIT my external-dns configuration looks like:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::*my*account*id*:role/EKSRole
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: tick
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  selector:
    matchLabels:
      app: external-dns
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::*my*account*id*:role/EKSRole
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.5.9
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=external-dns-test.my-org.com   #external-dns-test.my-org.com # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
        - --provider=aws
        - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
        - --aws-zone-type=public # only look at public hosted zones (valid values are public, private or no value for both)
        - --registry=txt
        - --txt-owner-id=my-identifier
      securityContext:
        fsGroup: 65534

marcincuber · Accepted Answer · 2020-02-21T12:30:24.743

8

Your error suggests that service account with name external-dns in tick namespace can't perform certain actions. In this case it is list services. To solve this you can apply the following:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: tick
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns-role
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns-role
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: tick

Note, that first rule inside ClusterRole is granting correct permissions to list services in "" apiGroup, which is solving the error you have reported in your question.

edited Feb 21 '20 at 12:30

answered Feb 21 '20 at 08:21

marcincuber

3,451
1
17
29

1

Thanks , the only thing that was different was the field ```namespace``` which was ```default``` and changed it to ```tick``` . Now I need to figure out how to access my app through the service – Flora Biletsiou Feb 21 '20 at 11:59
1

@FloraBiletsiou I just updated my answer. In you ServiceAccount definition, you are also missing a namespace. If you don't specify it, the service account usually gets created in kube-system or default namespace. – marcincuber Feb 21 '20 at 12:31
1

the ServiceAccount was already at my ```tick``` namespace because I set it to be the default one, but I changed anyway – Flora Biletsiou Feb 21 '20 at 16:17
1

I am currently struggling the same issue, or at least the same error, despite the command: `kubectl auth can-i list service/poc-service --as system:serviceaccount:poc:poc-sa` returns `yes`. I tried assigning `cluster-admin` to `poc-sa` (with `kubectl create clusterrolebinding poc-admin --clusterrole cluster-admin --serviceaccount=poc:poc-sa`) and that seems to get rid of the error. But that's a really bad solution, given the security risk involved. – lubumbax Jul 18 '20 at 15:23

score 0 · Answer 2 · answered Feb 20 '20 at 21:25

0

I think you are using Amazon EKS with IAM Roles for Service Accounts therefore following this should work. The walkthrough you were following, in the external DNS section Step 2 yaml file doesn't have your

annotations:
    # Substitute your account ID and IAM service role name below.
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT-ID:role/IAM-SERVICE-ROLE-NAME

This error practically means your serviceaccount (external-dns) doesn't have permissions to access the kubernetes api server

answered Feb 20 '20 at 21:25

garlicFrancium

2,013
14
23

1

Thank you for your answer, I actually added this part to my external-dns ServiceAccount configuration ,but since I am a beginner at AWS I am not sure if I have the correct **IAM-SERVICE-ROLE-NAME**. my configuration looks like : `annotations: eks.amazonaws.com/role-arn:arn:aws:iam::**myaccountid**:role/EKSRole ` I am not sure how to find the IAM-SERVICE-ROLE-NAME – Flora Biletsiou Feb 21 '20 at 09:58

"services is forbidden: User \"system:serviceaccount:tick:external-dns\" cannot list resource \"services\" in API group \"\" at the cluster scope"

2 Answers2