Kubernetes forbidden: User "system:anonymous" cannot get path "/"

Question

I'm struggling to expose my app over the Internet when deployed to AWS EKS.

I have created a deployment and a service, I can see both of these running when using kubectl. I can see that the app has successfully connected to an external database as it runs a script on startup that initialises said database.

My issue is arising when trying to access the app over the internet. I have tried accessing the cluster endpoint and I am getting this error:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User "system:anonymous" cannot get path "/"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

However, if I access the "/readyz" path I get "ok" returned. "/version" returns the following:

{
  "major": "1",
  "minor": "16+",
  "gitVersion": "v1.16.8-eks-e16311",
  "gitCommit": "e163110a04dcb2f39c3325af96d019b4925419eb",
  "gitTreeState": "clean",
  "buildDate": "2020-03-27T22:37:12Z",
  "goVersion": "go1.13.8",
  "compiler": "gc",
  "platform": "linux/amd64"
}

My deployment.yml file contains the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
  labels:
    app: client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
      - name: client
        image: image/repo
        ports:
        - containerPort: 80
        imagePullPolicy: Always

My service.yml:

apiVersion: v1
kind: Service
metadata:
  name: client
  labels:
    run: client
spec:
  type: LoadBalancer
  ports:
    - name: "80"
      port: 80
      targetPort: 80
      protocol: TCP
  selector:
    run: client

I can see the Load Balancer has been created in the AWS console and I have tried updating the security group of the LB to be able to talk to the cluster endpoint. The LB dashboard is showing the one attached instance is 'OutOfService' and also under the monitoring tab, I can see one Unhealthy Host.

I've tried accessing the Load Balancer endpoint as provided in the EC2 area of the console (this matches what is returned from kubectl get services as the EXTERNAL-IP of the LB service) and I'm getting an empty response from there.

curl XXXXXXX.eu-west-2.elb.amazonaws.com:80
curl: (52) Empty reply from server

This is the same when accessing in a web browser.

I seem to be going round in circles with this one any help at all would be greatly appreciated.

I never got to the bottom of the issue here. I started again and used A pre-made Helm chart for the software I was trying to deploy and it worked. — SteveJDB, Jun 23 '20 at 14:43

score 1 · Answer 1 · answered Jun 05 '20 at 02:46

1

I've tried accessing the Load Balancer endpoint

You are accessing the EKS URL, which is the kubernetes apiserver endpoint, and not the LoadBalancer that was (hopefully) created for your client Service

You will want to kubectl get -o wide svc client and if it was successful in provisioning a LoadBalancer for you, then its URL will appear in the output. You can get more details about that situation by kubectl describe svc client, which will include any events that affected it during provisioning

answered Jun 05 '20 at 02:46

mdaniel

31,240
5
55
58

Thank you for getting back to me, I realise that the end of my question may not have been clear so I have revised it (it had been a long day trying to sort this out). I have been trying to access the LB endpoint (as well as the cluster endpoint which is giving me the output referenced above). – SteveJDB Jun 05 '20 at 08:42

score 0 · Answer 2 · answered Jun 05 '20 at 09:01

Because your eks instance is OutOfService in the LoadBalancer section, you should check which port the LoadBalancer is doing Health Check on.

You can do that by executing kubectl get svc client -oyaml and seeing the nodePort section.

After that, check that your LoadBalancer is doing the Health Check to this exact port, if not than change it to the correct one.

If you have the correct port and but the instance is still OutOfService then i suggest you go to the security group of your eks instance and give the specific port access from the ELB.

Kubernetes forbidden: User "system:anonymous" cannot get path "/"

2 Answers2

Linked