nginx-ingress stopped working, despite Ready service + pod ("does not have any active Endpoint.")

Question

I have transferred my microk8s setup to a new server and found that the once-working ingress setup my trial setup stopped working.

I am running this minimal whoami-app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: default
  labels:
    app: whoami
spec:
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami
        image: containous/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: default
spec:
  selector:
    app: whoami
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami
  namespace: default
  annotations:
    kubernetes.io/ingress.class: public
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: whoami
            port:
              number: 80

Pod is up and running, service exposed it properly, but the ingress is not working:

kubectl get services whoami
NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
whoami   ClusterIP   10.152.183.184   <none>        80/TCP    26m

curl 10.152.183.184
Hostname: whoami-567b85d54d-qbbd5
IP: 127.0.0.1
IP: ::1
IP: 10.1.76.7
IP: fe80::e850:aaff:fe72:91c4
RemoteAddr: 192.168.0.102:21910
GET / HTTP/1.1
Host: 10.152.183.184
User-Agent: curl/7.68.0
Accept: */*

kubectl get ingress whoami
NAME     CLASS    HOSTS   ADDRESS     PORTS   AGE
whoami   <none>   *       127.0.0.1   80      28m

The nginx-ingress-controller log shows these entries:

controller.go:1076] Service "default/whoami" does not have any active Endpoint.

But again, accessing through the clusterIP works, so both the Pod and the Service are doing their job.

Be aware that endpoints are determined by a container's `Readiness`, so that could impact things, and also that `Endpoint` objects are formal objects in kubernetes, so you can check for yourself: `kubectl -n default get endpoint` — mdaniel, Mar 02 '22 at 16:51
output of `kubeclt -n default get endpoints` is `whoami 10.1.76.11:80`, but the pod is running on `10.1.76.21`. Should those two be the same? — petwri, Mar 03 '22 at 19:24

score 0 · Accepted Answer · answered Mar 03 '22 at 19:37

0

I don't really know how it happened, but the endpoint was not matching the pod IP. I deleted the endpoint manually usind kubectl delete endpoints whoami, it got recreated with the now correct IP, now ingress seems to work.

answered Mar 03 '22 at 19:37

petwri

553
1
4
11

score 0 · Answer 2 · answered Mar 16 '22 at 21:42

I am glad to hear you managed it work.

I tried to find the reason of that behavior but unfortunately I haven't found it yet. At this link there are some solutions to similar warning. Maybe it will help someone.

First one is situation when ingress class of the ingress controller mismatch ingress class in the ingress resource manifest used for services.

Nginx installation command, short example:

helm install stable/nginx-ingress \   
--name ${INGRESS_RELEASE_NAME} \   
--namespace ${K8S_NAMESPACE} \   
--set controller.scope.enabled=true \   
--set controller.scope.namespace=${K8S_NAMESPACE} \   
--set controller.ingressClass=${NGINX_INGRESS_CLASS}

ingress resource spec. , excerpt:

apiVersion: extensions/v1beta1 
kind: Ingress metadata:  
labels:
   annotations:
    # folowing line is not valid for K8s or Helm, 
    # but reflects the values must be the same
    kubernetes.io/ingress.class: ${NGINX_INGRESS_CLASS}

In second solution is including id directive in service selector:

  selector:
    app: whoami
    id: "0"

See also these questions on StackOverflow:

nginx-ingress stopped working, despite Ready service + pod ("does not have any active Endpoint.")

2 Answers2

Linked