2

I created a Deployment, Service and an Ingress. Unfortunately, the ingress-nginx-controller pods are complaining that my Service does not have an Active Endpoint:

controller.go:920] Service "<namespace>/web-server" does not have any active Endpoint.

My Service definition:

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/should_be_scraped: "false"
  creationTimestamp: "2021-06-22T07:07:18Z"
  labels:
    chart: <namespace>-core-1.9.2
    release: <namespace>
  name: web-server
  namespace: <namespace>
  resourceVersion: "9050796"
  selfLink: /api/v1/namespaces/<namespace>/services/web-server
  uid: 82b3c3b4-a181-4ba2-887a-a4498346bc81
spec:
  clusterIP: 10.233.56.52
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: web-server
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

My Deployment definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2021-06-22T07:07:19Z"
  generation: 1
  labels:
    app: web-server
    chart: <namespace>-core-1.9.2
    release: <namespace>
  name: web-server
  namespace: <namespace>
  resourceVersion: "9051062"
  selfLink: /apis/apps/v1/namespaces/<namespace>/deployments/web-server
  uid: fb085727-9e8a-4931-8067-fd4ed410b8ca
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: web-server
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: web-server
    spec:
      containers:
      - env:
        <removed environment variables>
        image: <url>/<namespace>/web-server:1.10.1
        imagePullPolicy: IfNotPresent
        name: web-server
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 8082
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /actuator/health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 1Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config
          name: <namespace>-config
      dnsPolicy: ClusterFirst
      hostAliases:
      - hostnames:
        - <url>
        ip: 10.0.1.178
      imagePullSecrets:
      - name: registry-pull-secret
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: <namespace>-config
        name: <namespace>-config
status:
  conditions:
  - lastTransitionTime: "2021-06-22T07:07:19Z"
    lastUpdateTime: "2021-06-22T07:07:19Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2021-06-22T07:17:20Z"
    lastUpdateTime: "2021-06-22T07:17:20Z"
    message: ReplicaSet "web-server-6df6d6565b" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

In the same namespace, I have more Service and Deployment resources, all of them work, except this one (+ another, see below).

# kubectl get endpoints -n <namespace>
NAME                 ENDPOINTS                                                          AGE
activemq             10.233.64.3:61613,10.233.64.3:8161,10.233.64.3:61616 + 1 more...   26d
content-backend      10.233.96.17:8080                                                  26d
datastore3           10.233.96.16:8080                                                  26d
web-server                                                                              74m
web-server-metrics                                                                      26d

As you can see, the selector/label are the same (web-server) in the Service as well as in the Deployment definition.

C-nan
  • 312
  • 2
  • 10
  • Based on some past issues: check that the service and deployment are on the same namespace. Check that there is really pods created by your deployment since the status of your deployment is reporting that something is wrong: `MinimumReplicasUnavailable` – Arnaud Develay Jun 22 '21 at 08:44
  • Thanks, I found the issue. The Pod was started, but not in Ready state due to a failing readinessProbe. I wasn't aware that an endpoint wouldn't be created until the Pod is in Ready state. Removing the readinessProbe created the Endpoint. Now I have to figure out what is wrong with the Pod itself. Thanks again! – C-nan Jun 22 '21 at 10:12
  • Could you post your solution as an answer? – Mikołaj Głodziak Jun 22 '21 at 10:26

2 Answers2

2

C-Nan has solved the problem, and has posted a solution as a comment:

I found the issue. The Pod was started, but not in Ready state due to a failing readinessProbe. I wasn't aware that an endpoint wouldn't be created until the Pod is in Ready state. Removing the readinessProbe created the Endpoint.

Mikołaj Głodziak
  • 4,775
  • 7
  • 28
1

From the status of your Deployment, it seems that no pod is running for the Deployment.

status:
  conditions:
  - lastTransitionTime: "2021-06-22T07:07:19Z"
    lastUpdateTime: "2021-06-22T07:07:19Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2021-06-22T07:17:20Z"
    lastUpdateTime: "2021-06-22T07:17:20Z"
    message: ReplicaSet "web-server-6df6d6565b" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

The unavailableReplicas: 1 filed is indicating that the desired pod is not available. As a result the Service has no active endpoint.

You can describe the deployment to see why the pod is unavailable.

Emruz Hossain
  • 4,764
  • 18
  • 26