Kubernetes - Ingress-nginx routing error (Cannot connect frontend to backend)

Question

Need some serious help here ! Thanks in advance.

I am trying to deploy a microservice based Java application. I am able to get to the frontend service(webapp) on my browser, but I am unable to connect it with the backend (auth service) and hence it shows authentication failure.

HTML LOGIN FORM form points to "/login?referrerURL="

I checked the ingress nginx logs :

Service "default/auth-srv" does not have any active Endpoint.
Service "default/voice-srv" does not have any active Endpoint.
Service "default/reporting-srv" does not have any active Endpoint.
Service "default/webapp-srv" does not have any active Endpoint.

The ingress nginx config file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: ingress-service
    annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
    rules:
        - host: <domain_name>
          http:
            paths:
                - path: /auth/?(.*)
                  backend:
                    serviceName: auth-srv
                    servicePort: 8080
                - path: /emotion/?(.*)
                  backend:
                    serviceName: emotion-srv
                    servicePort: 8080
                - path: /storage/?(.*)
                  backend:
                    serviceName: storage-srv
                    servicePort: 8080
                - path: /voice/?(.*)
                  backend:
                    serviceName: voice-srv
                    servicePort: 8080
                - path: /backend/?(.*)
                  backend:
                    serviceName: backend-srv
                    servicePort: 8080
                - path: /reporting/?(.*)
                  backend:
                    serviceName: reporting-srv
                    servicePort: 8080
                ## frontend
                - path: /?(.*)
                  backend:
                    serviceName: webapp-srv
                    servicePort: 8080

How is the webapp(frontend) connecting to auth service internally?

Using the below endpoint

http://ingress-nginx-controller.ingress-nginx.svc.cluster.local/auth

Above endpoint obtained from: Using the pattern === http://name-of-service.namespace.svc.cluster.local

namespaces :

$ kubectl get namespace
NAME              STATUS   AGE
default           Active   10h
ingress-nginx     Active   10h
kube-node-lease   Active   10h
kube-public       Active   10h
kube-system       Active   10h

$ kubectl get service -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.100.99.130    <loadbalancer>.amazonaws.com   80:32794/TCP,443:30053/TCP   10h
ingress-nginx-controller-admission   ClusterIP      10.100.230.126   <none>                                                                          443/TCP

Webapp (frontend) pod logs :

2020-07-28 20:57:08.139  INFO 1 --- [io-8080-exec-10] com.symtrain.controller.AdminController  : Auth Controller User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36
2020-07-28 20:57:08.139  INFO 1 --- [io-8080-exec-10] com.symtrain.controller.AdminController  : Auth Controller URL: http://testprod.symtrain.com/index
2020-07-28 20:57:08.139  INFO 1 --- [io-8080-exec-10] com.symtrain.controller.AdminController  : Auth Controller flag:::::: Not IE
2020-07-28 20:57:08.139  INFO 1 --- [io-8080-exec-10] com.symtrain.controller.AdminController  : Auth Controller URL inside normal return:

Some extra information for deployments :

$ kubectl get deploy
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
auth-depl        2/2     2            2           4h40m
backend-depl     2/2     2            2           4h40m
emotion-depl     2/2     2            2           4h40m
reporting-depl   2/2     2            2           4h40m
storage-depl     2/2     2            2           4h40m
voice-depl       2/2     2            2           4h40m
webapp-depl      2/2     2            2           4h40m


$ kubectl get svc
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
auth-srv        ClusterIP   10.100.258.118   <none>        8080/TCP   4h41m
backend-srv     ClusterIP   10.100.132.251   <none>        8080/TCP   4h41m
emotion-srv     ClusterIP   10.100.32.154    <none>        8080/TCP   4h41m
kubernetes      ClusterIP   10.100.0.1       <none>        443/TCP    10h
reporting-srv   ClusterIP   10.100.64.80     <none>        8080/TCP   4h41m
storage-srv     ClusterIP   10.100.36.25     <none>        8080/TCP   4h41m
voice-srv       ClusterIP   10.100.212.180   <none>        8080/TCP   4h41m
webapp-srv      ClusterIP   10.100.21.170    <none>        8080/TCP   4h41m

Endpoints

kubectl get endpoints
NAME            ENDPOINTS                                 AGE
auth-srv        192.168.14.60:8080,192.168.44.116:8080    4h53m
backend-srv     192.168.32.14:8080,192.168.37.180:8080    4h53m
emotion-srv     192.168.58.110:8080,192.168.6.148:8080    4h53m
kubernetes      192.168.118.66:443,192.168.82.184:443     10h
reporting-srv   192.168.31.233:8080,192.168.33.218:8080   4h53m
storage-srv     192.168.23.217:8080,192.168.38.48:8080    4h53m
voice-srv       192.168.4.211:8080,192.168.59.186:8080    4h53m
webapp-srv      192.168.31.164:8080,192.168.62.143:8080   4h53m

Auth backend Deployment and Service :

$ kubectl describe deploy auth-depl
Name:                   auth-depl
Namespace:              default
CreationTimestamp:      Tue, 28 Jul 2020 16:32:44 +0000
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=auth
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=auth
  Containers:
   auth:
    Image:        <my_image_name>
    Port:         8080/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   auth-depl-787446c4db (2/2 replicas created)
Events:          <none>


#####################################

$ kubectl describe svc auth-srv
Name:              auth-srv
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=auth
Type:              ClusterIP
IP:                10.100.218.108
Port:              auth  8080/TCP
TargetPort:        8080/TCP
Endpoints:         192.168.14.60:8080,192.178.44.136:8080
Session Affinity:  None
Events:            <none>

NOTE: I am altering the IPs here for security purposes.

Can you run `kubectl get endpoints ` for one of the Services in default please. Also if you can describe that Service and the corresponding Deployment. Its possible that the Service is not selecting the labels that exist on the Deployment. — Serge, Jul 28 '20 at 21:24
Your Services are fine and the Endpoints exist. I believe the error you saw was from the past and is not an issue. See the comment to this answer, they experienced the same thing: https://stackoverflow.com/questions/51878195/kubernetes-cross-namespace-ingress-network/51899301#51899301. Can you clarify what happens when you try to hit the endpoint on the Ingress Controller Service? — Serge, Jul 28 '20 at 21:47
@KaranKumar, Out of curiosity, any reason why are you using ingress controller svc URL `http://ingress-nginx-controller.ingress-nginx.svc.cluster.local/auth`, instead of auth service URL `http://auth-srv` in the frontend webapp? — Abhishek, Jul 28 '20 at 23:35

DBSand · Answer 1 · 2020-07-29T05:54:53.910

As per the error messages, it could be that the labels you are using inside of the Service may be a cause for concern. Your service will lookup pods based on their pod labels.

Fetch the labels of pods of your deployments, example below (app: nginx).

kubectl get pods --show-labels

nginx     1/1       Running   0          16m      app=nginx

Edit you service yaml to match the labels of pods in your deployment, ensure they are pod labels.

    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx

test if you are able to use the Service, spin a temporary busy box pod with the same labels (app:nginx) and test using wget.

Kubectl run bb --image=busybox - it - - wget - o- auth-serv:8080

If the above does not solve your problem, you may also need to ensure that the containerPort defined inside of your deployment is matching with the service port you are exposing (8080 in your case)

controllers/nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 8080

if the above does not resolve the problem, then you might want to look at policies. Maybe, you could create a new ingress network policy policy based on your pod labels to ensure that traffic hits your pods. (https://kubernetes.io/docs/concepts/services-networking/network-policies/)

All the settings are correct. I double checked just to be sure. — Karan Kumar, Jul 29 '20 at 06:43
can you create a temp pod to access the service or does it timeout? @KaranKumar — DBSand, Jul 29 '20 at 06:54

Abhishek · Answer 2 · 2020-07-28T23:43:22.790

0

You are trying to hit /login?referrerURL= and this path is not defined in your ingress rules.

EDIT:

It's not common to use ingress service internally within the cluster. Ingress resource is designed to manage external access to internal services.

Note this is also a security concern as you are exposing the auth service (which is a backend service used by your UI layer) externally.

edited Jul 28 '20 at 23:43

answered Jul 28 '20 at 22:55

Abhishek

763
7
18

1

This rule should cover that - no? `- path: /?(.*)` – Serge Jul 28 '20 at 23:03
yes you are correct Serge. I missed the regex nginx rules. – Abhishek Jul 28 '20 at 23:26

score 0 · Answer 3 · answered Feb 15 '22 at 20:54

0

A quick solution for me was to delete the service and recreate it. This is only if all your configs that others have suggested are correct

answered Feb 15 '22 at 20:54

Sean

360
3
17

Kubernetes - Ingress-nginx routing error (Cannot connect frontend to backend)

3 Answers3

Linked