1

ENV:
k8s: v1.20.5
ingress-nginx: v1.6.4

I created ingress-nginx-controller from offical yaml: https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.6.4/deploy/static/provider/baremetal/deploy.yaml

and I changed the network type to hostnetwork:

hostNetwork: true

Then I created a deployment to create a backend server.Below is the yaml file:

apiVersion: v1
kind: Service
metadata:
  name: http-svc
spec:
  selector:
    app: http
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-deployment
  labels:
    app: http
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http
  template:
    metadata:
      labels:
        app: http
    spec:
      containers:
      - name: http
        image: hashicorp/http-echo:alpine
        args: ["-text", "hello", "-listen=:80"]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
spec:
  rules:
  - host: "test.com"
    http:
      paths:
      - pathType: Prefix
        path: "/test"
        backend:
          service:
            name: http-svc
            port:
              number: 80

Every things looks running fine, but I still got "500 Internal Server Error" when I access the web server via ingress-nginx.

Below is the info for the resources:

#kubectl describe ingress
Name:             test-ingress
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  test.com    
              /test   http-svc:80 (192.168.107.203:80,192.168.122.81:80)
Annotations:  <none>
Events:       <none>

#kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
http-svc     ClusterIP   10.100.58.107   <none>        80/TCP    48m
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   283d

#kubectl get pods -owide
NAME                               READY   STATUS    RESTARTS   AGE   IP                NODE        NOMINATED NODE   READINESS GATES
http-deployment-764c4597c5-rdks7   1/1     Running   0          48m   192.168.122.81    k8s-node4   <none>           <none>
http-deployment-764c4597c5-rf99t   1/1     Running   0          48m   192.168.107.203   k8s-node3   <none>           <none>

#kubectl get pods -n ingress-nginx -owide
NAME                                        READY   STATUS      RESTARTS   AGE   IP                NODE        NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-kb64z        0/1     Completed   0          44m   192.168.107.204   k8s-node3   <none>           <none>
ingress-nginx-admission-patch-xmswb         0/1     Completed   1          44m   192.168.122.82    k8s-node4   <none>           <none>
ingress-nginx-controller-69695968f9-7dtxf   1/1     Running     0          44m   10.1.1.12         k8s-node2   <none>           <none>

I can directly acces my backend server via service ip, so I think the problem maybe is ingress-nginx can not forward the request to the backend?

Below is the log in ingress-nginx-controller:

2023/03/19 07:31:54 [error] 25#25: *25199 could not find named location "@custom_upstream-default-backend_404", client: 127.0.0.1, server: , request: "GET /test HTTP/1.1", host: "test.com"
2023/03/19 07:31:54 [error] 26#26: *25201 could not find named location "@custom_upstream-default-backend_404", client: 127.0.0.1, server: , request: "GET / HTTP/1.0", host: "test.com"
10.1.1.11 - - [19/Mar/2023:07:31:54 +0000] "GET /test HTTP/1.1" 500 170 "-" "curl/7.29.0" 76 0.000 [upstream-default-backend] [] 127.0.0.1:8181 : 127.0.0.1:8181 0 : 170 0.000 : 0.000 500 : 500 28acaef695f43cac09e7dfc932511c92

Looks like it forwarded the request to localhost,but why? Did I miss something?

Below is the different respones from different way to access:

#curl http://10.100.58.107
hello
#curl http://test.com/test
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>

I want to access the backend server via ingress-nginx successfully.

Peter Pan
  • 11
  • 2
  • did you use minikube or some other tool to setup the cluster? Check this link as well - https://stackoverflow.com/questions/63558461/endpoints-default-http-backend-not-found-in-ingress-resource?rq=1 – Sibtain Mar 19 '23 at 12:22
  • I created cluster by kube admin – Peter Pan Mar 19 '23 at 12:30
  • can you update your question to include the `Service` created for ingress controller in `ingress-nginx` namespace? – Sibtain Mar 19 '23 at 12:50
  • A point to check would be a compatibility matrix between K8S and nginx-ingress. Your nginx-ingress supports K8S 1.23 and up. Maybe try with an older version of your ingress controller. – Dawid Kruk Mar 19 '23 at 16:06

1 Answers1

1

Debug pointers:

  • Application Level
    • Check if you have a /test endpoint in your service what does it respond with. Because based on your test requests, with the IP you made request to / and with DNS you made a request to /test
  • K8s Service Level
    • Port forward the service and try to run it on your local machine. 
      kubectl port-forward svc/http-svc 80:80
  • Ingress Level
    • Is the error thrown by Nginx? If yes, then nginx ingress controller is throwing the error.
    • Check if you have used the proper ingress class name in your ingress object based on what you configured while deploying your nginx ingress controller.
    • As visible in your logs it says [upstream-default-backend] , that I feel based on my understanding once the controller receives the request it doesn't have any configured server configuration for this host.
      • This happens because nginx ingress controller only adds the server configuration of the hosts that happens to have an ingress class in their ingress object.
Shubham Vaishnav
  • 1,637
  • 6
  • 18