Istio Ingress resulting in "no healthy upstream"

Question

I am using deploying an outward facing service, that is exposed behind a nodeport and then an istio ingress. The deployment is using manual sidecar injection. Once the deployment, nodeport and ingress are running, I can make a request to the istio ingress.

For some unkown reason, the request does not route through to my deployment and instead displays the text "no healthy upstream". Why is this, and what is causing it?

I can see in the http response that the status code is 503 (Service Unavailable) and the server is "envoy". The deployment is functioning as I can map a port forward to it and everything works as expected.

It seems to me that the App container is not accepting the request.could you check the logs of the App container in your pod ? — Suresh Vishnoi, Dec 06 '17 at 00:38
@SureshVishnoi The last log to be printed is `[warning][main] external/envoy/source/server/drain_manager_impl.cc:62] shutting down parent after drain` And there is an error two lines before that stating `"GET / HTTP/1.1" 404 NR 0 0 0 - "-" "google-api-nodejs-client/0.10.0"` — Pegladon, Dec 06 '17 at 05:08
what does parent mean here? If its the Main App container then its getting restarted. you can check logs of main container by using following command `kubectl logs POD_NAME -c CONTAINER_NAME`. Check if its restarting or terminating — Suresh Vishnoi, Dec 06 '17 at 08:38

score 5 · Answer 1 · answered May 30 '20 at 22:02

Just in case, like me, you get curious... Even though in my scenario it was clear the case of the error...

Error cause: I had two versions of the same service (v1 and v2), and an Istio VirtualService configured with traffic route destination using weights. Then, 95% goes to v1 and 5% goes to v2. As I didn't have the v1 deployed (yet), of course, the error "503 - no healthy upstream" shows up 95% of the requests.

Ok, even so, I knew the problem and how to fix it (just deploy v1), I was wondering... But, how can I have more information about this error? How could I get a deeper analysis of this error to find out what was happening?

This is a way of investigating using the configuration command line utility of Istio, the istioctl:

# 1) Check the proxies status -->
  $ istioctl proxy-status
# Result -->
  NAME                                                   CDS        LDS        EDS        RDS          PILOT                       VERSION
  ...
  teachstore-course-v1-74f965bd84-8lmnf.development      SYNCED     SYNCED     SYNCED     SYNCED       istiod-86798869b8-bqw7c     1.5.0
  ...
  ...

# 2) Get the name outbound from JSON result using the proxy (service with the problem) -->
  $ istioctl proxy-config cluster teachstore-course-v1-74f965bd84-8lmnf.development --fqdn teachstore-student.development.svc.cluster.local -o json
# 2) If you have jq install locally (only what we need, already extracted) -->
  $ istioctl proxy-config cluster teachstore-course-v1-74f965bd84-8lmnf.development --fqdn teachstore-course.development.svc.cluster.local -o json | jq -r .[].name
# Result -->
  outbound|80||teachstore-course.development.svc.cluster.local
  inbound|80|9180-tcp|teachstore-course.development.svc.cluster.local
  outbound|80|v1|teachstore-course.development.svc.cluster.local
  outbound|80|v2|teachstore-course.development.svc.cluster.local

# 3) Check the endpoints of "outbound|80|v2|teachstore-course..." using v1 proxy -->
  $ istioctl proxy-config endpoints teachstore-course-v1-74f965bd84-8lmnf.development --cluster "outbound|80|v2|teachstore-course.development.svc.cluster.local"
# Result (the v2, 5% of the traffic route is ok, there are healthy targets) -->
  ENDPOINT             STATUS      OUTLIER CHECK     CLUSTER
  172.17.0.28:9180     HEALTHY     OK                outbound|80|v2|teachstore-course.development.svc.cluster.local
  172.17.0.29:9180     HEALTHY     OK                outbound|80|v2|teachstore-course.development.svc.cluster.local

# 4) However, for the v1 version "outbound|80|v1|teachstore-course..." -->
$ istioctl proxy-config endpoints teachstore-course-v1-74f965bd84-8lmnf.development --cluster "outbound|80|v1|teachstore-course.development.svc.cluster.local"
  ENDPOINT             STATUS      OUTLIER CHECK     CLUSTER
# Nothing! Emtpy, no Pods, that's explain the "no healthy upstream" 95% of time.

score 3 · Accepted Answer · answered Sep 10 '18 at 04:23

3

Although this is a somewhat general error resulting from a routing issue within an improper Istio setup, I will provide a general solution/piece of advice to anyone coming across the same issue.

In my case the issue was due to incorrect route rule configuration, the Kubernetes native services were functioning however the Istio routing rules were incorrectly configured so Istio could not route from the ingress into the service.

answered Sep 10 '18 at 04:23

Pegladon

503
1
3
12

is there any workaround to solve this problem? I cannot find log regarding this issue in related pods, i've tried to restart those pods but still cant works – HQM Feb 09 '19 at 08:55
I am facing similar issue. Would you mind sharing more details (probably yaml etc) on how you fixed the issue please? That'd be very helpful. – Biju Jun 14 '21 at 03:09

score 2 · Answer 3 · answered May 29 '19 at 12:27

2

I faced the issue, when I my pod was in ContainerCreating state. So, it resulted in 503 error. Also as @pegaldon, explained it can also occur due to incorrect route configuration or there are no gateways created by user.

answered May 29 '19 at 12:27

Malathi

2,119
15
40

仲夏叶 · Answer 4 · 2019-11-20T11:51:04.740

delete destinationrules.networking.istio.io and recreate the virtualservice.networking.istio.io

[root@10-20-10-110 ~]# curl http://dprovider.example.com:31400/dw/provider/beat
no healthy upstream[root@10-20-10-110 ~]# 
[root@10-20-10-110 ~]# curl http://10.210.11.221:10100/dw/provider/beat
"该服务节点  10.210.11.221  心跳正常!"[root@10-20-10-110 ~]# 
[root@10-20-10-110 ~]# 
[root@10-20-10-110 ~]# cat /home/example_service_yaml/vs/dw-provider-service.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: dw-provider-service
  namespace: example
spec:
  hosts:
  - "dprovider.example.com"
  gateways:
  - example-gateway
  http:
  - route:
    - destination:
        host: dw-provider-service 
        port:
          number: 10100
        subset: "v1-0-0"
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: dw-provider-service
  namespace: example
spec:
  host: dw-provider-service
  subsets:
  - name: "v1-0-0"
    labels:
      version: 1.0.0

[root@10-20-10-110 ~]# vi /home/example_service_yaml/vs/dw-provider-service.yaml 
[root@10-20-10-110 ~]# kubectl -n example get vs -o wide | grep dw                       
dw-collection-service    [example-gateway]   [dw.collection.example.com]                       72d
dw-platform-service      [example-gateway]   [dplatform.example.com]                           81d
dw-provider-service      [example-gateway]   [dprovider.example.com]                           21m
dw-sync-service          [example-gateway]   [dw-sync-service dsync.example.com]               34d
[root@10-20-10-110 ~]# kubectl -n example delete vs dw-provider-service 
virtualservice.networking.istio.io "dw-provider-service" deleted
[root@10-20-10-110 ~]# kubectl -n example delete d dw-provider-service   
daemonsets.apps                       deniers.config.istio.io               deployments.extensions                dogstatsds.config.istio.io            
daemonsets.extensions                 deployments.apps                      destinationrules.networking.istio.io  
[root@10-20-10-110 ~]# kubectl -n example delete destinationrules.networking.istio.io dw-provider-service 
destinationrule.networking.istio.io "dw-provider-service" deleted
[root@10-20-10-110 ~]# kubectl apply -f /home/example_service_yaml/vs/dw-provider-service.yaml 
virtualservice.networking.istio.io/dw-provider-service created
[root@10-20-10-110 ~]# curl http://dprovider.example.com:31400/dw/provider/beat
"该服务节点  10.210.11.221  心跳正常!"[root@10-20-10-110 ~]# 
[root@10-20-10-110 ~]#

John Bob Joe · Answer 5 · 2022-05-18T07:14:31.193

From my experience, the "no healthy upstream" error can have different causes. Usually, Istio has received ingress traffic that should be forwarded (the client request, or Istio downstream), but the destination is unavailable (istio upstream / kubernetes service). This results in a HTTP 503 "no healthy upstream" error.

1.) Broken Virtualservice definitions If you have a destination in your VirtualService context where the traffic should be routed, ensure this destination exists (in terms of the hostname is correct, or the service is available from this namespace)

2.) ImagePullBack / Terminating / Service is not available

Ensure your destination is available in general. Sometimes no pod is available, so no upstream will be available too.

3.) ServiceEntry - same destination in 2 lists, but lists with different DNS Rules

Check your namespace for ServiceEntry objects with:

kubectl -n <namespace> get serviceentry

If the result has more than one entry (multiple lines in one ServiceEntry object), check if a destination address (e.g. foo.com) is available in various lines. If the same destination address (e.g. foo.com) is available in various lines, ensure that the column "DNS" does not have different resolution settings (e.g. one line uses DNS, the other line has NONE). If yes, this is an indicator that you try to apply different DNS settings to the same destination address.

A solution is:

a) to unify the DNS setting, setting all lines to NONE or DNS, but not to mix it up.

b) Ensure the destination (foo.com) is available in one line, and a collision of different DNS rules does not appear.

a) involves restarting istio-ingressgateway pods (data plane) to make it work.

b) Involves no restart of istio data or istio control plane.

Basically: It helps to check the status between Control Plane (istiod) and DatapPlane (istio-ingressgateway) with

istioctl proxy-status

The output of istioctl proxy-status should ensure that the columns say "SYNC" this ensures that the control plane and Data Plane are synced. If not, you can restart the istio-ingressgateway deployment or the istiod daemonset, to force "fresh" processes.

Further, it helped to run

istioctl analyze -A

to ensure that targets are checked in the VirtualService context and do exist. If a virtual service definition exists with routing definitions whose destination is unavailable, istioctl analyze -A can detect these unavailable destinations.

Furthermore, reading the logfiles of the istiod container helps. The istiod error messages often indicate the context of the error in the routing (which namespace and service or istio setting). You can use the default way with

kubectl -n istio-system logs <nameOfIstioDPod>

Referenes:

Istio Ingress resulting in "no healthy upstream"

5 Answers5