kube service domain name not working, but clusterIP does work with Jupyter Enterprise Gateway

Question

I have a Jupyter notebook setup in the jupyter namespace on a kubernetes cluster, and Jupyter Enterprise Gateway setup in the enterprise-gateway namespace as a Service in the same cluster.

If I configure the notebook to connect to the enterprise-gateway service using the clusterIP it works fine.

--gateway-url=http://172.20.186.249:8888

but if I switch to using the service domain name the notebook receives a 503 Connection Refused error

--gateway-url=http://enterprise-gateway.enterprise-gateway.svc.cluster.local:8888

When I use busybox check to check the kubernetes dns, the domain resolves as expected.

kubectl -n default exec -ti busybox nslookup enterprise-gateway.enterprise-gateway
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Server:    172.20.0.10
Address 1: 172.20.0.10 kube-dns.kube-system.svc.cluster.local

Name:      enterprise-gateway.enterprise-gateway
Address 1: 172.20.186.249 enterprise-gateway.enterprise-gateway.svc.cluster.local

How do I get the domain name to work?

The Service config for the JEG looks like this...

kubectl describe svc enterprise-gateway --namespace enterprise-gateway
Name:                     enterprise-gateway
Namespace:                enterprise-gateway
Labels:                   app=enterprise-gateway
                          app.kubernetes.io/managed-by=Helm
                          chart=enterprise-gateway-2.6.0
                          component=enterprise-gateway
                          heritage=Helm
                          release=enterprise-gateway
Annotations:              meta.helm.sh/release-name: enterprise-gateway
                          meta.helm.sh/release-namespace: enterprise-gateway
Selector:                 app=enterprise-gateway
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.20.250.15
IPs:                      172.20.250.15
Port:                     http  8888/TCP
TargetPort:               8888/TCP
NodePort:                 http  31366/TCP
Endpoints:                10.1.16.136:8888,10.1.2.228:8888,10.1.30.90:8888
Port:                     response  8877/TCP
TargetPort:               8877/TCP
NodePort:                 response  31201/TCP
Endpoints:                10.1.16.136:8877,10.1.2.228:8877,10.1.30.90:8877
Session Affinity:         ClientIP
External Traffic Policy:  Cluster
Events:                   <none>

take a look at https://jupyter-enterprise-gateway.readthedocs.io/en/v2.0.0/kernel-kubernetes.html#setting-up-a-kubernetes-ingress-for-use-with-enterprise-gateway if you need further info im up to assist you — jmvcollaborator, Aug 31 '22 at 02:41
@jmvcollaborator thanks for looking into this. An Ingress is for exposing a Service to clients external to the cluster. That is not what I am doing. I have a Service that an application in the same cluster needs to connect to. A Service should be sufficient for this https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#creating-a-service — user3188040, Aug 31 '22 at 14:06
As mentioned in the question the accessing the Service via the clusterIP does work, it is access via the service name in the DNS that is not working. — user3188040, Aug 31 '22 at 14:11
clusterip is for local access only ip:port you should add maybe a load balancer service and link the external ip to the DNS — jmvcollaborator, Aug 31 '22 at 15:09
I am doing local access. The client application is in a Pod in the same cluster, that is why using clusterIP works. According to this page the application should also be able to use the service dns which resolves to the clusterIP. https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ — user3188040, Sep 01 '22 at 00:27

jmvcollaborator · Answer 1 · 2022-09-01T07:58:52.993

Ok, i dont know where to start i have a bunch of findings. I will start with the eye catcher one, i have a working test project i can share later on and i have to elaborate more in this answer if needed.

Step1

1- I see a mismatch on your IPs. The DNS lookup did not resolved the service DNS to the correct IP. Address 1: 172.20.186.249 is different than IP: 172.20.250.15

To debug DNS:

kubectl exec "YOURPODNAME" cat /etc/resolv.conf
Verify that a search path and a name server are set up correctly

nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5
check if the kubedns/coredns pods are running

kubectl get pods --namespace=kube-system NAME READY STATUS RESTARTS AGE .... kube-dns-86f4d74b45-2qkfd 3/3 Running 232 133d kube-proxy-b2frq 1/1 Running 0 15m ...

If the pod is running, there might be something wrong with the global DNS service

kubectl get svc --namespace=kube-system
NAME      TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE kube-dns ClusterIP   10.96.0.10       <none>   53/UDP,53/TCP

You might also need to check whether DNS endpoints are exposed:

kubectl get ep kube-dns --namespace=kube-system
NAME       ENDPOINTS                     AGE
kube-dns   172.17.0.5:53,172.17.0.5:53   133d

These debugging actions will usually indicate the problem with your DNS configuration, or it will simply show you that a DNS add-on should be enabled in your cluster configuration.

Step 2

When using busybox to check the kubernetes dns This seems incorrect when looking Address 1: 172.20.186.249 im expecting to get an IP 10.X.X.X Install dnsutils on the pod as pointed below

kubectl exec --stdin --tty "YOURPODNAME" -- apt update && sudo apt-get -y install dnsutils
kubectl exec -it "YOURPODNAME" -- /bin/bash
Inside the pod and again (weird) run apt-get install dnsutils
Stay inside the pod and run nslookup "YOURSERVICENAME" you will get an IP and a Name(DNS). Check this IP since it needs to match with the IP of the service description.
kubectl describe svc "YOURSERVICENAME", the IP should be the same as #4
What you must see:

Step 3

Once you have Step #2 solved you will be able to use the service name(FQDN) returned in Step 2 item #4

To be continued...

thanks, I'll allocate the bounty to you. I've been occupied with some higher priority tasks, so will be coming back to this issue soon — user3188040, Sep 07 '22 at 11:33

kube service domain name not working, but clusterIP does work with Jupyter Enterprise Gateway

1 Answers1