3

I'm having issues when accessing a service present in another namespace.

I have 2 namespaces (in the same cluster) airflow-dev and dask-dev.

enter image description here

In dask-dev namespace, I have dask cluster(dask scheduler and workers) deployed. Also, created a service (cluster IP) to dask-scheduler pod. I'm able to access dask-scheduler pod from chrome using 'kubectl port-forward' command.

kubectl port-forward --namespace dask-dev svc/dask-dev-scheduler 5002:80

However, am not able to access the service (or dask-scheduler pod) from a pod (airflow-scheduler) present in airflow-dev namespace. Getting 'Host or service not found' error when trying to access it using the below

dask-dev-scheduler.dask-dev.svc.cluster.local:8786

Below is the service that I have created for dask-dev-scheduler. Could you please let me know how to access the service from airflow-dev namespace.

apiVersion: v1
metadata:
  name: dask-dev-scheduler
  namespace: dask-dev
  labels:
    app: dask-dev
    app.kubernetes.io/managed-by: Helm
    chart: dask-dev-4.5.7
    component: scheduler
    heritage: Helm
    release: dask-dev
  annotations:
    meta.helm.sh/release-name: dask-dev
    meta.helm.sh/release-namespace: dask-dev
spec:
  ports:
    - name: dask-dev-scheduler
      protocol: TCP
      port: 8786
      targetPort: 8786
    - name: dask-dev-webui
      protocol: TCP
      port: 80
      targetPort: 8787
  selector:
    app: dask-dev
    component: scheduler
    release: dask-dev
  clusterIP: 10.0.249.111
  type: ClusterIP
  sessionAffinity: None
status:
  loadBalancer: {}

[1]: https://i.stack.imgur.com/UrA7u.jpg
Kiran
  • 451
  • 1
  • 6
  • 23
  • 1
    You can access the service using [servicename.namespace:PORT] i.e. dask-dev-scheduler.dask-dev:5002 Read more about it in the docs [here] (https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-services/) This'll work if you have DNS on the cluster, according to the doc here: "You can (and almost always should) set up a DNS service for your Kubernetes cluster using an add-on." This'll make it very easy for you. You can also try to use the IP address of the service to test, kubectl get svc -o wide will show you the IP. – samceena May 19 '21 at 01:45
  • Tried this but getting the below error OSError: Timed out trying to connect to tcp://dask-dev-scheduler.dask-dev:8786 after 10 s – Kiran May 19 '21 at 05:07
  • Similar question already exists. See if it solves your problem. [Here](https://stackoverflow.com/questions/37221483/service-located-in-another-namespace) –  May 19 '21 at 07:02

3 Answers3

3

You can use a local service to reference an external service (a service in a different namespace) using the service externalName Type.

ExternalName services do not have selectors, or any defined ports or endpoints, therefore, you can use an ExternalName service to direct traffic to an external service.

apiVersion: v1
kind: Service
metadata:
  name: service-b
  namespace: namespace-b
spec:
  selector:
    app: my-app-b
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000

apiVersion: v1
kind: Service
metadata:
  name: service-b-ref
  namespace: namespace-a
spec:
  type: ExternalName
  externalName: service-b.namespace-b.svc.cluster.local

Any traffic in namespace-a that connects to service-b-ref:<port> will be routed to service-b in namespace-b (service-b.namespace-b.svc.cluster.local) Therefore, a call to service-b-ref:3000 will route to our service-b.

In your example, you'd just need to create a service in airflow-dev that will route traffic to the dask-dev-scheduler in the dask-dev namespace:

apiVersion: v1
kind: Service
metadata:
  name: dask-dev-svc
  namespace: airflow-dev
spec:
  type: ExternalName
  externalName: dask-dev-scheduler.dask-dev.svc.cluster.local

Therefore, all airflow-dev resources that need to connect to the dask-dev-scheduler would call: dask-dev-svc:8786

apiVersion: v1
metadata:
  name: dask-dev-scheduler
  namespace: dask-dev
spec:
  ports:
    - name: dask-dev-scheduler
      protocol: TCP
      port: 8786
      targetPort: 8786
# ...
  selector:
    app: dask-dev
Highway of Life
  • 22,803
  • 16
  • 52
  • 80
1

The cluster domain doesn't always have to be cluster.local. Try just using dask-dev-scheduler.dask-dev.svc. Assuming Airflow respects the ndots lookup strategy in the generated resolv.conf mounted into the pod, that should find it.

coderanger
  • 52,400
  • 4
  • 52
  • 75
  • I have tried this approach as well..still getting the same issue OSError: Timed out trying to connect to tcp://dask-dev-scheduler.dask-dev.svc:8786 after 10 s – Kiran May 19 '21 at 05:08
0

Are you upgrading coredns to 1.8.3? I have just faced same issue with my eks cluster after upgrading from 1.19 to 1.20. Downgrading coredns to 1.8.0 solved the issue.

HNQ
  • 110
  • 2
  • 9