0

I have number of statefulsets PODs and they are not replica of each other. My application register service on the hostname of the pod and other services try to reach the service based on hostname. It works fine with headless service but application logic cannot be changed as it reads the hostname of POD and unfortunately I have to use that.

I have following scenario:

[test@shard1-0 ~]$ nslookup shard2
Server:     10.96.5.5
Address:    10.96.5.5#53

Name:   shard2.default.svc.cluster.local
Address: 10.244.0.50

[test@shard1-0 ~]$ nslookup shard2-0
Server:     10.96.5.5
Address:    10.96.5.5#53

** server can't find shard2-0: NXDOMAIN

[test@shard1-0 ~]$

Is there a way that shards can resolve using the hostname of POD name ?and can reach each other as application is registered based on pod hostname.

Just to clarify if we are able to resolve this problem, we also need to expose these pods externally. I think for that I can use that and hopefully it will work: How to set hostname for kubernetes pod in statefulset

drifter
  • 389
  • 1
  • 5
  • 17
  • can you not modify your application to call podName.serviceName? ie shard2-0.shard2? – Patrick W Nov 13 '19 at 01:09
  • I tried this and it seems my application code need some changes to enforce hostname.svcname as by default it read hostname. – drifter Nov 14 '19 at 04:55

1 Answers1

0

Since this is directly related to the DNS resolution within the cluster, one possible approach for it would be to change the DNS policy in the pod/deployment of your application that needs to resolve hostnames.

You can change the dnsPolicy of a pod to set up your custom DNS settings, in which the records of each stateful set pod can be included.

  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 1.2.3.4        
    options:
      - name: ndots
        value: "0"

This solves the issue of non-existent records in the current DNS records within the cluster and allows you to control the unqualified domain names associated with your pods (web-0, web-1 ) by removing the DNS suffixes.

Additionally, is safer since the change is only applied at your pod/deployment instead of the whole cluster, which keeps you out of potential trouble when exposing your stateful pods afterwards.

Finally and regarding the DNS configuration, you can use ALIAS records pointed to the pod's FQDNs to preserve network identity through their original names. This avoids the operational cost of fetching the IP addresses since you'll be using the hostnames (they never change) instead.

Host:   Type:   Points to:                                  TTL
web-0   ALIAS   web-0.web-svc.namespace.svc.cluster.local   1 Hour

Please note that this approach requires that you setup and maintain your own DNS server (represented as 1.2.3.4 in the definition above).

If in your case is impossible to change the application logic to use the DNS standard within the cluster, this might be a workaround.

yyyyahir
  • 2,262
  • 1
  • 5
  • 14
  • This solution looks good as custom DNS gives lot of benefits. However, I donno how to change IPs in DNS for stateful set? This will be lot of work to change IPs on every new scheduling of stateful pod. I am not sure if I can enforce the IP on statefulset so that I can maintain the network identify. – drifter Nov 13 '19 at 00:53
  • I was thinking and you might use ALIAS records to "forward" requests to the original hostnames to avoid fetching the IP addresses every time. Updated my answer to reflect this. – yyyyahir Nov 13 '19 at 13:57
  • Thanks for pointing to this. Let me try this and will update you if I see any issue. – drifter Nov 14 '19 at 04:04
  • did you eventually try this? – Black_Bacardi Nov 25 '19 at 12:23