nginx won't resolve hostname in K8S

Question

So, I would like to have nginx resolve hostnames for backends at request time. I expect to get HTTP 502 Bad Gateway when back-end service is down and I expect service response, when it's up.

I use nginx:1.15-alpine image for nginx and here is what I have in it's config:

server {

  resolver kube-dns.kube-system.svc.cluster.local valid=5s;

  server_name  mysystem.com;
  listen       80;

  client_max_body_size 20M;

  location = /nginx_status {
      stub_status on;
      access_log off;
  }

  # Services configuration

  location ~ /my-service/ {
      set $service_endpoint http://my-service.namespace:8080;
      proxy_pass $service_endpoint$request_uri;
      include includes/defaults-inc.conf;
      include includes/proxy-inc.conf;
  }

}

So, when I make the request to the nginx, I get 502 Bad Gateway response. Nginx's log say the name is not found:

2018/06/28 19:49:18 [error] 7#7: *1 my-service.namespace could not be resolved (3: Host not found), client: 10.44.0.1, server: mysystem.com, request: "GET /my-service/version HTTP/1.1", host: "35.229.17.63:8080"

However, when I log into the container with shell (kubectl exec ... -- sh) and test the DNS resolution, it works perfectly.

 # nslookup my-service.namespace kube-dns.kube-system.svc.cluster.local
Server:    10.47.240.10
Address 1: 10.47.240.10 kube-dns.kube-system.svc.cluster.local

Name:      my-service.namespace
Address 1: 10.44.0.75 mysystem-namespace-mysystem-namespace-my-service-0.my-service.namespace.svc.cluster.local

Moreover, I can wget http://my-service.namespace:8080/ and get a response.

Why nginx cannot resolve the hostname?

Update: How I managed to resolve it:

In nginx.conf at the server level I have added a resolver setting:

resolver kube-dns.kube-system.svc.cluster.local valid=10s;

Then I used a FQDN in proxy_pass:

proxy_pass http://SERVICE-NAME.YOUR-NAMESPACE.svc.cluster.local:8080;

Answer 1

It fails because you need to use the FQDN to Resolve the name.

Using just the hostname will usually work because in kubernetes the resolv.conf is configured with search domains so that you don't usually need to provide a service's FQDN.

However, specifying the FQDN is necessary when you tell nginx to use a custom name server because it does not get the benefit of these domain search specs.

In nginx.conf added at the server level:

resolver kube-dns.kube-system.svc.cluster.local valid=10s;

Then used a FQDN in proxy_pass:

proxy_pass http://SERVICE-NAME.YOUR-NAMESPACE.svc.cluster.local:8080;