NGINX Reverse Proxy + ngx_upstream_resolveMK - Trying to resolve SRV from ECS Service Discovery Route53 Auto Naming

Question

I'm currently having an issue with ECS Service Discovery and Route53 Auto Naming.

I have added the Service Registry to the service and all the Hosted Zones records are being populated automatically. But I cannot seem to work out how to resolve the DNS SRV records with NGINX + ngx_upstream_resolveMK.

# DNS RESOLVER
resolver ns-x.awsdns-xx.com valid=10s;

# UPSTREAMS
upstream kibana {
  resolveMK servicediscovery.ecs service=kibana;
}

# HOST - kibana.example.com
server {
  server_name kibana.example.com;
  listen 80;
  location / {
    proxy_pass https://kibana/;
    rewrite ^/(.*)$ /$1 break;
  }
}

ERROR: nginx: [emerg] host not found in upstream "servicediscovery.ecs" in /usr/local/nginx/sites-enabled/kibana.conf:3

So it appears that there is a missing A record needed to resolve "servicediscovery.ecs" to the Route 53 Private Zone.

Do I need to manually add this? or is there a way to dynamically add this A record?

I think this is the cause of the problem, but I'm still learning and might be way off.

UPDATE:

I read you can also use xxx.xxx.xxx.2 to access the DNS via AWS VPC I have tested using a new resolver without much luck.

# DNS RESOLVER
resolver xxx.xxx.0.2 valid=10s;

score 2 · Accepted Answer · edited Oct 15 '19 at 11:02

Nginx has a problem using internal DNS when you want use it in ECS.

I was successful after using HA-Proxy.

It uses the "links" option of Docker, and HA-Proxy support uses /etc/hosts file.

First, use "links" option of Docker and setting "environment variables" (eg. LINK_APP, LINK_PORT).
Second, fill these "environment variables" into haproxy.cfg.

Also, I recommend you use "dynamic port mapping" to ALB. it makes more flexible works.

taskdef.json:

{
    "executionRoleArn": "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<APP_NAME>_ecsTaskExecutionRole",
    "containerDefinitions": [
      {
        "name": "<APP_NAME>-rp",
        "image": "gnokoheat/ecs-reverse-proxy:latest",
        "essential": true,
        "memoryReservation": <MEMORY_RESV>,
        "portMappings": [
          {
            "hostPort": 0,
            "containerPort": 80,
            "protocol": "tcp"
          }
        ],
        "links": [
          "<APP_NAME>"
        ],
        "environment": [
          {
            "name": "LINK_PORT",
            "value": "<SERVICE_PORT>"
          },
          {
            "name": "LINK_APP",
            "value": "<APP_NAME>"
          }
        ]
      },
      {
        "name": "<APP_NAME>",
        "image": "<IMAGE_NAME>",
        "essential": true,
        "memoryReservation": <MEMORY_RESV>,
        "portMappings": [
          {
            "protocol": "tcp",
            "containerPort": <SERVICE_PORT>
          }
        ],
        "environment": [
          {
            "name": "PORT",
            "value": "<SERVICE_PORT>"
          },
          {
            "name": "APP_NAME",
            "value": "<APP_NAME>"
          }
        ]
      }
    ],
    "requiresCompatibilities": [
      "EC2"
    ],
    "networkMode": "bridge",
    "family": "<APP_NAME>"
  }

haproxy.cfg:


global
    daemon
    pidfile /var/run/haproxy.pid

defaults
    log global
    mode http
    retries 3
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend http
    bind *:80

    http-request set-header X-Forwarded-Host %[req.hdr(Host)]

    compression algo gzip
    compression type text/css text/javascript text/plain application/json application/xml

    default_backend app

backend app
    server static "${LINK_APP}":"${LINK_PORT}"

See :

Github : https://github.com/gnokoheat/ecs-reverse-proxy

Docker image : gnokoheat/ecs-reverse-proxy:latest

Here is more detail a solution!

Awesome response on this. I have changed the accepted answer to be yours with your detailed solution. — Adan Rehtla, Nov 08 '19 at 02:54

score 1 · Answer 2 · answered Mar 22 '19 at 03:38

We resolved this issue by swapping to HAProxy.

Another issue was to use the correct format for the Service Dsicovery.

Create a new private namespace with the name _ip.xxxxxxx.ecs and name the service discovery name _prometheus. Both can be achieved when you create a new service in ecs.

NGINX Reverse Proxy + ngx_upstream_resolveMK - Trying to resolve SRV from ECS Service Discovery Route53 Auto Naming

2 Answers2