504 response from ngingx reverse proxy with nodejs + docker

Question

UPDATE

I have found out that it works if I restart nginx on the container with docker exec -it backend_gateway_1 nginx -s reload. I have found some information from this question. It seems that connecting to 127.0.0.1 rather than localhost would work, however I am using docker networks so I'm not sure what my situation would be.

ORIGINAL QUESTION

I am trying to use nginx as a reverse proxy with docker. I cannot quite work out what's wrong as sometimes it works when but then if I use docker-compose down && docker-compose up to restart it all, i will then usually get this 504 error

I have a feeling this is to do with the nginx or docker configuration. I am able to make a request to / fine but any request to /user is causing the intermittent issue.

When I am not receiving a 504, the node app is working well and creates/returns a user. I am also able to connect to the database fine with mongo-express so that should help narrow something down.

docker-compose.yml

version: '3.5'
services:
  gateway:
    image: nginx:latest
    restart: always
    ports:
      - 8080:80
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/nginx-error.log:/var/log/nginx/error.log
    networks:
      - poker_network
  poker_mongo:
    image: mongo:latest
    container_name: poker_mongo
    restart: unless-stopped
    volumes:
      - db-data:/data/db
    expose: 
      - "27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: test
      MONGO_INITDB_ROOT_PASSWORD: 123
    networks:
      - poker_network
    depends_on:
      - gateway
  poker_user:
    container_name: poker_user
    build: ./user/
    image: poker/user:latest
    volumes:
      - ./user/:/usr/src/app
      #- /usr/src/app/node_modules
    expose:
      - "8080"
    depends_on:
      - gateway
      - poker_mongo
    networks:
      - poker_network
    environment:
      WAIT_HOSTS: poker_mongo:27017
  mongo-express:
    container_name: mongo-express
    image: mongo-express
    restart: always
    ports:
      - 8085:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: test
      ME_CONFIG_MONGODB_ADMINPASSWORD: 123
      ME_CONFIG_MONGODB_SERVER: poker_mongo
    depends_on:
     - gateway
     - poker_mongo
    networks:
      - poker_network

networks:
  poker_network:
volumes: 
  db-data:

nginx.conf

events { worker_connections 1024; }

http {

    server {
        listen 80;
        location /user {
            rewrite /user/(.*) /$1  break;
            proxy_pass http://poker_user:8080/;
            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header Host $host;
        }
        location / {
            return 200 'hello';
        }

    }
}

Any help is appreciated, thanks.

Are you sure your node app doesn’t sporadically fails to handle request and doesn’t return any response? You could try removing all logic from `/user` handler and return `200` status code and see if you can reproduce the error. If you could add handler code that would also help. — Igor Nikolaev, Mar 23 '19 at 17:20
yes sorry I meant to mention I have already tried this and It gets 504. It doesn't start out working, it either does work or doesn't and I get no output in the logs of the node app failing. I will try create a simple example to add — Emobe, Mar 23 '19 at 17:27
@IgorNikolaev I've added some updates with what I've found out — Emobe, Mar 24 '19 at 11:43
You could try to change the order of dependencies - `gateway` should depend on `poker_user` — Igor Nikolaev, Mar 24 '19 at 11:50
@IgorNikolaev Thank you, this worked. I seemed to have the idea about `depends_on` backwards when it came to NGINX. Could you please add this as an answer? — Emobe, Mar 24 '19 at 12:58
great that it helped. One more question though - would it work in previous set up if you wait a bit longer before testing? — Igor Nikolaev, Mar 24 '19 at 13:04
@IgorNikolaev no, I tried waiting as well and it was only after the reload it would work. — Emobe, Mar 24 '19 at 13:07

score 2 · Accepted Answer · answered Mar 25 '19 at 08:52

I found several similar related questions and will try to compile two possible solutions out of those:

First solution

First solution is to reverse dependency between services: instead of poker_user depend on gateway you can may gateway depend on user. This way when gateway starts if will be able to resolve poker_user hostname.

The Dockerfile would look this way in this case:

version: '3.5'
services:
  gateway:
    image: nginx:latest
    restart: always
    ports:
      - 8080:80
    depends_on: poker_user
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/nginx-error.log:/var/log/nginx/error.log
    networks:
      - poker_network
  poker_mongo:
    image: mongo:latest
    container_name: poker_mongo
    restart: unless-stopped
    volumes:
      - db-data:/data/db
    expose: 
      - "27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: test
      MONGO_INITDB_ROOT_PASSWORD: 123
    networks:
      - poker_network
  poker_user:
    container_name: poker_user
    build: ./user/
    image: poker/user:latest
    volumes:
      - ./user/:/usr/src/app
      #- /usr/src/app/node_modules
    expose:
      - "8080"
    depends_on:
      - poker_mongo
    networks:
      - poker_network
    environment:
      WAIT_HOSTS: poker_mongo:27017
  mongo-express:
    container_name: mongo-express
    image: mongo-express
    restart: always
    ports:
      - 8085:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: test
      ME_CONFIG_MONGODB_ADMINPASSWORD: 123
      ME_CONFIG_MONGODB_SERVER: poker_mongo
    depends_on:
     - gateway
    networks:
      - poker_network

networks:
  poker_network:
volumes: 
  db-data:

I also removed some other dependencies as I think they are redundant.

Second solution

This solution is based on changing nginx configuration. It looks like nginx tries to resolve hostname specified in proxy_pass upfront and since it's not available yet when nginx starts it will keep responding with 504 because it doesn't try to resolve it again.

There seems to be a workaround with using variables inside proxy_pass directive which are treated differently and force nginx to perform DNS resolution again. This requires resolver to be configured in nginx and luckily there's one built into Docker when using user-defined networks.

Here's an example nginx configuration (though I haven't tested it myself):

events { worker_connections 1024; }

http {

    server {
        listen 80;
        location /user {
            # Docker DNS server in user-defined networks
            # https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/
            resolver 127.0.0.11 ipv6=off;

            rewrite /user/(.*) /$1  break;

            # Using a variable will force nginx to re-resolve DNS name
            # https://serverfault.com/a/593003
            set $backend "http://poker_user:8080/";
            proxy_pass $backend;

            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header Host $host;
        }
        location / {
            return 200 'hello';
        }

    }
}

Hope this help!

504 response from ngingx reverse proxy with nodejs + docker

1 Answers1