nginx reverse proxy server does not fetch full bundle from the upstream server in docker stack, net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK)

Question

I deployed an nginx:1.22.1 instance alongside a static react app server on a worker node in a docker swarm. This is docker swarm mode, not classic swarm.

The advertise address that I listed when joining the swarm is internal to the data center, I do not know if that matters because I can still access these services with the public addresses.

Both containers are pinned to the same worker node and communicate over a user-created overlay network.

I can retrieve the full bundle directly from the react app server over the public network.

I cannot retrieve the full bundle through the nginx reverse-proxy server over the public network.

When I attempt to fetch the bundle using chrome browser as the user-agent I get 2 errors:

1. net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK).

The app bundle is cutoff mid js function as if a chunk of data was not transmitted.

2. Rarely, the upstream server will send html and not a js bundle. But I receive that whole response body and it is not truncated like the js bundle.

I have played with all kinds of configuration and cannot get it to work.

(most relevant)

This is my configuration under /etc/nginx/conf.d/default.conf

resolver 127.0.0.11 valid=10s;
error_log /dev/stdout info;

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen 80;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    ssl_certificate /etc/nginx/certs/nginx.pem;
    ssl_certificate_key /etc/nginx/certs/key.pem;
    client_max_body_size 100M;
    proxy_buffers 8 1024k;
    proxy_buffer_size 1024k;
    proxy_max_temp_file_size 1024m;

    location / {
        set $reactapp reacthost;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://$reactapp:3000/;
        proxy_redirect off;
    }
        root /usr/share/nginx/html;
    }

    error_page 500 502 503 504 /50x.html;
}

I use the variable $reactapp for service discovery after nginx server start. See NGINX blog here.

Note that the nginx:1.22.1 instance runs with user nginx after it is deployed to the stack. I only see this below message when I deploy via docker stack. If I start the container directly using docker engine, I do not see it.

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)

However, I can exec into the container as the nginx user, access /var/cache/nginx/, and create a directory.

I do not know if:

1. my server / location configuration is plain bad.

2. The NGINX server cannot write a part of the container it needs to write when the service is deployed in stack mode.

3. If I cannot access the server properly over the public network via the overlay network.

Prior to using docker stack I was able to use this reverse proxy.

1. The two containers were on the same host without swarm mode running.
2. The containers communicated over a bridge network.
3. The reverse proxy server port was published on the public interface of the server it was deployed on.
4. The NGINX server started after the upstream server.

Because there is no depends_on key honored in stack mode I have to allow DNS service discovery after the NGINX server starts up. Placing them on an overlay gives me more flexibility in how I do my deployments, but this has become a bit muddled. There are enough differences between the two environments that it has become difficult to get the stack to behave as I expect.