NGINX : Using Upstream server name

Question

I am running nginx/1.19.6 on Ubuntu.

I am struggling to get the upstream module to work without returning a 404.

My *.conf files are located in /etc/nginx/conf.d/

FILE factory_upstream.conf:

upstream factoryserver  {
    server  factory.service.corp.com:443;
}

FILE factory_service.conf:

server
{
    listen 80;
    root /data/www;

    proxy_cache factorycache;
    proxy_cache_min_uses 1;
    proxy_cache_methods GET HEAD POST;

    proxy_cache_valid 200  72h;
    #proxy_cache_valid any  5m;

    location /factory/ {
        access_log /var/log/nginx/access-factory.log  main;
        proxy_set_header x-apikey abcdefgh12345678;
        ### Works when expressed as a literal.#   proxy_pass https://factory.service.corp.com/;
        ### 404 when using the upstream name.
        proxy_pass https://factoryserver/;
    }
}

I have error logging set to debug, but after reloading the configuration and attempting a call, there are no new records in the error log.

nginx -t     # Scans OK
nginx -s reload    # no errors
cat /var/log/nginx/error.log
...
2021/03/16 11:29:52 [notice] 26157#26157: signal process started
2021/03/16 11:38:20 [notice] 27593#27593: signal process started

The access-factory.log does record the request :

127.1.2.3 --;[16/Mar/2021:11:38:50 -0400];";GET /factory/api/manifest/get-full-manifest/PID/904227100052 HTTP/1.1" ";/factory/api/manifest/get-full-manifest/PID/904227100052" ;404; - ;/factory/api/manifest/get-full-manifest/PID/904227100052";-" ";ID="c4cfc014e3faa803c8fe136d6eae347d ";-" ";10.8.9.123:443" ";404" ";-"

To help with debugging, I cached the 404 error, "proxy_cache_valid any 5m;" commented out in the example above: When I use the upstream name, the cache file contains the followiing:

<@#$ non-printable characters $%^>
KEY: https://factoryserver/api/manifest/get-full-manifest/PID/904227100052
HTTP/1.1 404 Not Found
Server: nginx/1.17.8
Date: Tue, 16 Mar 2021 15:38:50 GMT
...

The key contains the name 'factoryserver' I don't know if that matters or not. Does it? The server version is different than what I see when I enter the command nginx -v, which is: nginx version: nginx/1.19.6 Does the difference in version in the cache file and the command line indicate anything?

When I switch back to the literal server name in the proxy_pass, I get a 200 response with the requested data. The Key in the cache file then contains the literal upstream server name.

<@#$ non-printable characters $%^>
KEY: https://factory.service.corp.com/api/manifest/get-full-manifest/PID/904227100052
HTTP/1.1 200
Server: nginx/1.17.8
Date: Tue, 16 Mar 2021 15:59:30 GMT
...

I will have multiple upstream servers, each providing different services. The configuration will be deployed to multiple factories, each with its own upstream servers. I would like for the deployment team to only have to update the *_upstream.conf files, and keep the *_service.conf files static from deployment site to deployment site.

• factory_upstream.conf
• product_upstream.conf
• shipping_upstream.conf
• abc123_upstream.conf

Why do I get a 404 when using a named upstream server?

Answer 1

Based on the nginx version in the cached response not matching what you see on the command line, it seems that maybe the 404 is coming from the upstream server. I.e, your proxying is working, but the upstream server is returning a 404. To troubleshoot further, I would check the nginx logs for the upstream server and if the incoming request is what you expect.

Note that when using proxy_pass, it makes a big difference whether you have a / at the end or not. With a trailing slash, nginx treats this as the URI it should send the upstream request to and it doesn't include the URI matched by the location block (/factory/) while without, it includes the full URI as-is.

proxy_pass https://factoryserver/ results in https://factory.service.corp.com:443/

proxy_pass https://factoryserver results in https://factory.service.corp.com:443/factory/

Docs: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/

So maybe when switching between using an upstream and specifying the literal server name, you're inadvertently being inconsistent with the trailing slash. It's a really easy detail to miss, especially when you don't know it's important.

Answer 2

Please provide more information about your server configuration where you proxy pass the requests. Only difference I see now is that you specify the port (443) in your upstream server.