9

I have a angular application running in a docker ubuntu image that has nginx installed. I want to deploy this image to Kubernetes and use a nginx proxy to redirect all calls to /api to my backend service in Kubernetes.

My static web resources lie in /var/www/html and I add the following config to /etc/nginx/conf.d:

upstream backend-service {
  server backend-service:8080;
}

server {
  listen 80;

  location / {
    try_files $uri $uri/ /index.html;
  }

  location ^~ /api {
    proxy_pass http://backend-service;
  }
}

Accessing the frontend service on / or /#/dashboard returns the expected component of my Angular page, but a call to /api/v1/data only shows the default nginx 404 Not Found page.

What do I need to modify to have my backend calls redirected to my backend?

I use nginx 1.10.3 on ubuntu 16.04 and my frontend Dockerfile looks like this:

FROM ubuntu:16.04

# Install curl, nodejs and nginx
RUN apt-get update && \
  apt-get install -y curl && \
  curl -sL https://deb.nodesource.com/setup_8.x | bash - && \
  apt-get install -y nodejs nginx && \
  rm -rf /var/lib/apt/lists/*

# Create directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Copy and build rest of the app
COPY . /usr/src/app
RUN npm install
RUN node_modules/@angular/cli/bin/ng build --prod
RUN cp -a dist/. /var/www/html

# Configure and start nginx
COPY frontend.conf /etc/nginx/conf.d

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

Edit: Information about backend-service

The backend service listens to get and post requests on /api/v1/data and is reachable in Kubernetes via a Service named backend-service.

Edit2: Nginx access.log

https://gist.github.com/Steffen911/a56e3175bf12e511048d01359a475724

172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET / HTTP/1.1" 200 380 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /styles.d41d8cd98f00b204e980.bundle.css HTTP/1.1" 200 0 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /inline.c9a1a6b995c65c13f605.bundle.js HTTP/1.1" 200 1447 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /polyfills.117078cae3e3d00fc376.bundle.js HTTP/1.1" 200 97253 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /main.3e9a37b4dd0f3bf2465f.bundle.js HTTP/1.1" 200 64481 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /vendor.146173c1a99cc2172a5f.bundle.js HTTP/1.1" 200 661261 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /api/v1/data/ HTTP/1.1" 404 209 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/home.jpg HTTP/1.1" 200 2608 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/busy.gif HTTP/1.1" 200 48552 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/background_light.png HTTP/1.1" 200 170599 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/google.svg HTTP/1.1" 200 2232 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /assets/email.svg HTTP/1.1" 200 1596 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:40 +0000] "GET /favicon.ico HTTP/1.1" 200 198 "http://192.168.99.100:30497/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
172.17.0.1 - - [13/Aug/2017:13:11:44 +0000] "GET /api/v1/data/ HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"

The error.log file is empty.

Edit3: New nginx version and other SO thread

I also tried nginx 1.12.1 and it shows the same behaviour. The answers to this question also haven't helped: nginx proxy_pass 404 error, don't understand why

Edit4: I uploaded a minimal example that reproduces my problem on GitHub

https://github.com/Steffen911/nginx-sample

Steffen Schmitz
  • 860
  • 3
  • 16
  • 34

2 Answers2

24

From your troubleshooting of nginx, it appears that the nginx configuration file you have has effectively no effect — you report getting 404 Not Found errors for everything other than the index page, with all the directives, from try_files in location /, to proxy_pass and return 200 test in a more specific location ^~ /api, having no effect.

As such, the problem appears to be in the Dockerfile — it appears that most other NGINX + Docker tutorials remove default configurations (e.g., with RUN rm /etc/nginx/conf.d/default.conf), whereas your file is missing any such removal.

In fact, Debian/Ubuntu appear to have the non-standard directories of questionable utility called /etc/nginx/sites-available and /etc/nginx/sites-enabled, which, by default, must contain a default file with a presumptuous listen 80 default_server, effectively taking precedence over any other listen of the same port in the absence of a more specific server_name.

As such, there are multiple independent solutions:

  • Do not use fundamentally broken packages like those offered by Debian/Ubuntu. I once spent a good amount of time pulling my hair trying to figure out why my configs don't work, only to notice that even the backup files from emacs like test.conf~ get included in Debian through Debian's default include /etc/nginx/sites-enabled/*;. Sites-Enabled Is Evil.

    Note that NGINX provides official binary packages for most distributions, which wouldn't have had this issue, as it doesn't try to define a default_server in its /etc/nginx/conf.d/default.conf, instead doing a listen 80; with server_name localhost;, getting out of your way automatically by itself in most circumstances.

    E.g., replace FROM ubuntu:16.04 with FROM nginx in your Dockerfile to be using NGINX official image.

  • If still using nginx from Debian/Ubuntu, make sure to RUN rm /etc/nginx/sites-enabled/default in your Dockerfile to remove the default_server listen.

  • Use the server_name directive to define hostname-based servers, possibly together with the listen directive with the default_server parameter, too.

    Note that a duplicate server_name specification results in a configuration warning (with the [warn] severity), but a duplicate default_server is a configuration error ([emerg] severity), which might help troubleshoot the issue earlier.

cnst
  • 25,870
  • 6
  • 90
  • 122
  • 1
    OMG upvote * 1000. Took me ages to work out there was a default.conf server that was catching everything and ignoring my proxy_pass – Phil Sep 03 '20 at 11:20
7

a call to /api/v1/data only shows the default nginx 404 Not Found page

The proxy_pass directive is highly unlikely to be generating the default nginx 404 Not Found error page itself.

  • The 404 might be generated by the upstream, in which case, absent further instructions, nginx will simply propagate the message from upstream. If you get an nginx signature in your 404, then it means that the upstream is also running nginx, possibly to your surprise, revealing the configuration culprit.

  • If the 404 is actually generated by nginx indeed, then it might be the case that the location doesn't match. Try putting return 200 thisisatest; in place of proxy_pass to troubleshoot the issue.

    However, in your specific case, location ^~ /api { is an almost-impossible directive to not match a /api/v1/data/ request URI — the only thing I could think of is if you possibly have a try_files in your server config outside of any other location directive, which could then possibly make all location directives null and void. Are you sure your try_files is fully contained within the location / context?

cnst
  • 25,870
  • 6
  • 90
  • 122
  • I know that my upstream is running a nodejs docker container that is not based on nginx. I also tried your second suggestion and replaced the proxy_pass with return 200 thisisatest; but i still get a 404 for everything that isn't "/". I only added the config file that I pasted here. Does the default nginx config overwrite something that can break my expected behaviour? – Steffen Schmitz Aug 17 '17 at 07:15
  • I also added an example that reproduces my problem completely to the question – Steffen Schmitz Aug 17 '17 at 07:42
  • @SteffenSchmitz, wow, that's really strange that you can reproduce it like that; since you're still getting a 404 even with `return 200` in place, my only other suggestion would have been is that you still have an old config file around that takes precedence over your new one; personally, I've had an issue where I've tried modifying files in `/etc/nginx/sites-enabled`, and the backup files, e.g., `test.conf~`, would take precedence over `test.conf`, and hence things would work *very* weirdly. – cnst Aug 17 '17 at 16:52
  • @SteffenSchmitz, basically, the fact that replacing the proxy_pass with `return 200 test` does not affect your `404 Not Found` outcome, most certainly has to mean that the issue lies somewhere outside of this nginx configuration, as it obviously has to have no effect. Perhaps you're forgetting to run something like `RUN rm /etc/nginx/conf.d/default.conf` in your Docker? – cnst Aug 17 '17 at 18:34
  • 1
    @SteffenSchmitz, what's the total list of files in `/etc/nginx/` in your front-end? It seems like Debian/Ubuntu has these stupid `sites-available` / `sites-enabled` directories, which may have a `default` file with `listen 80 default_server`, likely taking precedence over what you have in yours, hence the 404. – cnst Aug 17 '17 at 18:59
  • You were right. It was the default config in sites-available and sites-enabled that overwrote my frontend config. I extended my Dockerfile with RUN rm -rf /etc/nginx/sites-available RUN rm -rf /etc/nginx/sites-enabled and the backend was contacted as expected. I also added root /var/www/html to my conf. Could you extend your answer with the last comment? – Steffen Schmitz Aug 18 '17 at 07:11
  • 1
    @SteffenSchmitz, great, I provided a new answer; feel free to accept the new answer instead. BTW, when doing bounties, it's best to accept the answer, but not award the bounty before the end of the 7-days, because most folks upvote questions/answers in the final few hours, and awarding the bounty makes it go away from this publicity (too late now, but for the future). – cnst Aug 21 '17 at 03:52
  • add one possibility which solves my problem. I do have try_files configuration in my upstream servers. after I changed the upstream port from 80 to 8080, everything works fine – efinal Nov 14 '19 at 08:51