How to force NGINX to use backup upstream and vice versa?

Question

Maybe it's uncommon but i'd love to use an upstream definition in my nginx loadbalancer, which looks like this:

upstream backend {
  server primary.local.net:80;
  server backup.local.net:80 backup;
}

to aid maintenance processes for those hosts. First i prepare backup.local.net with newest software, then switch over the service to backup and do the same with primary.local.net. In the end, again switch back to primary.

Right now i'm doing this by loading a second configuration file:

upstream backend {
  server primary.local.net:80 backup;
  server backup.local.net:80;
}

using the command:

nginx -s reload

But this is laborious and hope there is a much smarter way to do this?

Answer 1

First of all, using upstream definitions in NGINX should NOT be uncommon! It's the preferred way of doing it.

Unfortunately, there is not really an easy solution for NGINX OpenSource. But why not trying to build something that does not require any config reload.

So given we have two upstream defitions like mentioned above

upstream blue{
  server primary.local.net:80;
  server backup.local.net:80 backup;
}

upstream green{
  server primary.local.net:80;
  server backup.local.net:80 backup;
}

Blue is primary and green is secondary. If you are saying you prepare something, do you think it would be possible to have something on your backend telling NGINX what deployment is currently active. Blue or Green?

Another option could be a file on your NGINX instance keeping that information. njs will be able to read from that file and define the upstream to be used based on the information provided.

https://nginx.org/en/docs/njs/reference.html#njs_api_fs

Quick POC:

upstream.conf

upstream blue {
  server 127.1:9000;
  server 127.1:9100 backup;
}

upstream green {
  server 127.1:9000 backup;
  server 127.1:9100;
}


js_import upstream from conf.d/upstream.js;
js_set $upstream upstream.set;

server {
  listen 80;

  location / {
    proxy_pass http://$upstream/;
  }
}

upstream.js

export default { set }

function set(r) {
  var fs = require('fs');

  try {
    var c =  fs.readFileSync("/etc/nginx/conf.d/active.txt");
  } catch (e) {
    r.error("Error while reading upstrem file.");
    // maybe set c to somehting default then.
  }
  return c;
}

active.txt

blue

Note: Make sure creating the file without a new-line at the end like echo -n "blue" > active.txt.

You can now chnage the content of active.txt during runtime and the upstream will be configured dynamically. With this solution you can even check for request headers and if you want to test an inactive upstream this will work as well. Pretty flexible though.

Answer 2

There's a pattern for /etc/nginx where you have a master nginx.conf file that loads all of the config files in another directory, like "active_services".

Your actual config files are stored in "available_services", and symlinked into the active_services directory.

Either flip the link, or delete one and create the other.