2

My structure

  1. Kubernetes cluster on GKE
  2. Ingress controller deployed using helm
  3. An application which will return list of IP ranges note: it will get updated periodically
curl https://allowed.domain.com
172.30.1.210/32,172.30.2.60/32
  1. Secured application which is not working

What I am trying to do?

  1. Have my clients IPs in my API endpoint which is done
curl https://allowed.domain.com
172.30.1.210/32,172.30.2.60/32
  1. Deploy my example app with ingress so it can pull from the https://allowed.domain.com and allow people to access to the app

What I tried and didn't work?

  1. Deploy the application with include feature of nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
      include /tmp/allowed-ips.conf;
      deny all;

yes its working but the problem is when /tmp/allowed-ips.conf gets updated the ingress config doesn't

  1. I tried to use if condition to pull the IPs from the endpoint and deny if user is not in the list
    nginx.ingress.kubernetes.io/configuration-snippet: |
      set $deny_access off;
      if ($remote_addr !~ (https://2ce8-73-56-131-204.ngrok.io)) {
        set $deny_access on;
      }
  1. I am using nginx.ingress.kubernetes.io/whitelist-source-range annotation but that is not what I am looking for

None of the options are working for me.

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
Farkhod Sadykov
  • 67
  • 6
  • Hello Farkhod. It is not clear if the second option using 'if' is working for you. And why is the third option not what you are looking for? – mozello Mar 28 '22 at 16:46
  • I am trying to dynamically whitelist IP ranges to the application. The allowed IP ranges are getting updated on the endpoint and the ingress controller should automatically pick up the changes from the endpoint and white list the app – Farkhod Sadykov Mar 29 '22 at 22:31

2 Answers2

0

From the official docs of ingress-nginx controller:

The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app)

After the nginx ingress resource was initially created, the ingress controller assembles the nginx.conf file and uses it for routing traffic. Nginx web server does not auto-reload its configuration if the nginx.conf and other config files were changed.

So, you can work around this problem in several ways:

  • update the k8s ingress resource with new IP addresses and then apply changes to the Kubernetes cluster (kubectl apply / kubectl patch / smth else) / for your options 2 and 3.
  • run nginx -s reload inside an ingress Pod to reload nginx configuration / for your option 1 with include the allowed list file. 
    $ kubectl exec ingress-nginx-controller-xxx-xxx -n ingress-nginx -- nginx -s reload
  • try to write a Lua script (there is a good example for Nginx+Lua+Redis here and here). You should have a good understanding of nginx and lua to estimate if it is worth trying.
mozello
  • 1,083
  • 3
  • 8
  • Hey Mozello, Thank you for your answer I did try this before but unfortunately, this is not what I am looking for. Basically, I have a lot of clusters in different environments with an ingress controller running and the endpoints with IPs get changed in the range of 2-20 minutes, and going to each cluster and reloading the ingress resources is difficult for me to do it. It should be an easier way to accomplish this. Please leave your comment if you need more clarification – Farkhod Sadykov Apr 03 '22 at 01:48
0

Sharing what I implemented at my workplace. We had a managed monitoring tool called Site24x7. The tool pings our server from their VMs with dynamic IPs and we had to automate the whitelisting of the IPs at GKE.

nginx.ingress.kubernetes.io/configuration-snippet allows you to set arbitrary Nginx configurations.

  1. Set up a K8s CronJob resource on the specific namespace.
  2. The CronJob runs a shell script, which
  • fetches the list of IPs to be allowed (curl, getent, etc.)
  • generates a set of NGINX configurations (= the value for nginx.ingress.kubernetes.io/configuration-snippet)
  • runs a kubectl command which overwrites the annotation of the target ingresses.

Example shell/bash script:

#!/bin/bash

  site24x7_ip_lookup_url="site24x7.enduserexp.com"

  site247_ips=$(getent ahosts $site24x7_ip_lookup_url | awk '{print "allow "$1";"}' | sort -u)
ip_whitelist=$(cat <<-EOT

  # ---------- Default whitelist (Static IPs) ----------
  # Office
  allow vv.xx.yyy.zzz;
  # VPN
  allow aa.bbb.ccc.ddd;

  # ---------- Custom whitelist (Dynamic IPs) ----------

  $site247_ips                            # Here! 

  deny all;

  EOT
  )

  for target_ingress in $TARGET_INGRESS_NAMES; do
    kubectl -n $NAMESPACE annotate ingress/$target_ingress \
      --overwrite \
      nginx.ingress.kubernetes.io/satisfy="any" \
      nginx.ingress.kubernetes.io/configuration-snippet="$ip_whitelist" \
      description="*** $(date '+%Y/%m/%d %H:%M:%S') NGINX annotation 'configuration-snippet' updated by cronjob $CRONJOB_NAME ***"
  done

The shell/bash script can be stored as ConfigMap to be mounted on the CronJob resource.

snowpeak
  • 797
  • 9
  • 25