Nginx proxy + CDN

Question

We are using a VM running a nginx proxy ( caching turned off ) to redirect hosts to the correct CDN url.

We are experiencing issues with the proxy showing old content that does not match what shows on the CDN. The CDN provider we use is Verizon (by through Azure - Microsoft CDN by Verizon).

When we do updates to the origin we automatically send purge requests to the CDN. These can be both manual and automatic updates dynamically and both single url purges and wilcard ones. What seems to be happening is when we get 2 pruge requests close in time. The first one goes through to the proxy, but the second one does not. Although both show correctly when accessing the CDN url directly.

To mention is that this issue only happens about 30% of the time.

nginx samle conf:

server {
    resolver 8.8.8.8;
    server_name <CUSTOM HOST>;
    location / {
      # Turn off all caching
      proxy_no_cache 1;
      proxy_cache_bypass 1; 
      proxy_redirect off;
      expires -1;
      proxy_cache off;
      
      
      # Proxy settings
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Real-IP       $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_headers_hash_max_size 512;
      proxy_headers_hash_bucket_size 128;
      
      # Override cache headers to set 2min browser cache
      proxy_hide_header Cache-Control;
      add_header Cache-Control max-age=120;
      
      proxy_pass "<CDN URL>request_uri";
    }
}

nginx.conf:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 768;
    # multi_accept on;
}

http {

    sendfile off;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    gzip on;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Below is an example where the CDN is showing newer content then the proxy, we have a Last-Modifed mismatch:

CDN:

PROXY:

I have tried through a VPN to see if there is anything whit a particular POP that the proxy hits, but all POP:s show the correct content.

When the error is present, sending a curl request from the proxy to the CDN will result in the same incorrect headers.

After we perform a purge several requests goes through to origin directly until the CDN starts serving a Cached version again.

The we receive the first HIT about 1 min later.

I started to assume that this might have something internally to do with Azure & Verizon. So I created an exact duplicate proxy hosted on amazon, but the error seem to precist. Is there something else in the nginx that can cause this behavior?

Answer 1

Try adding just to see if the pages are retrieved each hit.

add_header Last-Modified $date_gmt;
add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
if_modified_since off;
expires off;
etag off;