0

My set up:

I am running Airflow 1.10.9 behind two nginx proxies. The first proxy forwards requests to a second proxy sitting within a private network. The second proxy forwards the request onto the requested Airflow instance, where the airflow host is specified in the URL. This essentially allows us to run multiple airflow instances and route to them correctly.

I have also updated the Airflow base_url to http://myhost/airflow-<airflowid>/ Here, the airflow id part of the URL is how the second proxy identifies which machine to forward on to.

The proxy set up is working, in that the Airflow UI is accessible and I can see my DAGS loaded into the UI. However, when I try to interact with a DAG I get the following error:

DAG "None" seems to be missing.

I tracked the issue down to the GET request that is reaching the Airflow UI. I can see that the request does not contain any DAG ID. On my working Airflow instance (one that is not running behind any proxies) The request contains the DAG ID run_etl:

"GET /graph?dag_id=run_etl&root=&execution_date=2020-02-10T11%3A42%3A14.454118%2B00%3A00 HTTP/1.1" 200 10174 "http://10.83.163.248:3128/graph?dag_id=run_etl" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36"

On the Airflow instances running in the private network the GET request does not contain the DAG ID:

"GET /admin/airflow/graph HTTP/1.1" 302 221 "https://<redacted host>/admin/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36"

My nginx configuration is as follows:

Proxy 1:

server {
    listen 443 ssl;
    server_name  <redacted server name>;

    location ~ /airflow-(.*) {
        proxy_pass http://172.23.7.79:80;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Proxy 2:

server {
    listen     80;

    location ~ /airflow-([^\/]+)/(.*) {
        proxy_pass http://$1-<redacted hostname>:8080/airflow-$1/$2;
        proxy_redirect  off;
        proxy_set_header Host $host;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

}

Any thoughts on what is wrong here would be greatly appreciated!

Brackl1
  • 79
  • 1
  • 10

1 Answers1

0

The answer turned out to be more straight forward that I was anticipating.

Proxy 2 needs some additional configuration in order to pass the URL query in the proxy pass.

proxy_pass http://$1-<redacted hostname>:8080/airflow-$1/$2$is_args$args;

Ref: How can query string parameters be forwarded through a proxy_pass with nginx?

Brackl1
  • 79
  • 1
  • 10