0

I am using some constants in my DAG which are being imported from another (configuration) Python file in the project directory.

Scenario

Airflow is running, I add a new DAG. I am importing the schedule_interval from that configuration file and some other constant as well which I am passing to a function being called in the PythonOperator in my DAG. I update the code base, so new dag gets added in the airflow_dag folder and the configuration file gets updated with the new constants.

Problem

The schedule_interval does not work and the dag does not get scheduled. It also throws an exception (import error) for any other constant which is being imported in the dag. In the web ui I can see the new dag, but I can also see a red label error that displays could not found constant XYZ in configuration_file.py while it's actually there.
It does not come here no matter how long I wait.

Bad Solution

I go and restart the airflow scheduler (and webserver as well just in case), and everything starts working again.

Question

Is there a solution to this where I will not have to restart airflow and update those things?

Note: The proposed solution to refresh dag in question Can airflow load dags file without restart scheduler did not work for me.

saadi
  • 646
  • 6
  • 29
  • what i get from your problem description is you need to fix import error rest will work itself as import error is not letting your DAG to refresh. if you could add some code snippet it would be helpful to diagnose problem. – Shahbaz Ali Jul 28 '20 at 13:43
  • make sure your configuration_file.py is reachable by your python environment, i.e check if it exists in python system path `sys.path` – Shahbaz Ali Jul 28 '20 at 13:45
  • @ShahbazAli as I told you, there is no import error. On IDE I can see the import and it works fine. Airflow is just not picking the latest change in the file without restarting. When I restart airflow import error goes away which means the code and import is there on the disk, airflow is just not picking it up. – saadi Jul 28 '20 at 13:50
  • Would you consider to switch from a config file to a JSON variable? – Javier Lopez Tomas Jul 29 '20 at 08:02
  • @JavierLópezTomás I have that option in mind, but that will require a LOT of work. Also, I am not sure if there's is an option to import/export those variables for QA & PROD or for later use. I wouldn't like to add all those variables every time I have to set up my airflow. So, I would do that if nothing else works out. Right now I am looking to make this method work. – saadi Jul 29 '20 at 10:19
  • @saadi have you tried to change airflow configuration? have you tried to change `dagbag_import_timeout` value in airflow.cfg also changing value of `dag_dir_list_interval` to some lower integers such as 30 or 20. please check if then can solve your problem – Shahbaz Ali Jul 29 '20 at 12:40
  • @saadi Yes, you could. You just have to have a json file in your git repository, and every time you set up airflow you can add those variables using something like "airflow variables -i json_file". I have that in my docker-compose file, for instance – Javier Lopez Tomas Jul 29 '20 at 15:11
  • 1
    @JavierLópezTomás that makes things a lot easier except in cases where those constants are used in other code as well, I will have to maintain two copies, one for airflow and one for code but otherwise sounds like a good plan. – saadi Jul 29 '20 at 15:14

0 Answers0