How to remove default example dags in airflow

Question

I am a new user of Airbnb's open source workflow/datapipeline software airflow. There are dozens of default example dags after the web UI is started. I tried many ways to remove these dags, but I've failed to do so.

load_examples = False is set in airflow.cfg.
Folder lib/python2.7/site-packages/airflow/example_dags is removed.

States of those example dags are changed to gray after I removed the dags folder, but the items still occupy the web UI screen. And a new dag folder is specified in airflow.cfg as dags_folder = /mnt/dag/1. I checked this dag folder, nothing is there. It's really weird to me why it is so difficult to remove these examples.

score 177 · Answer 1 · edited Mar 11 '23 at 08:12

177

When you startup airflow, make sure you set:

load_examples = False

inside your airflow.cfg

If you have already started airflow with this not set to false, you can set it to false and run airflow db reset(if you are using 1.x airflow resetdb) in the cli (!which will destroy all current dag information!).

Alternatively you can go into the airflow_db and manually delete those entries from the dag table.

edited Mar 11 '23 at 08:12

Spike Lee

117
2
10

answered Apr 14 '17 at 15:12

jhnclvr

9,137
5
50
55

9

I've just tried this in 1.8.1 and it doesn't seem to be enough. I can confirm that the dag table is empty after running `airflow resetdb`, and I've made the change to the cfg file, but the example dags are all still there in the webserver. I had to restart the webserver to make this work. That involved killing a lot of processes and running `airflow webserver` again. Probably best to run with systemd or upstart to manage restarts https://airflow.incubator.apache.org/configuration.html?highlight=systemd#integration-with-systemd – Davos Aug 23 '17 at 12:55
1

running airflow resetdb command works fine in 1.10.5. should be accepted solution (upvoting for now) – krinker Oct 02 '19 at 13:44
This answer makes sense. The database is reset and uploaded with new data. I think OP forgot to check. @Bronzels – Jossie Calderon Apr 18 '20 at 20:44
Since this is highly voted, people disabling example dags may also want to disable default connections. A similar property exists in 1.10.10 `load_default_connections = False` https://stackoverflow.com/questions/38407314/airflow-too-many-connections-as-a-default – Gabe May 22 '20 at 16:33
Please note, this might cause your webserver to throw an error and won't load up. I had to recreate the user for the DAG to be up, and examples still are there. They didn't go anywhere. – Jyoti Dhiman Aug 12 '20 at 06:15
2

Beware that resetdb will remove all the connections/variables that you have defined. – smishra Sep 02 '20 at 17:40
9

for the `airflow 2.0.1` , as I know; after setting `load_examples = False` inside your `airflow.cfg` use `airlow db reset` instead of `airflow resetdb` – Inadel Feb 25 '21 at 18:39

score 48 · Answer 2 · answered Feb 16 '21 at 07:44

48

For Airflow 2.0, in docker-compose.yaml you can set AIRFLOW__CORE__LOAD_EXAMPLES: 'false' to not load them instead of editing the .cfg file.

answered Feb 16 '21 at 07:44

paradox

634
5
13

2

This saved me a lot of banging my head against a wall. Thank you! The default docker-compose file in the documentation has this setting set and it overrides whatever is in the `airflow.cfg` file. – juniper- Mar 11 '21 at 23:51
Everyone focused on the cfg file while the problem was with compose file, thanks! – Mariusz Mączkowski Aug 17 '22 at 17:04

score 19 · Answer 3 · answered Nov 27 '20 at 21:35

Like others have said, you can change load_examples = False within airflow.cfg. However this requires that the cfg file already existing.

You can init the airflow DB without having to configure the cfg file by using ENV variables.

export AIRFLOW__CORE__LOAD_EXAMPLES=False
airflow initdb

See docs for more information.

score 6 · Answer 4 · answered Dec 04 '20 at 06:34

Before you start airflow make sure you set load_example variable to False in airflow.cfg file. By default it is set to True.

load_examples = False

If you have already started airflow, you have to manually delete example DAG from the airflow UI. Click on delete icon available on the right side of the DAG to delete it.

Instead of manually deleting example DAG, you can reset your database by using airflow resetdb command but that will delete your connections and variables and other important information. Do not use airflow resetdb option in production.

the command is now `airflow db reset` – Matthew Barlowe Mar 21 '22 at 20:27 — Matthew Barlowe, Mar 21 '22 at 20:27

Aku · Answer 5 · 2022-02-15T14:53:57.897

5

Easy Way:

put ..

load_examples = False

into the airflow.cfg file

then close and restart the webserver and scheduler

edited Feb 15 '22 at 14:53

answered Jul 19 '21 at 19:31

Aku

660
6
9

Restarting server `and` scheduler is the important part. Thank you! – GregK Feb 14 '22 at 10:37

yartem · Answer 6 · 2020-06-05T09:52:31.353

TL;DR: check that you have only DAG files in your dags_folder -- Airflow will traverse this directory recursively and try to load all .py files.

I've lost some time debugging similar behaviour of Airflow: even though load_examples = False, airflow was still loading tons of unnecessary stuff, including example_dags. The problem was that I was having a virtualenv directory venv/ in dags_folder, and -- I was not expecting that -- Airflow searches for dags recursively in the dags dir. So it was loading example_dags from the apache-airflow installed in that virtualenv.

UPD: there's a .airflowignore file to ignore directories from dags_folder

score 1 · Answer 7 · edited Dec 02 '21 at 11:52

1

while starting up the airflow make sure to change

load_examples = False

in the airflow.cfg file .Then close and restart the webserver and scheduler.

edited Dec 02 '21 at 11:52

buddemat

4,552
14
29
49

answered Dec 02 '21 at 11:24

jaykumar gupta

19
2

score 1 · Answer 8 · answered Jan 30 '23 at 21:12

As others pointed out, change the load_examples variable to False in airflow.cfg. Should look something like:

load_examples = False

If you have already started the server you need to reset the database. In the newer versions, the command to reset the DB is:

airflow db reset

Some of the answers above use the old CLI commands

score 0 · Answer 9 · answered Nov 21 '20 at 14:46

If LDAP based authentication is turned on after airflow resetdb and restarting airflow your login window may not appear and may give an error due to cached login id ( but no password ). If this happens, clear your cache and try. If that still doesn't work, turn off authentication, stop airflow and start. Then turn on authentication stop and start - you will be able to see login window and login with your LDAP authentication -Suresh

How to remove default example dags in airflow

9 Answers9

Linked