12

Is DAG versioning a thing ? I can't find much on the subject with a few Google searches. I would like to look at the DAGs screen in Airflow and be sure of what DAG code is in the wild.

The simplest solution would be to include a version number as part of the dag_id, but I would appreciate knowing if anyone has better, alternative solution. Tags would work too and migjht look good in the UI - they are designed for for filtering though, I'm not sure if there would be undesirable side-effects.

Phil
  • 598
  • 1
  • 9
  • 21
  • 3
    It was introduced for AirFlow 2.0 yesterday: https://www.meetup.com/NYC-Apache-Airflow-Meetup/events/270483933/ – briba May 14 '20 at 12:06
  • 1
    We I am doing is the same you mentioned: include version number in DAG ID hehe – briba May 14 '20 at 12:07
  • Airflow 2.0 ... awesome ! – Phil May 15 '20 at 12:47
  • It doesn't look like first-class versioning ever made it into 2.0... No mention of [AIP-36](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning) in either the [Press Release](https://airflow.apache.org/blog/airflow-two-point-oh-is-here/) or the [Changelog](https://airflow.apache.org/docs/apache-airflow/stable/changelog.html#airflow-2-0-1-2021-02-08) – Michael Wheeler Mar 25 '21 at 19:12

3 Answers3

10

As the author of the DAG Versioning AIP, I can say that this work has been deferred post 2.0 mainly to support end-to-end DAG Versioning.

Originally, we (Airflow Core Committers) were planning to have a Webserver-only DAG Versioning i.e. to improve the visibility behaviour but not execution:

The scope of this AIP to make sure that the visibility behavior of Airflow is correct, without changing the execution behaviour which will continue to be based on the most recent version of the DAG.

This means it overcomes the issues where you can go back to an old version of the DAG, to view the shape of the DAG few months back and you can see the correct representation instead of "always-latest".

Currently, Airflow suffers from the issue where if you add/remove a task, it gets added/removed in all the previous DagRuns in the Webserver.

However, what we have decided is that we will accomplish Remote DAG Fetcher + DAG Versioning and enable versioning of DAG on the worker side, so a user will be able to run a DAG with the previous version too.

Currently, we don't have a date but mostly planning to do it around 2023/24.

kaxil
  • 17,706
  • 2
  • 59
  • 78
2

The Airflow project has a draft feature open to support DAG versions. The answer currently is that Airflow does not support versions.

The first use case in the link describes a key limitation, log files from previous runs can only surface nodes from the current DAG.

Merlin
  • 1,780
  • 1
  • 18
  • 20
0

As mentioned above, as of yet, Airflow doesn't has its own functionality of versioning workflows. However you can manage that on your own by managing DAGs on their own git repository and fetching its state into airflow reposiroty as submodules. More on that; https://www.youtube.com/watch?v=a-4yRne3ba4&lc=UgwiIO-ECVFSZPz1hOt4AaABAg

Anum Sheraz
  • 2,383
  • 1
  • 29
  • 54
  • 1
    Nice video. This is a workable solution that unfortunately has to add a lot of complexity to achieve this, but AIP-36 has been stalled a long time. Worth mentioning that Astronomer just mentioned (yesterday!) that they'll be resurrecting AIP-36, so hopefully we'll see some action on that now! – Forest Feb 16 '22 at 17:30