27

Airflow veterans please help,

I was looking for a cron replacement and came across apache airflow.

We have a setup where multiple users should be able to create their own DAGs and schedule their jobs.

Our users are a mix of people who may not know how to write the DAG python file. Also, they may not have access to the server where airflow is running.

Is it possible to create an airflow DAG via UI. I could not find any reference to the same. All examples speak about creating a python file and uploading it to the $AIRFLOW_HOME/dag/ directory. Users will not have access to this directory.

Rundeck for example allows user to add workflows and task dependencies via UI. Is there a plugin/functionality similar to this in airflow.

PS: I really like the way airflow shows the dependency graphs and want to try it out. But If creating a DAG is so complicated, then it will be a major problem for lots of my end users.

Mukul Jain
  • 1,807
  • 9
  • 26
  • 38

5 Answers5

14

I don't think there is an out of the box solution. What I would try with your task is the combination: CWL(Common Workflow Language), Airflow, Rabix Composer

So, CWL is just a specification, you describe your workflow in YAML like syntax.

There is a package pip3 install cwl-airflow it can parse CWL into Airflow DAG https://github.com/Barski-lab/cwl-airflow.

Rabix Composer UI/graphical CWL editor.

Andrey Kartashov
  • 1,368
  • 1
  • 12
  • 20
7

I believe there is one now: https://github.com/andreax79/airflow-code-editor

And it looks promising.

enter image description here

human
  • 2,250
  • 20
  • 24
6

elyra seems to be able to create airflow pipelines (did not try it):

https://elyra.readthedocs.io/en/latest/getting_started/overview.html

enter image description here

Stefan
  • 10,010
  • 7
  • 61
  • 117
  • [Disclaimer: I am one of the maintainers of the Elyra open source project] It does! Users can add Jupyter notebooks, Python scripts, R scripts, and custom components/operators to pipelines using the [Visual Pipelines Editor](https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html#creating-pipelines-using-the-visual-pipeline-editor), which is available for JupyterLab. We've got a couple of [tutorials](https://elyra.readthedocs.io/en/latest/getting_started/tutorials.html) that should provide a pretty good overview. – ptitzler Feb 07 '22 at 18:20
2

There is an open source project that looks interesting:

https://github.com/timkpaine/paperboy

Pav K.
  • 2,548
  • 2
  • 19
  • 29
  • Hello, I am trying to use **paperboy** but facing issue : https://stackoverflow.com/questions/59154669/paperboy-throwing-error-typeerror-the-json-object-must-be-str-not-bytes Can you please have a look and help me to get out of it. – Umesh Gaikwad Dec 04 '19 at 05:04
2

Use Airflow Variable model, it could do it.

Step 1, define you biz model with user inputs Step 2, write in as dag file in python, the user input could be read by airflow variable model.(key/value mode) step 3. exchange tasks info by airflow xcom model.

in production mode, user input their parameter in airflow web ui->admin->variable for certain DAG.(key value mode)

then it done.

Recently, I use airflow and dag with user viable to do a lot of different kinds of outlier detection dag for different scenarios. work smoothly.

hope above experience could help you.

Yong Wang
  • 1,200
  • 10
  • 15