2

I'm trying to develop a choreography architecture using the Apache Airflow Framework, but I don't want the DAGs to run in a scheduled way.

I read that the Apache Airflow framework can be used for service choreography. But I don't need any scheduling, I just need the framework to get a choreography between the nodes of a workflow.

Is it possible to use Apache Airflow without the scheduler? Is there any other framework (Java or Python) that you recommend me for this?

Thank you! :)

MonkeyDev
  • 21
  • 2

2 Answers2

0

You can certainly create unscheduled DAGs

schedule_interval=None

You'll have to have the scheduler running to execute the task though, which you do like this:

airflow trigger_dag my-dag

As for alternatives, I suppose it all depends on what you are trying to accomplish. Most of the big choreography solutions have some scheduling built into the run-time. Apache Beam doesn't, but that's more specifically for data transformation and stream processing.

The closest thing might be luigi: https://github.com/spotify/luigi. It's a python project that allows you to choreograph tasks like AirFlow. It seems not to have a scheduler. It also seems to have a much simpler UI.

Robert Moskal
  • 21,737
  • 8
  • 62
  • 86
  • First of all, thanks for your comment, Robert!!! I need to know if there is any way to create a choreography architecture using Apache Airflow. How can I make each step of the DAG generate an event and it remains in a queue (JMS) that it's listening for the following step and it's activated when it finds the data? Is it possible with Apache Airflow? – MonkeyDev Jun 13 '19 at 08:33
  • I think you should ask another question! – Robert Moskal Jun 13 '19 at 13:30
0

Uber Cadence Workflow is perfect for service orchestration scenarios. This answer provides more info.

Maxim Fateev
  • 6,458
  • 3
  • 20
  • 35