12

I'm new to airflow and celery, and I have finished drawing dag by now, but I want to run task in two computers which are in the same subnet, I want to know how to modify the airflow.cfg. Some examples could be better. Thanks to any answers orz.

Fewfy
  • 135
  • 1
  • 1
  • 6

1 Answers1

20

The Airflow documentation covers this quite nicely:

First, you will need a celery backend. This can be for example Redis or RabbitMQ. Then, the executor parameter in your airflow.cfg should be set to CeleryExecutor.

Then, in the celery section of the airflow.cfg, set the broker_url to point to your celery backend (e.g. redis://your_redis_host:your_redis_port/1). Point celery_result_backend to a sql database (you can use the same as your main airflow db).

Then, on your worker machines simply kick off airflow worker and your jobs should start on the two machines.

Matthijs Brouns
  • 2,299
  • 1
  • 27
  • 37
  • Are the broker_url of two machines' airflow.cfg the same? I have a question that is the broker_url in airflow.cfg is this machine send tasks to and other machine to get tasks from? – Fewfy Jul 24 '17 at 08:09
  • 1
    The broker_url is the same for both hosts. Airflow assumes the configuration file is the same for all instances of an airflow cluster. The broker_url does not necessarily point to a host running airflow, but instead points to a celery backend (i.e. Redis or RabbitMQ) – Matthijs Brouns Jul 24 '17 at 08:34
  • Refer [this](https://stackoverflow.com/questions/44979811/adding-extra-celery-configs-to-airflow) to know how to set `Celery` configurations for `Airflow` – y2k-shubham Jul 19 '18 at 07:49
  • Link 404s now. Should either be updated or more context should be added to answer. – Slater Victoroff Dec 10 '18 at 19:43
  • @MatthijsBrouns Does this mean that, if I have a celery configured + 3 workers, then (in the web interface) start a task, this task will run in triplicate (3 copies of the task each in a worker)? – JonyD May 17 '19 at 14:45
  • @JonyD no each task will run only on one worker. If there are multiple tasks however which do not depend on each other they will get distributed over the 3 workers – Matthijs Brouns May 19 '19 at 09:53
  • @MatthijsBrouns - check if this is the latest version of your link ? https://airflow.apache.org/docs/apache-airflow/stable/executor/celery.html – fixxxer May 27 '21 at 10:06