5

I'm new with Airflow and I cannot understand how to reload an operator/plugin when I change its code. I'm using the LocalExecutor and an external database (MySql). I've tried to restart the webserver and the scheduler but I still get the same error when importing dags:

    File "/home/ec2-user/airflow/dags/extractor.py", line 2, in <module>
    from airflow.contrib.operators.emr_spark_plugin import EmrSparkOperator
ImportError: No module named emr_spark_plugin
Erica
  • 1,608
  • 2
  • 21
  • 32
  • Restarting the scheduler should work. Are you sure that you are calling the operator correctly? – skozz Jul 06 '18 at 14:31
  • My mistake. I should have called it as "from airflow.operators.emr_spark_plugin import EmrSparkOperator". Now restarting the scheduler and the webserver works. – Erica Jul 06 '18 at 14:44

2 Answers2

9

I know that this is super old, but for those still looking for a good solution, new versions of Airflow now handle this automatically.

https://airflow.apache.org/docs/stable/configurations-ref.html#reload-on-plugin-change

You will need to set reload_on_plugin_change = True in your airflow.cfg file under the [webserver] section.

Dharman
  • 30,962
  • 25
  • 85
  • 135
Nadim Younes
  • 800
  • 1
  • 6
  • 11
  • I'm running airflow locally with docker and this configuration doesn't seem to work for me. I need to restart webserver/scheduler for airflow to pick up plugins. any idea why? – Dominik Sajovic Feb 26 '23 at 00:58
6

Try restarting the web server and scheduler.

There isn't actually a good built-in way to do this, I recommend use systemd as proposed in this answer.

Mike
  • 2,514
  • 2
  • 23
  • 37