How do I load new Airflow configuration?

Question

I'm using Airflow for my ETL and now I want to make some changes (donot_pickle=True) in my Airflow config. How do I load the new config? Using:

docker-compose up -d --force-recreate --build

Just updates the container config. Am I forced to recreate the whole thing?

Does this answer your question? [setting up s3 for logs in airflow](https://stackoverflow.com/questions/44780736/setting-up-s3-for-logs-in-airflow) — Francisco Maria Calisto, May 19 '20 at 16:00
Where do I run "export AIRFLOW__CORE__DONOT_PICKLE=True"? Almost there... — Rodrigo Augusto Sarmento, May 19 '20 at 17:26

score 4 · Answer 1 · answered May 20 '20 at 15:41

I will assume you're using puckel's docker-compose deployment (https://github.com/puckel/docker-airflow) because it's the most popular one and you didn't specify any.

You can share any airflow configuration variable such as AIRFLOW__CORE__DONOT_PICKLE as an environment variable on each of the services specified inside your docker-compose.yml files.

Let's see how that can be accomplish for the scheduler service of the celery-executor version, for example:

scheduler:
    ...
    environment:
        - LOAD_EX=n
        - EXECUTOR=Celery
        - AIRFLOW__CORE__DONOT_PICKLE=True
        - ...
    ...

Each environment value you specify in there will be made available to the service the next time its corresponding docker container gets restarted.

How do I load new Airflow configuration?

1 Answers1