Format Airflow logs in Json - Issue

Question

Following Format Airflow Logs in JSON and the guide mentioned in it, I copied log_config to airflow/config folder. But when I run the webserver or scheduler, I get 'module not defined' (init.py was defined in the config folder) And PYTHONPATH was set as airflow/config

Error was

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airflow/logging_config.py", line 40, in configure_logging
    logging_config = import_string(logging_class_path)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airflow/utils/module_loading.py", line 33, in import_string
    module = import_module(module_path)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'airflow.config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/bin/airflow", line 25, in <module>
    from airflow.configuration import conf
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airflow/__init__.py", line 47, in <module>
    settings.initialize()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airflow/settings.py", line 374, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airflow/logging_config.py", line 52, in configure_logging
    'Unable to load custom logging from {} due to {}'.format(logging_class_path,err), isinstance('airflow',dict), isinstance('airflow.config',dict), type('airflow.config')
ImportError: ("Unable to load custom logging from airflow.config.log_config.LOGGING_CONFIG due to No module named 'airflow.config'",

Now it returns, ImportError: Unable to load custom logging from airflow.config.log_config.LOGGING_CONFIG due to section/key [logging/logging_level] not found in config

I'm facing the same issue, were you able to resolve it? – Meny Issakov Dec 01 '20 at 16:52 — Meny Issakov, Dec 01 '20 at 16:52

score 2 · Answer 1 · answered Dec 07 '20 at 18:08

In case you haven't already figured it out. It looks like the code that loads the logging_level is looking for it to be in a section named logging in the airflow.cfg file

But, this attribute actually resides under core section by default So if you'll change your code to LOG_LEVEL: str = conf.get("core", "LOGGING_LEVEL").upper()
this should resolve for you, or you can add another [logging] section in your airflow.cfg. (requires some reordering)

score 1 · Answer 2 · edited Jan 25 '21 at 10:51

Follwing the same answer like Meny Issakov:

Solution:

Go to your file airflow.cfg

Include a below the [core] section the following code:

[logging]
logging_config_class = log_config.DEFAULT_LOGGING_CONFIG

Restart your scheduler

Reason of problem: The file airflow.cfg is missing a section called [logging].

Understanding the problem

At the time that the scheduler starts, it access the [core] section in the airflow.cfg and search the logging path.

And the information where the logs will be stored it's found in logging_config_class parameter.

However, even if we put logging_config_class = log_config.DEFAULT_LOGGING_CONFIG in the [core] section, the scheduler it's not gonna work either.

Why? It's because there's a mismatch between the log_config.py logging handlers and with the airflow.cfg.

In the logging handlers in the log_config.py they have the following conf.get to get the logging configurations:

LOG_LEVEL: str = conf.get('logging', 'LOGGING_LEVEL').upper()  
LOG_FORMAT: str = conf.get('logging', 'LOG_FORMAT')

The first parameter it's the section that will be scanned in the airflow.cfg file, but by default, there's no section called [logging] in the airflow.cfg file, and this causes the following error in the scheduler inicialization:

ImportError: Unable to load custom logging from log_config.DEFAULT_LOGGING_CONFIG due to section/key [logging/fab_logging_level] not found in config

This is the best answer for this issue, I spent like 3 hours on this issue, feeling stupid . — Mohammed Ali, May 30 '21 at 19:33

Format Airflow logs in Json - Issue

2 Answers2