Writing to Airflow Logs

Question

One way to write to the logs in Airflow is to return a string from a PythonOperator like on line 44 here.

Are there other ways that allow me to write to the airflow log files? I've found that print statements are not saved to the logs.

Possible duplicate of [Adding logs to Airflow Logs](https://stackoverflow.com/questions/40120467/adding-logs-to-airflow-logs) — SergiyKolesnikov, Sep 28 '18 at 14:23

score 84 · Accepted Answer · edited Mar 07 '19 at 20:29

84

You can import the logging module into your code and write to logs that way

import logging

logging.info('Hello')

Here are some more options

import logging    

logging.debug('This is a debug message')
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')

edited Mar 07 '19 at 20:29

Tigerjz32

4,324
4
26
34

answered Jan 12 '17 at 18:24

J.Fratzke

1,415
15
23

9

Does this work for anything other than "info"? I got it to work when my warning level was "info", but then I tried setting the airflow.cfg logging_level to WARNING and ERROR and tried using logging.warning('my message') and logging.error('my message'), and neither or these two things worked. Sadly, the airflow logs are really verbose and I don't want to spam my logs with redundant information. I prefer to just write out custom application messages with "INFO" and not get all the clutter in the log. Not sure if this isn't supported yet or not in Airflow 1.10.1? – Mark Dec 06 '18 at 04:36
2

I am still trying to figure out why I might use logging.info over printing (for simple debugging purposes or to read information about a DAG exectution). – theStud54 Oct 23 '19 at 04:52
Good point @theStud54 - to support that, in my case, print statements end up as INFO-level log statements anyways. – cellepo Jul 24 '20 at 17:15
2

My "logging.info" is not shown in the log. Print-statements are on the other hand – CutePoison May 25 '21 at 12:42

score 18 · Answer 2 · answered May 12 '20 at 08:02

18

You might use airflow logger

import logging


logger = logging.getLogger("airflow.task")
logger.error("Your custom error")

answered May 12 '20 at 08:02

Heedge

181
1
4

1

Where would the logs go in this case? What happens when I'm logging from an independent python script which is not connected to any DAG or Task? – Abhishek Dalvi Oct 12 '21 at 08:29
1

You ask Airflow to provide a logger configured by Airflow by calling `logging.getLogger("airflow.task")`. Logs go to a directory specified in `airflow.cfg` file – Heedge Oct 15 '21 at 10:18
1

Airflow's preconfigured logger is returned only for a process started by airflow and all threads started by the main process. It doesn't work for a new process(a new default logger is created for a new process). But it does work for a new thread started by the same process. – Heedge Oct 15 '21 at 10:21

score 5 · Answer 3 · answered Jul 07 '21 at 16:45

There's a logger mixin class you can use:

https://github.com/apache/airflow/blob/main/airflow/utils/log/logging_mixin.py

from airflow.utils.log.logging_mixin import LoggingMixin

LoggingMixin().log.info("Hello")

Then in airflow logs you'll see:

[2021-07-07 15:55:42,370] {{logging_mixin.py:112}} INFO - Hello

Writing to Airflow Logs

3 Answers3

Linked