How to check runtime of each function within a task in Airflow DAG?

Question

I have a simple DAG which goes like this:

start >> create_cluster >> run_code >> delete_cluster >> end

The run_code runs a python script which is stored in Google Cloud Storage. This python script runs multiple functions to produce the output.

I would like to know if the runtime of these functions can be fetched.

Please provide enough code so others can better understand or reproduce the problem. — Community, Jul 26 '22 at 23:38

score 0 · Answer 1 · answered Jul 27 '22 at 07:43

Assuming you are using the PythonOperator for the run_code task:

For your run_code task's function, you can simply add the following:

import time

def run_code_function():
    # Starting the function.
    start_time = time.time() 

    # Business Code Logic...

    # You can log it as well
    print("--- %s seconds ---" % (time.time() - start_time))
    return important_stuff

To investigate more about it, check here.

Printing these values will give you the result at the log file where you initialized your DAG.

How to check runtime of each function within a task in Airflow DAG?

1 Answers1