How to persist a value during airflow retry

Question

Everytime a dag triggers a task I would like to generate a value and keep the value during retry (if the task fails to run for any reason)

I try to declare a variable inside the dag (basically set current time to the variable) When the task fails and it retries, it set the variable with latest time again

y2k-shubham · Answer 1 · 2019-02-13T16:36:40.957

0

You can make use of on_failure_callback and on_retry_callback params of BaseOperator to push the value that you want to persist as xcom
Thereafter, you can retrieve that data from Xcom in any task across any DAG of your Airflow deployment, including the retry attempt of same / different task

Also have a look at

edited Feb 13 '19 at 16:36

answered Feb 13 '19 at 10:34

y2k-shubham

10,183
11
55
131

2

"including the retry attempt of same / different task" - do you have a working implementation of that? My experimentation, and sources [here](http://mail-archives.apache.org/mod_mbox/airflow-dev/201809.mbox/%3cCAERDx9eKTwU5Urq+pnq_8Q-hb-nHtFNq_xwkGGpxVo4MhB_Brg@mail.gmail.com%3e) and [here](https://github.com/apache/airflow/pull/6370), suggest that xcom is cleared before execution. – scubbo Aug 25 '20 at 02:43
did you find a way to do that in the end @scubbo ? – Pierre Jun 22 '21 at 23:17

How to persist a value during airflow retry

1 Answers1

Linked