Trying to update Airflow Variable within python operator in GCP

Question

I am trying to update the airflow variable within a python operator. Outside the operator, I can update, but within operator, getting some error

Variable.update('inv_test_var', '2023-06-28 04:12:00') # it works
def py_function():

  print("**************************************************")
  #new_var = Variable.get('inv_test_var')
  Variable.update('inv_test_var','2023-06-28 04:20:00') # not worked
  print("**************************************************")

within a dag, I have called this function py_function

ERROR - Google Cloud API Call Error (NotFound): Secret ID inv_test_var not found.

Not sure, why in operator, its not getting updated. Can anyone please let me know?

from the error looks like it looking for a variable in the secret backend and it variable does not exist in secret backend. have you configured the google secret backend? — Pankaj Singh, Jun 28 '23 at 17:03
I am not sure on that Pankaj.. But my question is, Is it possible to access the airflow variable within Operators in Airflow. If yes, how do we can achieve it? or is it mandated to configure google secret backend to access the Airflow variable.. please let me know — Mani Shankar.S, Jun 29 '23 at 06:24
its working.. I haven't made any changes.. it seems like Airflow used to look for secret first, whenever we are using Variable.get / Variable.update in operator.. But subsequently the airflow variable is getting updated.. but we used to get that error in the logs.. this mentioned code will work... — Mani Shankar.S, Jun 29 '23 at 09:39
yes, it works with both `Variable.set(...)` as well as `Variable.update(...)` — Pankaj Singh, Jun 29 '23 at 10:14
> is it mandated to configure google secret backend to access the Airflow variable No, but from the error log looks like we have google secret backed so I thought maybe you are getting error because of that — Pankaj Singh, Jun 29 '23 at 10:16

score 0 · Answer 1 · answered Jun 29 '23 at 18:44

Google Cloud API Call Error (NotFound): Secret ID inv_test_var not found is a log statement in Google secret backend code in the Airflow codebase.

Otherwise both

Variable.set(key="some_key", value="some_value") as well as Variable.update(key="some_key", value="some_value) work inside operator.

Trying to update Airflow Variable within python operator in GCP

1 Answers1