18

I am able to access the macros in python code like below:

partition_dt = macros.ds_add(ds, 1)

But i am not able to figure out how to get hold of the ds variable itself which seemingly can only be accessed in templates. Any pointers?

Pratik Khadloya
  • 12,509
  • 11
  • 81
  • 106
  • Maybe you need to [provide the context](http://stackoverflow.com/a/40554473/2646526)? See similar example [here](http://stackoverflow.com/a/36754930/2646526). – heenenee Apr 04 '17 at 00:42
  • can you clarify what partition_dt is? is it a field in a task instantiation? – jhnclvr Apr 06 '17 at 15:30

2 Answers2

34

I assume you want to call one of the default variables built-in AirFlow ds - the execution date as YYYY-MM-DD

To call just ds, you can do: 

EXEC_DATE = '{{ ds }}'

To call what you wanted - macros.ds_add:

EXEC_DATE = '{{ macros.ds_add(ds, 1) }}'

And load it this way: 

T1 = BashOperator(\
        task_id='test_ds',
        bash_command='echo ' + EXEC_DATE
        dag=DAG)

In case you want to format it (like I had to), you can do:

EXEC_DATE = '{{ macros.ds_format(macros.ds_add(ds, 1), "%Y-%m-%d", "%Y%m%d") }}'
Oleg Yamin
  • 1,434
  • 1
  • 12
  • 13
  • 3
    where can I call a macro, if I call '{{ ds }}' inside dag it doesn't work but if I call it in any templated fields of operator it works. is this defined to work like that or am I doing anything wrong – Shahbaz Ali Nov 22 '19 at 05:49
  • As of airflow 2.3.3 'execution_date' is replaced by 'logical_date'. Also 'ds' is NOT time that DAG executed but rather the start of the DAG's period as explained here: https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html?highlight=pass%20data#data-interval. For instance, if a DAG has a period of (midnight August 9 - midnight August 10), it will execute after this period, namely August 11 00:00:01. Its logical date is (midnight August 9), which is different than time it is executing on. – Diana Vazquez Romo Aug 12 '22 at 18:40
5

Short answer, ds and macros variables can only be accessed through template as they only exists during execution and not during python code parsing (when the dag is loaded by airflow).

The question is really similar to this one : execution_date in airflow: need to access as a variable and I try to explain the difference between the 2 steps and how to have the execution date in a variable in the last answer : https://stackoverflow.com/a/45725005/3756307

Babcool
  • 2,103
  • 1
  • 17
  • 15
  • They can also be passed to python_callable functions as used in the python family of operators ( PythonOperator BranchPythonOperator, ShortCircuitOperator), e.g. see https://stackoverflow.com/a/40554473/1335793 – Davos Dec 04 '17 at 07:26