19

With assistance from this answer https://stackoverflow.com/a/41730510/4200352 I am executing a python file.

I use PythonOperator and am trying to include the execution date as an argument passed to the script.

I believe I can access it somehow through kwargs['execution_date'].

The below fails

DAG.py

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta

import sys
import os
sys.path.append(os.path.abspath("/home/glsam/OmegaAPI/airflow/scripts/PyPer_ogi_simple"))
from update_benchmarks import *


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2018, 4, 23),
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG('run_pyPer', default_args=default_args)

update_BM_G027 = PythonOperator(
    task_id='update_BM_G027',
    python_callable=update_bmk,
    dag=dag,
    op_kwargs={
        'bmk_code': 'G027',
        'is_hedged': False,
        'from_date': kwargs['execution_date'],
    })

Do maybe i need to use this answer to get the date then XCOM it to the task? https://stackoverflow.com/a/36754930/4200352

Glenn Sampson
  • 1,188
  • 3
  • 12
  • 30

1 Answers1

48

This is really a bit confusing and not very well documented.

You are already using the PythonOperator.

Now just add the option

provide_context=True,

and extend your callable with a pointer, e.g.

update_bmk(bmk_code, is_hedged, **context)

Now, within your function you will have access to all information about the task, including the execution date like so:

task_instance = context['task_instance']
execution_date = context['execution_date']

To see a full reference of items in the context, see https://airflow.apache.org/docs/apache-airflow/stable/macros-ref.html

Those are the docs for macros, but you can use the items in the context dictionary.

villasv
  • 6,304
  • 2
  • 44
  • 78
tobi6
  • 8,033
  • 6
  • 26
  • 41
  • 3
    Great tip on the macros (loving them) and I did end up finding that provide_context thanks – Glenn Sampson Apr 30 '18 at 07:29
  • Is it possible to do this without extending the function? When we import function, we essentially need to wrap them in order to pass `macros` as arguments via the `PythonOperator`... which is a pain. – Newskooler Mar 22 '19 at 15:42
  • @Newskooler This seems to be a new question. Regardless, this might help you: https://stackoverflow.com/questions/54894418/how-to-pass-parameter-to-pythonoperator-in-airflow – tobi6 Mar 25 '19 at 10:08
  • 2
    @JoshHerzberg The link returns 404 – JohnnyHuo Aug 22 '19 at 22:37
  • 1
    https://airflow.apache.org/howto/operator/python.html?highlight=op_kwargs#passing-in-arguments New link – Josh Herzberg Aug 25 '19 at 23:39