Convert python script to Airflow PythonOperator(s)

Question

I have a working python script with runs from CronJob. I want to convert it to DAG with PythonOperator(s) as we now are converting to Airflow.

Say that I have functions: a(),b(),c(),d() And their execution order is : a->b->c->d

Lets say that the function codes are:

def a(): 
    print("Happy")

def b(): 
    print("Birthday")

def c(): 
    print("to")

def d(): 
    print("you!")

** This is just an example my code for all functions is more complex

I have this DAG:

args = {
    'owner': 'airflow',
    'start_date': airflow.utils.dates.days_ago(2),
    'schedule_interval': '0 10 * * *'
}

dag = DAG(dag_id='example', default_args=args)

a = PythonOperator(task_id='a', dag=dag)
b = PythonOperator(task_id='b', dag=dag)
c = PythonOperator(task_id='c', dag=dag)
d = PythonOperator(task_id='d', dag=dag)

a.set_downstream(b)
b.set_downstream(c)
c.set_downstream(d)

What I don't understand is where do I put the codes of a(),b(),c(),d() and where do I specify thier names in the execution of the PythonOperator.

You could say that I'm looking for a way to convert my Python script into Airflow as each function will be a separate operator.

I thought this should be very simple and basic but I didn't find any information about how to do that.

Be aware that tasks a, b c, and d could run on different workers. — dlamblin, Jun 14 '18 at 06:53

score 2 · Accepted Answer · answered Jun 10 '18 at 16:09

In the python operator, the python function that should be executed is passed into the operator. So you will want to pass a python_callable kwarg like so:

def do_a():
    print('running a')

a = PythonOperator(task_id='a', python_callable=do_a, dag=dag)

The source for the operators will usually document the params for them. Python operator docs

Convert python script to Airflow PythonOperator(s)

1 Answers1

Linked