Need to access schedule time in DockerOperator in Airflow

Question

Need to access schedule time in airflow's docker operator. For example

t1 = DockerOperator(
task_id="task",
dag=dag,
image="test-runner:1.0",
docker_url="xxx.xxx.xxx.xxx:2376",
environment={"FROM": "{{(execution_date + macros.timedelta(hours=6,minutes=(30))).isoformat()}}"})

Basically, I need to populate schedule time as docker environment.

This might be connected to the fact that `environment` is not a templated field and thus the template will not work. See more on that here https://stackoverflow.com/questions/44855949/make-custom-airflow-macros-expand-other-macros — tobi6, Jul 31 '18 at 10:53

score 3 · Accepted Answer · answered Jul 31 '18 at 14:30

First macros only works if it is a template_fields. Second, you need to check which version of airflow you are using, if you are using 1.9 or below, it is likely it won't work for DockerOperator which @tobi6 mentioned in the comment, since template_fields only included command (https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/operators/docker_operator.py#L91)
However, 1.10 stable add environment there for DockerOperator. https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/operators/docker_operator.py#L103

See new answer describing how to extend templated fields – andscoop Aug 01 '18 at 04:23 — andscoop, Aug 01 '18 at 04:23

score 2 · Answer 2 · answered Aug 01 '18 at 04:22

2

As other poster mentioned, the DockerOperator in Airflow 1.9 is only expecting the command field to be templated, but it is fairly trivial to modify the templatable fields on an Operator.

from airflow.operators import DockerOperator
DockerOperator.template_fields = (command, environment)

And then you can instantiate your operator as you normally would.

answered Aug 01 '18 at 04:22

andscoop

939
5
8

So this line, you would it just add to the top of a DAG? Or is this inside a function? – tobi6 Aug 01 '18 at 06:09
global on a DAG file – andscoop Aug 01 '18 at 14:02

score 0 · Answer 3 · answered Apr 17 '19 at 22:56

A quick update :)

The environment and command parameters of the DockerOperator can be templated. I made an article about it: https://marclamberti.com/blog/how-to-use-dockeroperator-apache-airflow/

Also, don't hesitate to take a look at the source code to see which parameter can be templated or not: https://github.com/apache/airflow/blob/master/airflow/operators/docker_operator.py Have a good day

Need to access schedule time in DockerOperator in Airflow

3 Answers3

Linked