2

Almost all of my DAGs will have a subset of repeating Operators. And because of my use case, it works out really well to create new wrapper Operators that combines multiple Operators in order to reduce boilerplate.

My question is how to go about combining them?

An example would be to query a database followed by sending a Slack message.

postgres_operator_task >> slack_operator_task -> query_then_slack_operator_task

drum
  • 5,416
  • 7
  • 57
  • 91
  • There are probably many different ways to do this. You can define a custom operator which combines the different operators by executing the instructions in the right seqience, or define a subdag (which might be better if you are concerned about error recovery). – absolutelydevastated Sep 07 '19 at 07:29
  • Refer to [Fusing operators together](https://stackoverflow.com/q/53308306/3679900) – y2k-shubham Sep 07 '19 at 11:48

1 Answers1

0

Check the idea of subdags.
Resources:

TobKed
  • 71
  • 1
  • 5
  • This is good. If I want to remove as much boilerplate as possible, I would like to wrap `SubDagOperator` + params into a custom operator. Is that an okay practice? – drum Sep 10 '19 at 01:03
  • Sounds good for me, however I've heard 'SubDagOperator' can cause some troubles (and will be improved in v2). Friend gave me an idea that you can write just another 'Operator' which combines other operators (so it's almost what you wrote :) ) – TobKed Sep 11 '19 at 17:11