Dagster sensor to check for new records in a table

Question

I have 2 tables where 2nd is dependent on 1st. Whenever new records are added in 1st, I want to run a dagster job. I came across sensors but I am not sure if my requirement can be fulfilled using the functionality they provide. Any ideas?

score 1 · Answer 1 · answered Dec 13 '22 at 16:59

1

Asset sensors definitely apply here. If you model your tables as assets, then when the first table is materialized the asset sensor will trigger, which can yield a RunRequest to trigger the Dagster job you want to run.

If you don't have control over the code that defines the table, you could also use a normal sensor, and just have it query your table for new records to determine whether to run your Dagster job.

answered Dec 13 '22 at 16:59

zyd

833
7
16

Thanks for your help! I will explore these options. – Abi Dec 14 '22 at 22:40

Dagster sensor to check for new records in a table

1 Answers1