Scheduling Spark for Nightly run batch - Run Every Night Like an ETL

Question

I have a Spark job which reads a source table, does a number of map / flatten / DATAFRAME operations and then stores the results into a separate table FROM TEMP TABLE we use for reporting. Currently this job is run manually using the spark-submit script. I want to schedule it to run every night.

is there any way to schedule spark job for batch processing similarly like an nightly batch ETL

@dassum: any other alternative than Cron JOBS. I never used before — Aryan, Aug 13 '19 at 17:20

score 0 · Accepted Answer · answered Aug 13 '19 at 17:24

0

There is no built-in mechanism in Spark that will help. A cron job seems reasonable. Other options are

answered Aug 13 '19 at 17:24

dassum

4,727
2
25
38

score 0 · Answer 2 · answered Aug 13 '19 at 21:17

If you know python you can check Airflow. Airflow allows one to regularly schedule a task like cron, but is additionally more flexible in allowing for certain tasks to depend on each other and makes it easy to define complex relations even in a large distributed environment. You can check below link

How to run Spark code in Airflow?

Scheduling Spark for Nightly run batch - Run Every Night Like an ETL

2 Answers2