-1

I am currently running Airflow through Ubuntu WSL on my PC, which is working great. However I am setting up a pipeline which will need to be running constantly (24/7), so I am looking for ideas and recommendations on what to run Airflow on. I do not want to have my system on all the time obviously.

Surprisingly, I cannot find much information on this! It seems it is not discussed at length...

H_Empty
  • 346
  • 2
  • 8
  • **[1]** For Airflow deployment, I'd suggest *containerized* approach is best for long-term. You can refer to [`puckel/docker-airflow`](https://github.com/puckel/docker-airflow) for this. But even with this there are options like `ECS` / `Fargate` (Celery), `Kubernetes` (which is also an executor in Airflow). **[2]** For your Airflow workflow set-up, have a look at [this](https://stackoverflow.com/questions/57857780/howto-run-parallel-spark-job-using-airflow#comment102153201_57857780) – y2k-shubham Sep 26 '19 at 08:02

1 Answers1

1

It depends on your workload.

If you have few tasks to run you can just create a VM on any cloud provider (GCP, AWS, Azure, etc.) and install Airflow on it that would run 24x7.

If your workload is high you can user K8s (GKE, EKS, etc.) and install Airflow on it.

kaxil
  • 17,706
  • 2
  • 59
  • 78