1

I want to submit spark job to azure hdInsights cluster from airflow, I don't want to use livy as it doesn't accumulate logs on airflow. Is it possible to do remotely submit job. SSH is 1 option but if job is long running it might break connection. Is there any other option? Note - Airflow cluster is remote cluster, it's not colocated with spark cluster.

vermaji
  • 539
  • 2
  • 10
  • have a look at [this](https://stackoverflow.com/a/53344713/3679900) and also the 4th point [here](https://stackoverflow.com/a/54092691/3679900) – y2k-shubham Oct 04 '19 at 10:43
  • Already checked with both of them, None of them is useful, as Azure Hdinsights cluster doesn't expose those ports. – vermaji Oct 04 '19 at 13:09

0 Answers0