From what I see, each driver occupies at least 1 CPU. So I would like to leave as much as resource for executors I don't want too many drivers running.
Is there much benefit or best practise when use one vs many drivers?
Asked
Active
Viewed 24 times
0
-
Similar question: https://stackoverflow.com/questions/43999388/how-to-submit-multiple-spark-applications-in-parallel-without-spawning-separate – Christophe Roussy Oct 28 '19 at 10:29
-
I think it is kind of different questions. I found this article which more likely answered my concern. https://www.waitingforcode.com/apache-spark-sql/multiple-sparksession-one-sparkcontext/read – mmdc Oct 31 '19 at 09:36