Can two executors / drivers from different Spark applications run on same node in cluster mode?

Question

I read an article in Medium which claims that the number of executors + 1 (for driver), should be a multiple of 3, to efficiently utilize the core on a machine (16 cores, in this case, i.e, 5 per executor and 1 will be reserved for OS and node manager)

I am unable to validate this statement using experimenting on the cluster due to practical reasons. Did anybody try this? or have reference to code/documentation stating Yarn nodes will/not share cluster resources between another Spark application?

it is too broad question to answer... it depends on many factors ... like what kind scheduling enabled (capacity,fairscheduler) in yarn or what kind of tuning like (number of executors per each job) or is dynamic allocation enabled?? ... obviously yarn will share cluster resources between spark applications... but again it depends whether resources are left to share or not... — kavetiraviteja, Aug 24 '20 at 13:23
I agree that it depends on various factors but I would like to know if it is possible et all, If so, under what conditions. Per say, of it is possible with capacity or fair scheduler?? I did not get how does dynamic allocation play role in possibility. I am taking about a feasibility here so assume the resources are sufficient, nodes can accommodate multiple executors, multiple spark applications running parallelly — Saiteja Parsi, Aug 25 '20 at 10:50
It is possible under all conditions you state, unless there are no resources to allocate. Spark Stand alone is different to YARN. — thebluephantom, Aug 25 '20 at 13:38

thebluephantom · Answer 1 · 2020-08-18T09:21:15.877

0

It's a big question, but in short - basing on the title and YARN in the text:

You get resources allocated by YARN that you requested via Spark(submit).
A Node has many Executors.
You cannot share an Executor at the same time, but the Executor can be relinquished if YARN Dynamic Resource Allocation is in effect, after a Stage has completed.
As a Node has many Executors, many Spark Apps can run their Tasks concurrently on the same Node, Worker, if they were granted those resources.

edited Aug 18 '20 at 09:21

answered Aug 17 '20 at 20:11

thebluephantom

16,458
8
40
83

Sorry, that didn't answer my question – Saiteja Parsi Aug 17 '20 at 20:21

Can two executors / drivers from different Spark applications run on same node in cluster mode?

1 Answers1