3

I have an Emr cluster for spark with below configuration of 2 Instances.

r4.2xlarge
8 vCore

So my total vCores is 16 and the same is reflected in yarn Vcores

I have submitted a spark streaming job with parameters --num-executors 2 --executor-cores 5. So I was assuming it will use up 2*5 total 10 vcores for executors, but what it's doing only using 2 cores in total from the cluster (+1 for the driver)

.yarn memory aloc

And in spark, the job is still running with parallel tasks of 10 (2*5). Seems like it's just running only 5 threads within each executor core. tasks

I have read in different questions and in documentation --executor-cores uses actual vCores but here, it only running tasks as threads. Is my understanding correct here?

kavetiraviteja
  • 2,058
  • 1
  • 15
  • 35
sp_user123
  • 512
  • 3
  • 6
  • 28
  • can you provide spark-submit command ... and what is the memory configurations you have given for driver and executor.. did you enabled dynamic allocation? – kavetiraviteja Sep 16 '20 at 06:22
  • As you can see on yarn the allocation is constant..memory allocated 6g per executor and 4g for driver...there is nothing more in the spark submit command – sp_user123 Sep 16 '20 at 06:32
  • Looks very similar - https://stackoverflow.com/questions/38368985/spark-on-yarn-too-less-vcores-used – Gelerion Dec 04 '20 at 07:58

0 Answers0