Spark executor on yarn-client does not take executor core count configuration.

Question

Irrespective of the spark executor core count, yarn container for the executor does not use more than 1 core.

Here is the spark configuration: spark.executor.cores=2 and spark.executor.instances=5. Number of executors started in yarn is 5 but each of them only uses 1 core. Spark properties in driver UI does show executor core as 2 but in Yarn UI, it shows using 1 core per container(executor). — Santosh Kumar, Oct 21 '15 at 00:00

score 10 · Answer 1 · answered Oct 21 '15 at 05:30

10

YARN is showing 1 core per executor irrespective of spark.executor.cores because by default DefaultResourceCalculator is used. It considers only memory.

public int computeAvailableContainers(Resource available, Resource required) {
// Only consider memory
return available.getMemory() / required.getMemory();
  }

Use DominantResourceCalculator, It uses both cpu and memory.

Set below config in capacity-scheduler.xml

yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

More about DominantResourceCalculator

answered Oct 21 '15 at 05:30

banjara

3,800
3
38
61

Thank You Shekhar. It really helped. – Santosh Kumar Oct 21 '15 at 09:43
Man, thanks a lot. You've saved me a ton of time =) – AlexanderLedovsky Jul 05 '22 at 12:03

Spark executor on yarn-client does not take executor core count configuration.

1 Answers1

Linked