How do the yarn and spark parameters interplay together?

Question

There are parameters that decide the maximum, minimum and total of the memory and cpu that yarn can allocate via containers

example:

yarn.nodemanager.resource.memory-mb

yarn.scheduler.maximum-allocation-mb

yarn.scheduler.minimum-allocation-mb

yarn.nodemanager.resource.cpu-vcores

yarn.scheduler.maximum-allocation-vcores

yarn.scheduler.minimum-allocation-vcores

There are also spark side parameters that seemingly would control similar kind of allocations:

spark.executor.instances

spark.executor.memory

spark.executor.cores

etc

What happens when the two set of parameters are infeasible according to the bounds set by the other. For example: What if yarn.scheduler.maximum-allocation-mb is set to 1G and the spark.executor.memory is set to 2G? Similar conflicts and infeasibilities could be imagined for the other parameters as well.

What happens in such cases? And, what is the suggested way to set these parameters?

How do I try out and see the "what is the suggested way to set these parameters?". That's an asinine reason to vote to close a question. — figs_and_nuts, Nov 27 '22 at 15:58
Were you able to understand my comment completely? "What is the suggested way to set these parameters?". Also, I didn't call your comments silly. I used a much stronger word — figs_and_nuts, Nov 28 '22 at 06:08

score 2 · Answer 1 · answered Nov 27 '22 at 07:07

When running Spark on YARN, each Spark executor runs as a YARN container

So take spark.executor.memory as an example:

If spark.executor.memory is 2G and yarn.scheduler.maximum-allocation-mb is 1G, then your container will be OOM killer
If spark.executor.memory is 2G and yarn.scheduler.minimum-allocation-mb is 4G, then your container is much bigger than needed by the Spark application

Suggestions for setting parameters depend on your hardware resources and other services running on this machine. You can try to use the default value first, and then make adjustments by monitoring machine resources

thebluephantom · Answer 2 · 2022-11-29T22:11:43.777

This excellent https://community.cloudera.com/t5/Support-Questions/Yarn-container-size-flexible-to-satisfy-what-application-ask/m-p/115458 and Difference between `yarn.scheduler.maximum-allocation-mb` and `yarn.nodemanager.resource.memory-mb`? should give you the basics. Additionally, here is a good SO-related answer Spark on YARN resource manager: Relation between YARN Containers and Spark Executors.

TL;DR

As you are not talking about Kubernetes, then YARN as Resource / Cluster Mgr allocates Executors with needed resouces, based on Spark params / defaults that are allocated based on those YARN params for the Containers.
1 Container = 1 Executor. Some state incorrectly, 1 Container N Executors, not so.
There is minimum allocation and max allocation of resources, based on those YARN params. So,YARN will provide Executors with some wastage of resources, if it can - or of restricted size.
If non-dynamic YARN resource allocation, then Apps start with less resources, else there will be a wait to get all resources, and those acquired are not available for others.
There is also a fair scheduler for more smooth, uniform throughput for many concurrent apps.

How do the yarn and spark parameters interplay together?

2 Answers2