spark standalone cluster: how to limit number of workers?

Question

Example:

spark standalone cluster add 1 machine(16 cpus) as worker

1. commit application not setting spark.executor.cores then it will create

   1 worker with 16 cores

2. commit with spark.executor.cores=15 then it will create 1 worker with 15 cores

3. commit with spark.executor.cores=5 then it will create 3 workers with 5 cores each worker

Now i want to limit cpu usage, only want 5 cores to be used

my question is:

how to set only 1 worker with 5 cores

Answer 1

Check this question please. I think it's a similar problem: https://stackoverflow.com/a/39400195/7779865

I think you can try to set spark.executor.cores=5 and spark.cores.max=5 or use spark.deploy.defaultCores=5 instead of spark.cores.max

Read about spark.max.cores here

Read about spark.deplo.defaultCores here