11

So I have a spark standalone server with 16 cores and 64GB of RAM. I have both the master and worker running on the server. I don't have dynamic allocation enabled. I am on Spark 2.0

What I dont understand is when I submit my job and specify:

--num-executors 2
--executor-cores 2

Only 4 cores should be taken up. Yet when the job is submitted, it takes all 16 cores and spins up 8 executors regardless, bypassing the num-executors parameter. But if I change the executor-cores parameter to 4 it will adjust accordingly and 4 executors will spin up.

Sajjad Hossain
  • 111
  • 1
  • 9
theMadKing
  • 2,064
  • 7
  • 32
  • 59

1 Answers1

23

Disclaimer: I really don't know if --num-executors should work or not in standalone mode. I haven't seen it used outside YARN.

Note: As pointed out by Marco --num-executors is no longer in use on YARN.

You can effectively control number of executors in standalone mode with static allocation (this works on Mesos as well) by combining spark.cores.max and spark.executor.cores where number of executors is determined as:

floor(spark.cores.max / spark.executor.cores)

For example:

--conf "spark.cores.max=4" --conf "spark.executor.cores=2"
zero323
  • 322,348
  • 103
  • 959
  • 935
  • Thanks, yeah the reason we are doing a standalone is we have 3 streaming jobs that feed one another and for testing we are just using 1 box for now. So we are also just doing --master local[4] – theMadKing Sep 08 '16 at 21:45
  • 1
    I tried to use the same "--num-executors" property with my standalone environment and It did not work and when i tried this with YARN cluster it worked fine. Looks like it is only for YARN deployment. You can control the number of executors in standalone deployment only using the "spark.cores.max" and "spark.executor.cores" property. – Hokam Sep 09 '16 at 07:06
  • --num-executors was only a YARN parameter. Take a look at this question http://stackoverflow.com/questions/32621990/what-are-workers-executors-cores-in-spark-standalone-cluster/32628057#32628057 – Marco Sep 09 '16 at 08:26
  • 2
    Annoying part is that `--num-executors` is still randomly referenced by docs, has some test coverage and is happily accepted as an option. There is still a lot of cleaning to be done. – zero323 Sep 09 '16 at 12:10
  • Perfect Answer ! you have to set either spark.cores.max or spark.deploy.defaultCores – fattah.safa Jun 16 '17 at 01:12
  • On spark-2.2.2.5, standalone mode using --num-executors and --execuotr-cores working as expected. – VirtualLogic May 29 '19 at 17:40
  • Apparently spark standalone ignores the spark.cores.max setting. Tested on single node standalone setup. – Darshan Jun 28 '19 at 07:31