Spark Job using more executors than allocated in jobs

Asked Jan 17 '19 at 21:25

Active Jan 23 '19 at 20:52

Viewed 150 times

I have following settings in my Spark job:

--num-executors 2 
--executor-cores 1 
--executor-memory 12G 
--driver memory 16G 
--conf spark.streaming.dynamicAllocation.enabled=false \
--conf spark.dynamicAllocation.enabled=false \
--conf spark.streaming.receiver.writeAheadLog.enable=false 
--conf spark.executor.memoryOverhead=8192 
--conf spark.driver.memoryOverhead=8192'

My understanding is job should run with 2 executors however it is running with 3. This is happening to multiple of my jobs. Could someone please explain the reason?

edited Jan 23 '19 at 20:52

asked Jan 17 '19 at 21:25

Ritesh K Bansal

[Spark Standalone Number Executors/Cores Control](https://stackoverflow.com/q/39399205/10465355) – 10465355 Jan 17 '19 at 21:52
We made sure to have dynamic allocation as false, so that job uses fix amount of resources. – Ritesh K Bansal Jan 23 '19 at 20:53

Spark Job using more executors than allocated in jobs

0 Answers0