9

I have setup a spark (1.6)standalone cluster. have 1 master and added 3 machines under conf/slaves file as workers. Even though I have allocated 4GB memory to each of my workers in spark, why does it use only 1024MB when the application is running? I would like for it use all 4 GB allocated to it. Help me figure out where and what I am doing wrong.

Below is the screenshot of the spark master page (when the application is running using spark-submit) where under the Memory column it shows 1024.0 MB used in brackets next to 4.0 GB.

I also tried setting --executor-memory 4G option with spark-submit and it does not work (as suggested in How to change memory per node for apache spark worker).

These are the options I have set in spark-env.sh file

export SPARK_WORKER_CORES=3

export SPARK_WORKER_MEMORY=4g

export SPARK_WORKER_INSTANCES=2

enter image description here

Community
  • 1
  • 1
B1K
  • 198
  • 1
  • 2
  • 9

3 Answers3

5

One other workaround is try setting the following parameters inside the conf/spark-defaults.conf file:

spark.driver.cores              4
spark.driver.memory             2g
spark.executor.memory           4g

Once you set the above (only the last line in your case) shut down all the workers at once and restart them. It is better to initialize the executors memory this way since your problem seems to be that no executor can allocate all the available memory of his worker.

raschild
  • 198
  • 1
  • 9
  • Where it needs to be set matters. This has to be in the master (driver???) node. In my case both master and driver are the same. Once I set the spark.executor.memory as 3g in spark-defaults.conf, the memory used increased to 3g. May be it was in the documentation and I missed it. But thanks for the start. So I am going to mark this as answer. – B1K Apr 08 '16 at 14:54
  • You are welcome. From my understanding passing the `spark.executor.memory` parameter through the script initializes the executor on the worker with the specified memory. While the `--executor-memory` flag on the driver defines how much memory must be reserved from each executor so that the submitting job can be executed. – raschild Apr 08 '16 at 19:42
1

The parameter you are looking for is executor-memory try providing that to your spark application when you start it.

--executor-memory 4g

When you set worker-memory to 4g then the biggest executor you run on that worker is of 4g. PS: you can have different configurations (each having different worker memory).

Manas
  • 519
  • 4
  • 14
  • Like I had mentioned in my original post, I did try --executor-memory 4G option when I start my application and still there is no change. Thanks. – B1K Apr 04 '16 at 19:32
0

create a file called spark-env.sh in spark/conf directory add this line SPARK_EXECUTOR_MEMORY=4g

Mohamed Thasin ah
  • 10,754
  • 11
  • 52
  • 111