7

Running Spark Application with local mode, I used the command, such as:

spark-submit --master local[*] my_spark_application.py

In this case, dose which mean that my application used all memory of my local computer? Are the other parameters, like driver-memory and executor-memory still working?

Garren S
  • 5,552
  • 3
  • 30
  • 45
Ivan Lee
  • 3,420
  • 4
  • 30
  • 45

2 Answers2

11

Setting driver memory is the only way to increase memory in a local spark application.

"Since you are running Spark in local mode, setting spark.executor.memory won't have any effect, as you have noticed. The reason for this is that the Worker "lives" within the driver JVM process that you start when you start spark-shell and the default memory used for that is 512M. You can increase that by setting spark.driver.memory to something higher, for example 5g" from How to set Apache Spark Executor memory

Community
  • 1
  • 1
Garren S
  • 5,552
  • 3
  • 30
  • 45
1

It depends on which virtual environment tool is used , if you install just spark without virtual environment (like docker) it takes your full your local memory , So , I recommend to use spark inside docker container which it takes about 220MB (default)

First install docker ;

then , Create container ;

install spark into container .

Alex
  • 29
  • 8
  • Hi, I used virtual machine to create ubuntu environment, and allocated the fixed memory for this system, on which spark is running. In this case, is spark application using all memory? – Ivan Lee May 15 '17 at 03:21
  • No , When you run spark app like my_spark_application.py it takes just executor-memory not all memory exactly Ram memory of your vertual machine – Alex May 15 '17 at 03:25
  • Wether I can not allocate fixed memory for the spark application in this case unless using the parameter of executor-memory. Is there default value of the parameter if I do not configure this? – Ivan Lee May 15 '17 at 03:58
  • 1
    Allocate driver memory as well when running in local mode. I've found it to be more useful than executor memory albeit slightly disingenuous in my opinion. – Garren S May 15 '17 at 04:06
  • Hi @Garren, I found same issue with you. But, I am not sure. Could you add new answer to describe this? thanks! – Ivan Lee May 15 '17 at 04:17