Apache Spark does not see all the ram of my machines

Question

I have created a Spark cluster of 8 machines. Each machine have 104 GB of RAM and 16 virtual cores.

I seems that Spark only sees 42 GB of RAM per machine which is not correct. Do you know why Spark does not see all the RAM of the machines?

enter image description here

PS : I am using Apache Spark 1.2

Why do I have 3 downvotes? Am I missing something? – poiuytrez Feb 11 '15 at 08:21 — poiuytrez, Feb 11 '15 at 08:21

score 2 · Accepted Answer · edited May 23 '17 at 10:28

2

Seems like a common misconception. What is displayed is the spark.storage.memoryFraction : https://stackoverflow.com/a/28363743/4278362

edited May 23 '17 at 10:28

Community

1
1

answered Feb 11 '15 at 00:04

Sietse

201
1
4

score 0 · Answer 2 · answered Feb 10 '15 at 20:52

0

Spark makes no attempt at guessing the available memory. Executors use as much memory as you specify with the spark.executor.memory setting. Looks like it's set to 42 GB.

answered Feb 10 '15 at 20:52

Daniel Darabos

26,991
10
102
114

spark.executor.memory was set to 83971m. – poiuytrez Feb 11 '15 at 17:52

Apache Spark does not see all the ram of my machines

2 Answers2