2

I run a scrip on spark/scala to collect a calculated results. The result data set is not too large however when I run the following:

Result.collect()

I get the following error:

#java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 10466"...
/usr/lib/spark/bin/spark-shell: line 41: 10466 Killed                         " $FWDIR"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"

I am not sure why I got this error for running scala I used the following command to increase my memory:

spark-shell driver-memory 8G --executor-memory 8G --executor-cores 4 --num-executors 5

Could you please help on this?

Thanks, Amir

Amir
  • 1,017
  • 4
  • 14
  • 32
  • 1
    This is great group of answers if you haven't checked it out: http://stackoverflow.com/questions/21138751/spark-java-lang-outofmemoryerror-java-heap-space. – Jason McD Feb 15 '16 at 06:30
  • Thank you. It was helpful.Actually increasing the partition size resoled the issue. – Amir Feb 15 '16 at 08:50

1 Answers1

0

Since you are doing a collect, you should look into spark.driver.maxResultSize also in your spark-shell you missed the -- while specifying the driver-memory.

AkhlD
  • 2,596
  • 2
  • 16
  • 15