0

I have 32 GB of physical memoryand my input file size about 30 MB, I try to submit my spark job in yarn client mode using the below command 

spark-submit --master yarn --packages com.databricks:spark-xml_2.10:0.4.1 --driver-memory 8g ericsson_xml_parsing_version_6_stage1.py

and my executor space is 8g, but get the below error anyone please help me to configure the java heap memory. I read about the --driver-java-options using command line but I don't know how to set java heap space using this option.

Anyone please help me out.

java.lang.OutOfMemoryError: Java heap space

enter image description here

EddyTheDove
  • 12,979
  • 2
  • 37
  • 45
Jayabal
  • 11
  • This might help you http://stackoverflow.com/questions/21138751/spark-java-lang-outofmemoryerror-java-heap-space – BruceWayne Mar 08 '17 at 07:04

1 Answers1

0

Did you try to configure executor memory as well? like this: "--executor-memory 8g"

Hlib
  • 2,944
  • 6
  • 29
  • 33
  • yes, I have used the below also, but same error exists. spark-submit --master yarn --packages com.databricks:spark-xml_2.10:0.4.1 --driver-memory 8g --executor-memory 8gericsson_xml_parsing_version_6_stage1.py – Jayabal Mar 07 '17 at 09:17
  • than maybe indeed you don't have enough space. input file size 30MB does not guarantee that RDDs will not take 1000 times more space. – Hlib Mar 07 '17 at 09:49
  • I don't know how to fix this issue, Please give ideas to fix it. – Jayabal Mar 07 '17 at 10:02
  • analyze your code (maybe share it here), or try to work with even smaller input file, increase driver/executor memory – Hlib Mar 07 '17 at 13:29