0

I'm working on a machine learning application ( using pycharm ) and everything works perfectly fine.

But spark throws this java heap error when i decided to change the 'HandlInvalid' param.

Exception in thread "dispatcher-event-loop-1" java.lang.OutOfMemoryError: Java heap space
.....
.....
ERROR Utils: Uncaught exception in thread driver-heartbeater
java.lang.OutOfMemoryError: Java heap space
.....
.....
py4j.protocol.Py4JJavaError: An error occurred while calling o6473.fit.
: java.lang.OutOfMemoryError: Java heap space

And when i try to change th driver.memory i get this error:

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Invalid maximum heap size: -Xmx5g
The specified size exceeds the maximum representable size.

here's my spark session :

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Ope inp').master('local')\
    .config("spark.driver.memory", "5g")\
    .enableHiveSupport().getOrCreate()

quoted from apache.spark.config :

Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-memory command line option or in your default properties file

how can I set the driver.memroy and executor.memory config in files properties so i dont have to set them everytime i'm dubugging the application

benaou mouad
  • 392
  • 3
  • 10

1 Answers1

0

I figured it out,

The problem were solved thanks to 64 bit JVM.First I was running a 32 JVM which doesnt allow specific params of limit heap size.

to figure out which JVM you have . try the second answer [https://stackoverflow.com/a/2062263/7096004][1]

benaou mouad
  • 392
  • 3
  • 10