3

I have added a custom value to conf/spark-defaults.conf but that value is not being used.

stephen@ubuntu:~/spark-1.2.2$ cat conf/spark-defaults.conf
spark.akka.frameSize    92345678

Now let us run my program LBFGSRunner

sbt/sbt  '; project mllib; runMain org.apache.spark.mllib.optimization.LBFGSRunner spark://ubuntu:7077'

Notice the following error: the conf setting was not being used:

[error] Exception in thread "main" org.apache.spark.SparkException: 
Job aborted due to stage failure: Serialized task 0:0 was 26128706 bytes, 
which exceeds max allowed: spark.akka.frameSize (10485760 bytes) - 
reserved (204800 bytes). Consider increasing spark.akka.frameSize
 or using broadcast variables for large values
WestCoastProjects
  • 58,982
  • 91
  • 316
  • 560
  • Is it the only property which doesn't work? And all other modified values are working correctly? Try to see it on http://:4040/ environment tab and see whether it reflects modified value of your property. Thanks, Sumit – Sumit May 01 '15 at 10:40
  • @javadba have you checked your server_host:4040 tab environment to see if this variable is corrected set there? Is the conf folder into the classpath of your server execution? – dirceusemighini Jul 20 '15 at 18:23

1 Answers1

1

Note: Working In Linux Mint.

If you are setting properties in spark-defaults.conf, spark will take those settings only when you submit your job using spark-submit.

file: spark-defaults.conf

spark.driver.extraJavaOptions      -Dlog4j.configuration=file:log4j.properties -Dspark.yarn.app.container.log.dir=app-logs -Dlogfile.name=hello-spark
spark.jars.packages                 org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1,org.apache.spark:spark-avro_2.12:3.0.1

If you want to run your job in development mode.

spark = SparkSession.builder \
    .appName('Hello Spark') \
    .master('local[3]') \
    .config("spark.streaming.stopGracefullyOnShutdown", "true") \
    .config("spark.jars.packages", "org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1") \
    .getOrCreate()
Pramod Kumar Sharma
  • 560
  • 6
  • 18
  • for more detail implementation check this link https://stackoverflow.com/questions/57862801/spark-shell-add-multiple-drivers-jars-to-classpath-using-spark-defaults-conf/65799134#65799134 – Pramod Kumar Sharma Feb 11 '21 at 06:27