3

When using spark-submit to submit a Spark app to Yarn, I can pass java options to the driver via the --driver-java-options, for example: 

spark-submit --driver-java-options "-Dlog4j.configuration=file:///conf/log4j.properties" ...

How do I achieve the same when submitting via SparkLauncher? In particular, is there a way to achieve it with Spark 1.4?

Reddy
  • 8,737
  • 11
  • 55
  • 73
mitchus
  • 4,677
  • 3
  • 35
  • 70

1 Answers1

6

Not familiar with SparkLauncher but from looking at the code it appears you can pass configuration with setConf(). In this if you add the property SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS this should have the same effect.

For example

Process spark = new SparkLauncher()
     .setAppResource("/my/app.jar")
     .setMainClass("my.spark.app.Main")
     .setMaster("local[*]")
     .setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS, "-Dmy.property=someval")
     .launch();
Jem Tucker
  • 1,143
  • 16
  • 35
  • 2
    Are you sure ? From https://spark.apache.org/docs/latest/configuration.html "Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file. " – mathieu Jun 09 '17 at 12:33
  • I don't think the question is referring to client mode, tbh functionality of SparkLauncher has probably changed between version 1.4 and 2.1 (the version linked in your comment) – Jem Tucker Jun 09 '17 at 12:43