How to define log path configuration to spark-submit?

Question

I am trying to pass a log path to spark-submit as follows:

spark-submit \
  --master local[*] \
  --conf -Dlogback.configurationFile=C:\Users\A661758\Desktop\logback.xml \
  target\scala-2.11\dataintegrationRepriseExistant-assembly-0.1.0-SNAPSHOT.jar \
  C:\Users\A661758\Desktop\Atos_Integration_Donnees_V0.1_2017\AC_LIMOGES_JSON SourceAcLimoges 20170526 L AC-LIMOGES

Output :

Warning: Ignoring non-spark config property: -Dlogback.configurationFile=C:\User s\A661758\Desktop\logback.xml log4j:WARN No appenders could be found for logger (dataIntegrationENTLea.Main$).

score 2 · Answer 1 · answered Jun 21 '17 at 14:44

2

Spark is rightly telling you that any argument passed to --conf needs to be a spark configuration key. What you're looking for is spark.driver.extraJavaOptions and spark.executor.extraJavaOptions:

spark-submit \
--master local[*] \
--conf "spark.driver.extraJavaOptions=-Dlogback.configurationFile=C:\Users\A661758\Desktop\logback.xml" \ 
C:\Users\A661758\dataIntegrationLea\target\scala-2.11\dataintegrationRepriseExistant-assembly-0.1.0-SNAPSHOT.jarC:\Users\A661758\Desktop\Atos_Integration_Donnees_V0.1_2017\AC_LIMOGES_JSON SourceAcLimoges 20170526 L AC-LIMOGES

Note you need to set both the driver and the executor location accordingly, as they may be different on each machine.

For more on spark configuration values, see Add jars to a Spark Job - spark-submit

answered Jun 21 '17 at 14:44

Yuval Itzchakov

146,575
32
257
321

@Mounir What isn't working? What are the error messages? Add them to your question. – Yuval Itzchakov Jun 21 '17 at 20:21
The specified path was not found : spark-submit --class dataIntegrationENTLea.Main --master local[*] --conf spark.eventLog.enabled=false --conf "spark.driver.extraJavaOptions=-Dlogback.configurationFile=C:\Users\A661758\Desktop\test\logback.xml" local:/C:/Users/A661758/dataIntegrationLea/target/scala-2.11/dataIntegrationENTLea-assembly-0.1.0-SNAPSHOT.jar C:/Users/A661758/Desktop/Atos_Integration_Donnees_V0.1_2017/AC_LIMOGES_JSON SourceAcLimoges 20170526 L AC-LIMOGES – mham Jun 22 '17 at 08:52
Are you running this locally in your IDE? – Yuval Itzchakov Jun 22 '17 at 08:53
Yes, in my IDE it's woking – mham Jun 22 '17 at 09:05

score 2 · Answer 2 · edited Jun 20 '20 at 09:12

It won't work for local[*] as it's too late to set up the single JVM for the driver and the single executor (that's again the driver).

--master local[*] --conf -Dlogback.configurationFile=C:\Users\A661758\Desktop\logback.xml

Not to mention that setting such parameters is through spark.driver.extraJavaOptions or spark.executor.extraJavaOptions as described in Runtime Environment:

spark.driver.extraJavaOptions A string of extra JVM options to pass to the driver. For instance, GC settings or other logging.

spark.executor.extraJavaOptions A string of extra JVM options to pass to executors. For instance, GC settings or other logging.

When you read the documentation, you should find the following (highlighting mine):

Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file.

When you execute spark-submit --help you should see the following:

--driver-java-options Extra Java options to pass to the driver.

Use --driver-java-options as follows:

./bin/spark-shell --driver-java-options -Daaa=bbb

and check out web UI's Environment tab if the option is set.

How to define log path configuration to spark-submit?

2 Answers2