Spark Executor Custom Logs

Asked Dec 05 '17 at 23:10

Active Dec 06 '17 at 02:55

Viewed 313 times

I've been supplying custom log4j properties to spark-submit in below manner:

spark-submit --master yarn --queue qqqq \
  --driver-java-options "-Dlog4j.configuration=file:/absolute path/to properties file/driver-log4j.properties" \
  --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/absolute path/to properties file/executor-log4j.properties" \
  --class com.abc.spark.def.MyClass myApp.jar arg1 arg2

Spark logs show that these driver and executor log4j properties are being read correctly. I am able to see the application logs written to driver logs but nothing in executor logs.

I've read this, this,this and this. Has anyone run into this issue earlier? If it helps, this code runs on MapR platform.

edited Dec 06 '17 at 02:55

Community

asked Dec 05 '17 at 23:10

user123

May be you need to add `--files` option to put log4j file to executor – Thang Nguyen Dec 06 '17 at 03:24
@cue - thank you for your input. I tried it, didn't work for me. – user123 Dec 06 '17 at 20:07
Were you able to fix this? I'm facing similar issue – Krishna Modi Apr 27 '21 at 09:11

Spark Executor Custom Logs

0 Answers0