1

I want to collect the log messages created by spark application into a file on the local file system. I was able to achieve this when I ran the application in client mode, however I found difficulties once running in cluster mode.

My configuration for log4j:

# Root logger option
log4j.rootLogger=ERROR, stdout
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p: %c %m%n
#Direct log messages to a file
log4j.logger.com.bosch.bidos=INFO, file
log4j.additivity.com.bosch.bidos=false

log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.File=${user.home}/test.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %-5p: %c: %m%n

And here is my spark submit:

spark2-submit \
        --name test \
        --master yarn \
        --deploy-mode cluster \
        --class Main \
        --files path/to/test_log4j.properties \
        --conf spark.driver.extraJavaOptions="-Dlog4j.configuration=test_log4j.properties -Dlog4j.debug" \
        --conf spark.executor.extraJavaOptions="-Dlog4j.configuration=test_log4j.properties -Dlog4j.debug" \
        ${jar_path}

When I debug the application I can see, that the given test_log4j.properties file is read.

Am I missing something?

sanyi14ka
  • 809
  • 9
  • 14
  • Possible duplicate of [How to stop INFO messages displaying on spark console?](https://stackoverflow.com/questions/27781187/how-to-stop-info-messages-displaying-on-spark-console) – Rahul Sharma Oct 18 '18 at 15:33
  • 1
    In cluster mode user consoleAppender and use- yarn logs -applicationId to get the logs. – Rahul Sharma Oct 18 '18 at 15:35
  • Does that mean that the only way to retrieve the logs into a file, is if I pipe out the results of 'yarn logs -applicationId ' into a file? – sanyi14ka Oct 19 '18 at 05:56

0 Answers0