0

I have been researching it for 1 month and couldn't find a good solution. Default spark logs doesn't contain the application ID.

Default logs contains - "time":,"date":,"level":,"thread":,"message"

I tried to customize the log4j properties but I could't find a way. I am a newbie to the the big data area.

My default log4j.properties file is

log4j.rootLogger=${hadoop.root.logger}
hadoop.root.logger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

Does anyone know a solution ? Appreciate your help even it is a small help.

Rashmika Gamage
  • 13
  • 6
  • You can capture the applicationId using spark.sparkContext.applicationId and log it so that it will appear in the message. – linusRian Jun 07 '21 at 06:21
  • Thank you for your comment. How can I add this to log4j properties? – Rashmika Gamage Jun 07 '21 at 08:24
  • In the application code you can capture the applicationId and log it the same way you log other messages. log.info(spark.sparkContext.applicationId). In the log4j properties file you can only specify the format of logs and certain additional information such as class name and timestamp . please check this out. https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html – linusRian Jun 07 '21 at 08:33
  • I have never heard of this in properties. But you can do it programmatically. Look at this answer. https://stackoverflow.com/questions/8965946/configuring-log4j-loggers-programmatically – Saša Zejnilović Jun 07 '21 at 08:37

0 Answers0