0

Recently I updated my spark version to 3.3. Before update, my log4j.properties works fine, and the log show me only errors. But after update, my log4j.properties don't work anymore.

This is my log4j.properties:

# Set root logger level to desired log level (DEBUG, INFO, WARN, ERROR, or FATAL)
log4j.rootLogger=ERROR, console

# Define console appender
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set log level for specific packages or classes
# For example, to set the log level to WARN for the org.apache.spark package:
log4j.logger.org.apache.spark=WARN
Dmytro Mitin
  • 48,194
  • 3
  • 28
  • 66
Danilo Rodrigues
  • 373
  • 3
  • 17
  • https://stackoverflow.com/questions/27781187/how-to-stop-info-messages-displaying-on-spark-console https://stackoverflow.com/questions/25193488/how-to-turn-off-info-logging-in-spark https://stackoverflow.com/questions/65467400/how-to-disable-all-logging-info-to-the-spark-console-from-net-application https://stackoverflow.com/questions/48607642/disable-info-messages-in-spark-for-an-specific-application https://stackoverflow.com/questions/46941264/spark-turn-off-logging-when-using-spark-submit – Dmytro Mitin Apr 24 '23 at 21:57
  • Where do you put your log4j.properties file and deploy it? Are you running as a batch job or notebook? – Aravind Yarram Apr 25 '23 at 02:06
  • @DmytroMitin I tested all these options and none worked on spar 3.3. – Danilo Rodrigues Apr 25 '23 at 13:31
  • @AravindYarram I running as a batch job. – Danilo Rodrigues Apr 25 '23 at 13:31
  • @DaniloRodrigues So are you bundling the log4j props as part of jar file of the job? – Aravind Yarram Apr 25 '23 at 14:10
  • @AravindYarram Yes. Aparently this version (3.3) log4j.properties don't works. I change log4j.properties to log4j2.properties and now works fine. I don't know why, but works. – Danilo Rodrigues Apr 25 '23 at 15:48

2 Answers2

2

Aparently log4j.properties don't work with spark 3.3. After performing several tests I realized that spark was setting as default a file called log4j2.default. So to test, I updated my log4j.properties to log4j2.properties and it worked.

Danilo Rodrigues
  • 373
  • 3
  • 17
0

You can do the same thing programmatically by setting log level in spark job

import org.apache.logging.log4j.core.config.Configurator
Configurator.setLevel("org", Level.ERROR.toString);
Configurator.setLevel("akka", Level.ERROR.toString);

by this way only error log will be displayed from org and akka level and for any other log that you put in your own spark job as info will be displayed in log as well

Shalaj
  • 579
  • 8
  • 19