1

I did follow this existing stackoverflow link, though that the accepted answer works if Spark has been installed from the source.

However, I installed pyspark via pip as a result I am unable to locate the conf/log4j.properties.template in order to mute the logging info.

I am using the following alternate solution from the same link:

def quiet_logs( sc ):
  logger = sc._jvm.org.apache.log4j
  logger.LogManager.getLogger("org"). setLevel( logger.Level.ERROR )
  logger.LogManager.getLogger("akka").setLevel( logger.Level.ERROR )

The above solution still allows the logs from the start-up and closing down Spark info on the terminal, which does not happen if I am able to update conf/log4j.properties.template.

Any suggestions will be appreciated.

himanshuIIITian
  • 5,985
  • 6
  • 50
  • 70
letsBeePolite
  • 2,183
  • 1
  • 22
  • 37

1 Answers1

0

You can override default log4j configuration for pyspark when invoking the command, you don't need to modify global files. Create a file with name log4j-quiet.properties and content:

log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

Then start pyspark with:

$ pyspark --driver-java-options '-Dlog4j.configuration=file:log4j-quiet.properties'

And logging messages from startup and closing down will gone.

Mariusz
  • 13,481
  • 3
  • 60
  • 64