IntellijIdea - Disable Info Message when running Spark Application

Question

I'm getting so many message when running application that using Apache Spark and Hbase/Hadoop Library. For Example :

0 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory  - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])

How to disable it, so i just get straight to the point Log like println(varABC) only ?

That's Spark's choice to use log4j logging. Try turning down the level from DEBUG to INFO. — duffymo, Aug 01 '16 at 09:52
Duplicate: http://stackoverflow.com/questions/38637662/how-to-drop-messages-in-console-when-using-spark-submit/38637780?noredirect=1#comment64683380_38637780 — sebszyller, Aug 01 '16 at 11:00

score 0 · Answer 1 · answered Aug 01 '16 at 10:19

What you are seeing is logs produced by Spark through log4j, as by default it enables quite a log of printouts printed to stderr. You can configure it as you are usually configuring log4j behavior, e.g. through a log4j.properties configuration file. Refer to http://spark.apache.org/docs/latest/configuration.html#configuring-logging

score 0 · Answer 2 · answered Aug 01 '16 at 11:55

In /spark-2.0.0-bin-hadoop2.6/conf folder you have a file log4j.properties.template

Rename from log4j.properties.template to log4j.properties

and make the following change in log4j.properties

from: log4j.rootCategory=INFO, console to: log4j.rootCategory=ERROR, console

Hope this Help!!!...

score -1 · Answer 3 · answered Aug 01 '16 at 11:53

Under $SPARK_HOME/conf dir modify the log4j.properties file - change values INFO to ERROR as below:

log4j.rootLogger=${root.logger}
root.logger=ERROR,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
log4j.logger.org.apache.spark.repl.Main=WARN
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=ERROR
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=ERROR
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

this will disable all the INFO log messages and only will print ERROR or FATAL log messages. you can change these values according to your requirement(s).

IntellijIdea - Disable Info Message when running Spark Application

3 Answers3