How to set up logging level for Spark application in IntelliJ IDEA?

Question

I'm working on a Scala project in IntelliJ that was created through SBT. The project has Spark as one of its dependencies. I'm still in the development phase so everything is running on my local machine.

How can I change Spark configurations like logging level, for example?

if not of these work for you, see this : https://stackoverflow.com/a/60621331/13039907 set log config via logback instead of log4j. — Alastair L Q Yong, Mar 10 '20 at 16:00

score 14 · Answer 1 · answered May 31 '17 at 11:52

14

Setting the log level on the SparkContext worked for me under Eclipse

spark.sparkContext.setLogLevel("WARN")

answered May 31 '17 at 11:52

rschmidt13

349
3
3

This worked for me. The accepted answer of setting the log level for all the loggers didn't work. Although, one problem with this is that, I'm not seeing any logs, not even the ones from my application. Don't know if it's because of this or something else. – Durga Swaroop Jun 25 '18 at 12:15

score 9 · Accepted Answer · answered Aug 12 '15 at 08:13

9

If you are working on the local development with IDE, you can change the log level at run-time by:

LogManager.getRootLogger.setLevel(Level.ALL)

Ps: Put that line after the SparkContext/ SQLContext was created in your code.

answered Aug 12 '15 at 08:13

ngtrkhoa

754
5
6

4

Its not working for me. I tried `Level.OFF`, `Level.ERROR` but I always get the `INFO` log messages. I am using spark 2.1.0 – Durga Swaroop Jan 20 '17 at 06:27
I am encountering the same problem apparently. – Jonathan Liono May 23 '17 at 11:35
me too , can you follow up this https://stackoverflow.com/questions/59848344/how-to-change-log-level-in-spark – supernatural Jan 21 '20 at 20:12

score 2 · Answer 3 · answered Mar 19 '17 at 20:29

2

Put your log4j.properties file under a directory marked as resources, spark will read this log4j configuration.

answered Mar 19 '17 at 20:29

tianhaopx

53
4

score 2 · Answer 4 · answered Jul 24 '19 at 16:53

I would love to figure out how to do this with a project local properties file (an example file would be nice), but I was able to get this done in Spark 2.2 with the following code:

import org.apache.log4j.{Level, Logger}

object MySparkApp {

    def main(args: Array[String]): Unit = {
      Logger.getLogger("org.apache.spark").setLevel(Level.WARN)

score 1 · Answer 5 · answered Sep 18 '18 at 19:33

1

For shutting down/setting log level programmatically in spark 2.0+

Logger.getLogger("org.apache.spark").setLevel(Level.OFF);

answered Sep 18 '18 at 19:33

Kobynet

983
11
23

score 0 · Answer 6 · answered Aug 11 '15 at 23:14

Spark by default logs almost everything you would like to see in the logs, however, if you need to change the logging behaviour, you can edit log4j.properties in the conf directory of your Apache Spark configuration. If you're using a prebuilt version, you can find it in /home/coco/Applications/spark-1.4.0-bin-hadoop2.6/conf directory. There is a template file "log4j.properties.template" that you have to copy to "log4j.properties" and edit it depending on your needs. I hope it helps.

How to set up logging level for Spark application in IntelliJ IDEA?

6 Answers6

Linked