Disable INFO messages in Spark for an specific application

Question

I've tried to disable INFO messages in Spark in a particular application, but I couldn't, I was still seeing it.

The code I tried was from here: (How to stop messages displaying on spark console? )

import org.apache.log4j.Logger
import org.apache.log4j.Level

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)`

I've also edited the log4j file in Spark installation dir, putting this:

log4j.rootCategory=WARN, console

And it worked! But it is not the solution that I wanted... I would like to disable the INFO message from each application.

Any comments? Thank you!!

Yehor Krivokon · Accepted Answer · 2018-02-04T17:58:53.703

3

You must do this:

import org.apache.log4j.Logger
import org.apache.log4j.Level

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

before creating a SparkContext.
Do something like this:

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
val sc = new SparkContext(conf)

Also, you can use getRootLogger method:

Logger.getRootLogger().setLevel(Level.ERROR)

edited Feb 04 '18 at 17:58

answered Feb 04 '18 at 11:51

Yehor Krivokon

837
5
17

Thank you, you're right, both ways worked! Just, please edit the code after "Do something like this:" putting val sc... line at the end, as you said :) – mjbsgll Feb 04 '18 at 14:44
Yes, that's my bad) Thanks. – Yehor Krivokon Feb 04 '18 at 15:12
Is there a similar way to disable logging in pyspark before creating a SparkContext? – arch DJ Nov 24 '19 at 19:49

Disable INFO messages in Spark for an specific application

1 Answers1

Linked