3

Can anyone help me with the spark configuration needed to set logging level to debug and capture more logs

user13485171
  • 43
  • 1
  • 4
  • Does this answer your question? [How can set the default spark logging level?](https://stackoverflow.com/questions/40608412/how-can-set-the-default-spark-logging-level) – William Prigol Lopes May 06 '20 at 18:51

1 Answers1

4

Set setLogLevel property to DEBUG in sparksession.

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('test').getOrCreate()
spark.sparkContext.setLogLevel("DEBUG")

#or

log4j = sc._jvm.org.apache.log4j
log4j.LogManager.getRootLogger().setLevel(log4j.Level.DEBUG)

#for custom log
log4j.LogManager.getRootLogger().info("custom log")
#2020-05-06 13:43:42 INFO  root:? - custom log
notNull
  • 30,258
  • 4
  • 35
  • 50
  • Thanks. But i think this line is creating end less console output in my case. The error was around "connection error" – user13485171 May 06 '20 at 18:32
  • @user13485171, Could you update the question with steps you are `performing` and `error`? – notNull May 06 '20 at 18:45
  • I would like to but i can't as that's little confidential My code looks like Setting environment variables Creating spark session similarly Then i tried to change log level So with the new code recreted the issue I think it's more because of my server settings/permission I'll take this up with my IT and update you why it happened – user13485171 May 06 '20 at 19:10