1

I wrote a spark job. The job ran fine without any problems. However when I look at my error log file, I see so many messages of type

[error] 18/11/25 17:28:14 INFO CodeGenerator: Code 
generated in 16.947005 ms

and 

[error] 18/11/25 17:28:15 INFO ContextCleaner: Cleaned
accumulator 239819

and 

[error] 18/11/25 17:28:06 INFO BlockManagerInfo: Removed 
broadcast_13354_piece0 on 192.168.2.101:43753 in memory 
(size: 20.5 KB, free: 6.2 GB)

Is there any way to suppress these messages. they are just bloating up my log file.

Not sure why spark is reporting these as error. when they look like some kind of debug messages.

Knows Not Much
  • 30,395
  • 60
  • 197
  • 373

1 Answers1

6

when you are creating the object of SparkContext, use the following code with it to set the log level according to the requirement:

sparkContext.setLogLevel("WARN")

The above line will set the log level for Spark to WARN and you will not get any INFO or DEBUG level logs.

anuj saxena
  • 279
  • 1
  • 7
  • I found my `sparkContext` was in the `SparkSession`. From https://stackoverflow.com/a/40504350/2523501 - In the pyspark console, a default spark session will already be available `spark.sparkContext.setLogLevel('WARN')` – yeliabsalohcin Jan 31 '20 at 12:13