How logs printed directly onto the console in yarn-cluster mode using spark

Question

I am new in spark and i want to print logs on console using apache spark in yarn cluster mode.

You can only print logs to the console in client mode, AFAIK — OneCricketeer, Mar 20 '17 at 13:20
see http://stackoverflow.com/questions/40806225/pyspark-logging-from-the-executor — Zhang Tong, Mar 21 '17 at 06:25

score 0 · Answer 1 · answered Mar 20 '17 at 15:41

You need to check the value in log4j.properties file. In my case i have this file in /etc/spark/conf.dist directory

log4j.rootCategory=INFO,console

INFO - prints the all the logs on the console. You can change the value to ERROR, WARN to limit the information you would like to see on the console as sparks logs can be overwhelming

How logs printed directly onto the console in yarn-cluster mode using spark

1 Answers1