2

I am new to Spark. I am not able to find out how to handle logs in Spark Cluster Mode. I have added the below properties in Spark script.

spark.conf.set("yarn.log-aggregation-enable","true")
spark.conf.set("yarn.nodemanager.log-dirs","HDFS_LOCATION")
spark.conf.set("yarn.nodemanager.remote-app-log-dir","HDFS_LOCATION")
spark.conf.set("spark.eventLog.enabled", "true")
spark.conf.set("spark.eventLog.dir", "HDFS_LOCATION")
spark.conf.set("spark.scheduler.mode", "FAIR")

And when runnig the spark-submit I am adding the below option:

--driver-java-options "-Dlog4j.debug=true -Dlog4j.configuration=$LOCATION/log4j.properties"

But I am getting the below exception :

Exception in thread "main" org.apache.spark.SparkException: Application

And I am unable to find any log in HDFS log location.

Please help as I am stuck with the code.

vvg
  • 6,325
  • 19
  • 36
akshay
  • 41
  • 1
  • 3

0 Answers0