6

I am running spark on yarn cluster. I tried to start the history server

./start-history-server.sh

but got the following errors.

starting org.apache.spark.deploy.history.HistoryServer, logging to /home/abc/spark/spark-1.5.1-bin-hadoop2.6/sbin/../logs/spark-abc-org.apache.spark.deploy.history.HistoryServer-1-abc-Efg.out
failed to launch org.apache.spark.deploy.history.HistoryServer:
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:47)
... 6 more
full log in /home/abc/spark/spark-1.5.1-bin-hadoop2.6/sbin/../logs/spark-abc-org.apache.spark.deploy.history.HistoryServer-1-abc-Efg.out

I have set spark.eventLog.enabled = true and spark.history.fs.logDirectory and spark.eventLog.dir to the hdfs logging directory.

Why can't I get the history server to start?

Update 1:

Thank you stf for telling me to look at the log file; I didn't know it exists!

I realise my problem is in my setting in spark-env.sh

 export SPARK_HISTORY_OPTS="-Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs:///localhost/eventLogging spark.history.fs.logDirectory=hdfs:///localhost/eventLogging"

The forward slash becomes dot

 Error: Could not find or load main class spark.history.fs.logDirectory=hdfs:...localhost.eventLogging

Any idea how to prevent this from happening?

Update 2: Solved this problem thanks to stf's help. Correct setting in spark-env.sh is 

 SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs://localhost/eventLogging -Dspark.history.fs.logDirectory=hdfs://localhost/eventLogging"
Michael
  • 1,398
  • 5
  • 24
  • 40
  • What is in `spark-abc-org.apache.spark.deploy.history.HistoryServer-1-abc-Efg.out`. Please put that log here, so someone can get a better view – ѕтƒ Dec 15 '15 at 09:10
  • the error message is shown as it is. How do I increase the verbosity? – Michael Dec 15 '15 at 09:13
  • That error message says "full log in /home/abc/spark/spark-1.5.1-bin-hadoop2.6/sbin/../logs/spark-abc-org.apache.spark.deploy.history.HistoryServer-1-abc-Efg.out" - Have you looked on to that? – ѕтƒ Dec 15 '15 at 09:14
  • @ѕтƒ Thanks, I found the log and apparently the problem is spark convert forward slash (/) into dot (.) in spark-env.sh. Do you know how to prevent this conversion? – Michael Dec 15 '15 at 09:58
  • 1
    I doubt is it because of missing `-D` in front of `spark.history.fs.logDirectory`? – ѕтƒ Dec 15 '15 at 10:03
  • @ѕтƒThank you! I want to 'tick' you! It is really the missing -D What does this -D represent? – Michael Dec 15 '15 at 10:12

1 Answers1

4

For those still getting this error and not helped by the comment discussion. The following resolved this issue for me. Make sure that you have the following in in spark/conf/spark-defaults.conf

spark.eventLog.enabled          true
spark.eventLog.dir              /path/to/spark/logs
spark.history.fs.logDirectory   /path/to/spark/logs

Then run spark/sbin/start-history-server.sh /path/to/spark/logs

Jon Deaton
  • 3,943
  • 6
  • 28
  • 41