1

We have a CDH5.4.2 yarn cluster and submitting a very simple example spark job gets the following exception:

15/07/02 17:21:06 ERROR scheduler.DAGSchedulerEventProcessLoop: DAGSchedulerEventProcessLoop failed; shutting down SparkContext
java.lang.NumberFormatException: For input string: "spark.locality.wait"
    at     java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Long.parseLong(Long.java:441)
    at java.lang.Long.parseLong(Long.java:483)
    at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:230)
    at scala.collection.immutable.StringOps.toLong(StringOps.scala:31)
    at org.apache.spark.scheduler.TaskSetManager.org$apache$spark$scheduler$TaskSetManager$$getLocalityWait(TaskSetManager.scala:853)
    at org.apache.spark.scheduler.TaskSetManager.computeValidLocalityLevels(TaskSetManager.scala:872)

Looking at the log, I can see the value being set in the properties 

spark.localExecution.enabled=false
spark.locality.wait=3000
spark.locality.wait.node=spark.locality.wait
spark.locality.wait.process=spark.locality.wait
spark.locality.wait.rack=spark.locality.wait

So, the value for locality looks like a valid long, and this doesn't appear to be happening in my code, so can anybody point me in the right direction?

This leaves the spark executor in a RUNNING state and it never comes out. It also kills the http application-monitor.

Thank you in advance!

  • 1
    possible duplicate of [Duplicate property value](http://stackoverflow.com/questions/7349415/duplicate-property-value) – Makoto Jul 02 '15 at 21:48

1 Answers1

0

use

spark.localExecution.enabled=false
spark.locality.wait=3000
spark.locality.wait.node=3000
spark.locality.wait.process=3000
spark.locality.wait.rack=3000
banjara
  • 3,800
  • 3
  • 38
  • 61