spark scala throws java.util.NoSuchElementException: key not found: 0 exception

Question

I have a piece of spark scala code like this:

val conf = new SparkConf().setAppName("MatrixInversion").setMaster("local")

val sc = new SparkContext(conf)

  def main (args: Array[String]) {

    var array:Array[Array[Double]]=new Array(2)
    for(i<- 0 until 2)
      array(i)=new Array(2)

    array(0)(0)=1
    array(0)(1)=2
    array(1)(0)=3
    array(1)(1)=4

    sc.makeRDD(array).cache()

    //val matrixA3=sc.textFile("A3")
    testCache()

    sc.stop()

  }

  def testCache():Unit={

    val rdds=sc.getPersistentRDDs
    val cacheArray=rdds(0).asInstanceOf[RDD[Array[Double]]]
    println("print cachaArray")
    cacheArray.collect()
    val cacheLength=cacheArray.collect().length
    println("length"+cacheLength)
  }

and now it will be ok. But when I uncomment this line: val matrixA3=sc.textFile("A3")

It has something wrong like this:

Exception in thread "main" java.util.NoSuchElementException: key not found: 0
    at scala.collection.MapLike$class.default(MapLike.scala:228)
    at scala.collection.AbstractMap.default(Map.scala:58)
    at scala.collection.MapLike$class.apply(MapLike.scala:141)
    at scala.collection.AbstractMap.apply(Map.scala:58)
    at com.scala.inversion.test$.testCache(test.scala:117)

Why?

chenzhongpu · Answer 1 · 2015-03-26T09:29:29.283

3

Your code is wrong actually.

See the line

val cacheArray=rdds(0).asInstanceOf[RDD[Array[Double]]]

rdds here, its type is scala.collection.Map[Int,org.apache.spark.rdd.RDD[_]].

However, the key (Int) is not always a constant one. (you may cache other thing ahead). For your code, key == 0 is not found in the map.

For example, I have tested your code in my computer, the rdds is

Map(9 -> ParallelCollectionRDD[9]

edited Mar 26 '15 at 09:29

answered Mar 26 '15 at 08:49

chenzhongpu

6,193
8
41
79

But when I use rdds.value.head I will get the same question.If I comment this line:val matrixA3=sc.textFile("A3") ,I get none error.But when I uncomment this line it has errors like that.Idon't know whether the sc.textFile("something") would initialization the sc or create a new sc(SparkContext) – 赵祥宇 Mar 27 '15 at 01:18
Just now I tried it on another computer and it is all right...I don't know why.... – 赵祥宇 Mar 27 '15 at 03:05
That solved my issue. I expected that to be deep Spark problem, but indeed my code was off, thanks to your answer I considered it a possibility! – Akavall Feb 02 '17 at 01:57

spark scala throws java.util.NoSuchElementException: key not found: 0 exception

1 Answers1

Linked