-1

I have a code where I wanted to update an RDD as below:

val xRDD = xRDD.zip(tempRDD)

This gave me the error : recursive value x needs type

I want to maintain the xRDD over iterations and modifying it with tempRDD in each iteration. How can I achieve it?

Thanks in advance.

Sk Kamaruddin
  • 37
  • 6

2 Answers2

3

The compiler is telling you that you're attempting to define a variable with itself and use it in it's own definition within an action. To say this another way, you're attempting to use something that doesn't exist in an action to define it.

Edit:

If you have a list of actions that produce new RDD that you'd like to zip together, perhaps you should look at a Fold:

listMyActions.foldLeft(origRDD){ (rdd, f) =>
  val tempRDD = f(rdd)
  rdd.zip(tempRDD)
}
wheaties
  • 35,646
  • 15
  • 94
  • 131
2

Don't forget that vals are immutable, this means that you can't reassign something to a previously defined variable. However if you want to do this, you can replace it for a var, which is not recommended, this question is more related to Scala's feature than to Apache-Spark's one. Besides, if you want more information you can consult this post Use of def val and vars in scala.

Community
  • 1
  • 1
Alberto Bonsanto
  • 17,556
  • 10
  • 64
  • 93