0

Given that this works:

(1 to 5).iterator.sliding(3).toList

Then why does this not work?

val rdd1 = sc.parallelize(List(1,2,3,4,5,6,7,8,9,10), 3)
val z = rdd1.iterator.sliding(3).toList

I get the following error and try to apply the fix but that does not work either!

notebook:3: error: missing argument list for method iterator in class RDD
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `iterator _` or 
`iterator(_,_)` instead of `iterator`.
val z = rdd1.iterator.sliding(3).toList
             ^

I am just trying examples and this I cannot really follow.

thebluephantom
  • 16,458
  • 8
  • 40
  • 83

1 Answers1

0

It doesn't work, because iterator RDD is not a Collection and it's iterator method has different signature:

final def iterator(split: Partition, context: TaskContext): Iterator[T]

Internal method to this RDD; will read from cache if applicable, or otherwise compute it. This should not be called by users directly, but is available for implementors of custom subclasses of RDD.

If you want to convert RDD to local Iterator use toLocalIterator:

def toLocalIterator: Iterator[T]

Return an iterator that contains all of the elements in this RDD.

rdd1.toLocalIterator

but what you probably want is RDDFunctions.sliding - Operate on neighbor elements in RDD in Spark.

Alper t. Turker
  • 34,230
  • 9
  • 83
  • 115