Globally sequential access of Apache Spark RDD

Question

I want to implement some sequential algorithm on RDD.

For example:

val conf = new SparkConf()
conf.setMaster("local[2]").
  setAppName("SequentialSuite")
val sc = new SparkContext(conf)
val rdd = sc.
  parallelize(Array(1, 3, 2, 7, 1, 4, 2, 5, 1, 8, 9), 2).
  sortBy(x => x, true)
rdd.foreach(println)

I want to see the ordered number on my screen, but it shows unordered integers. The two partitions execute the println simultaneously.

How do I make the RDD execute a function globally sequential?

Partitions are processed in parallel, so why do you explictly set 2 partitions? — G Quintana, Feb 27 '15 at 10:07
Because I have a large amount of data to process and the code snippet is just a proof of concept. — wush978, Feb 27 '15 at 10:58

score 1 · Answer 1 · edited May 23 '17 at 10:24

1

I found the answer according to Spark: Best practice for retrieving big data from RDD to local machine:

val rdd : RDD[Int] = sc.parallelize(Array(1, 3, 2, 7, 1, 4, 2, 5, 1, 8, 9)).sortBy(x => x, true)
for(p <- rdd.partitions) {
    val partrdd = rdd.mapPartitionsWithIndex((i : Int, iter : Iterator[Int]) => if (i == p.index) iter else Iterator(), true)
    partrdd.foreach(println)
}

edited May 23 '17 at 10:24

Community

1
1

answered Feb 27 '15 at 10:59

wush978

3,114
2
19
23

Globally sequential access of Apache Spark RDD

1 Answers1