Using apache spark to iterate over string

Question

For example we have the string "abcdabcd"

And we want to count all the pairs (e.g: "ab" or "da") that are available in the string.

So how do we do that in apache spark?

I asked this cause it looks like that the RDD does not support sliding function:

rdd.sliding(2).toList
//Count number of pairs in list
//Returns syntax error on first line (sliding)

score 5 · Accepted Answer · edited May 23 '17 at 11:44

5

Apparently it supports sliding via mllib as shown by zero323 here

import org.apache.spark.mllib.rdd.RDDFunctions._

val str = "abcdabcd"

val rdd = sc.parallelize(str)

rdd.sliding(2).map(_.mkString).toLocalIterator.forEach(println)

will show

ab
bc
cd
da
ab
bc
cd

edited May 23 '17 at 11:44

Community

1
1

answered Dec 11 '15 at 17:07

Odomontois

15,918
2
36
71

and how do we count these pairs? Well btw you look like someone who is hunting my scala questions here :) – lkn2993 Dec 11 '15 at 17:12
@lkn2993 using the classic [word count](https://spark.apache.org/examples.html) approach in `Apache Spark` – Alberto Bonsanto Dec 11 '15 at 17:28

Using apache spark to iterate over string

1 Answers1