How can I group pairRDD by keys and turn the values into RDD

Asked Jan 23 '18 at 13:05

Active Jan 23 '18 at 13:36

Viewed 187 times

So what I have is RDD[(String, Int)] and I need to convert it into Map[String, RDD[Int]]

Ex.

My input looks like this:

RDD[("a", 1), ("a", 2), ("b", 1), ("c", 3)]

And the output I'm trying to get is:

Map["a" -> RDD[1, 2], "b" -> RDD[1], "c" -> RDD[3]]

Thanks in advance!

edited Jan 23 '18 at 13:19

asked Jan 23 '18 at 13:05

OguzGelal

1

what's the use case for this exactly? – mtoto Jan 23 '18 at 13:21
@mtoto Its for analyzing a bunch of hashtags and their popularity (in terms of the like/fav count). Pairs are in the format (Tag, Like). The output of this is going to be clustered with the k-means algorithm. – OguzGelal Jan 23 '18 at 13:25
don't think you are going to be able cluster anything in map object that contains distributed data as values – mtoto Jan 23 '18 at 13:35

0 Answers0