How can I add values in key value pairs generated in scala

Question

How to add the keys and values separately from the keys and value pairs generated in spark scala?

Given the following input

(5,1),(6,1),(8,1)

I'd like to get to the following output

(19,3)

This is what I've tried so far:

val spark = SparkSession.builder.appName("myapp").getOrCreate()   
val data = spark.read.textFile(args(0)).rdd  
val result =
  data.map { line => {  
    val tokens = line.split("\t")  
    (Float.parseFloat(tokens(4)),1)  
  }}.
  reduceByKey( _+ _)

Row 1:10-10-2010 Chennai VMall Raju 10,000 Row 2:10-11-2010 Hyderabad CRMall Ram 5000 and so on.. — Chaitanya, Jul 12 '18 at 17:45

score 1 · Accepted Answer · answered Jul 12 '18 at 18:05

1

You can use reduce or fold to get the result, You also need to convert the token(4) value to Int or any other Numeric type as you need.

val result = data.map{line => {  
  val tokens = line.split("\t")  
  (tokens(4).toInt,1)  
}}

Using fold

result.fold((0,0)) { (acc, x) => (acc._1 + x._1, acc._2 + x._2)}

Using reduce

result.reduce((x,y) => (x._1 + y._1, x._2 + y._2))

Hope this helps!

answered Jul 12 '18 at 18:05

koiralo

22,594
6
51
72

if I use the above reduce method saveAsTextFile is not working throwing prompt like it is not a member of (Float,Int) – Chaitanya Jul 12 '18 at 18:24
saveAsTextFile is performed on rdd but you already performed an action with reduce or fold so which gives you (Int, Float) – koiralo Jul 12 '18 at 18:26
Can you explain me how I can save and view the result after fold or reduce? – Chaitanya Jul 12 '18 at 18:29
Assign it to a variable and print it – koiralo Jul 12 '18 at 18:30
but if we use executable jar like exporting our program into jar and then executing it,then print method will not work right? – Chaitanya Jul 12 '18 at 18:42

score 0 · Answer 2 · answered Jul 12 '18 at 17:51

0

reduceByKey won't serve your purpose here. Please use foldLeft.

Refer Scala: How to sum a list of tuples for solving your problem.

answered Jul 12 '18 at 17:51

Vinod Chandak

373
1
6
15

Travis Hegner · Answer 3 · 2018-07-12T18:08:36.187

0

val spark = SparkSession.builder.appName("myapp").getOrCreate()   
val data = spark.read.textFile(args(0)).rdd  
val result = data.map{line => {  
  val tokens = line.split("\t")  
  (tokens(4).toInt,1)  
}}  
.reduce((l, r) => (l._1+r._1, l._2+r._2))

It's possible that a foldLeft (as suggested by Vinod Chandak) is more appropriate, but I tend to use reduce as I have more experience with it.

edited Jul 12 '18 at 18:08

answered Jul 12 '18 at 17:54

Travis Hegner

2,465
1
12
11

How can I add values in key value pairs generated in scala

3 Answers3