How to convert arrays to map?

Question

I have a spark dataframe like this:

The schema is: I want to have the final result like this: Namely, to create a map between _1: integer and _2: map in site_group.

How to do this in scala spark?

Can you add text instead of images so that it would be easy for users to copy and test it. — koiralo, Jun 11 '19 at 06:03

abiratsis · Accepted Answer · 2019-06-11T13:27:41.413

Here is one solution. First lets create some sample data similar to yours, btw it would be much more helpful to post a question with reproducible input and output data as discussed here.

val df = Seq(
(999, "2019-05-23", Seq((0,Map(2154 -> 0.545)))),
(511, "2019-06-30", Seq((1,Map(564654 -> 0.255699)))),
(322, "2019-02-10", Seq((2,Map(122 -> 0.896)))))
.toDF("user_id","dt", "site_group_collect")

// +-------+----------+---------------------------+
// |user_id|dt        |site_group_collect         |
// +-------+----------+---------------------------+
// |999    |2019-05-23|[[0, [2154 -> 0.545]]]     |
// |511    |2019-06-30|[[1, [564654 -> 0.255699]]]|
// |322    |2019-02-10|[[2, [122 -> 0.896]]]      |
// +-------+----------+---------------------------+

Then we iterate through each item and transform the values of site_group_collect using the map function of the dataframe:

df.map{case Row(uid: Int, dt: String, group: Seq[Row]) => 
     val transformed = group.map{ r => Map(r.getInt(0) -> r.get(1).asInstanceOf[Map[Int, Double]]) }
     (uid, dt, transformed)
}
.toDF("user_id","dt", "site_group_collect")
.show(false)

// +-------+----------+-----------------------------+
// |user_id|dt        |site_group_collect           |
// +-------+----------+-----------------------------+
// |999    |2019-05-23|[[0 -> [2154 -> 0.545]]]     |
// |511    |2019-06-30|[[1 -> [564654 -> 0.255699]]]|
// |322    |2019-02-10|[[2 -> [122 -> 0.896]]]      |
// +-------+----------+-----------------------------+

The key point here is the representation of the array of tuples [[0, [2154 -> 0.545]]] as an array of Rows. Another approach would be to represent the tuple as a case class i.e:

case class Item(pk: Int, m: Map[Int, Double])

The line:

val transformed = group.map{ r => Map(r.getInt(0) -> r.get(1).asInstanceOf[Map[Int, Double]]) }

Will extract the key/value combination from the existing tuple and assign it to the new created Map.

Some related posts:

How to convert arrays to map?

1 Answers1