How to iterate on spark RDD and insert into HBASE

Question

I have generated an RDD in Google Cloud Spark Shell.

Now i have to insert the same into HBASE table. The format of my RDD is -

RDD[(String, Map[String, String])]

The First String is the row key and the Map[String, String] is the combination of column and it's corresponding value..

I have to use the below command to insert data using Hbase.Put Command -

val put = new Put(Bytes.toBytes("Value 1"));
put.addColumn(Bytes.toBytes("cf1"), Bytes.toBytes("greeting"), Bytes.toBytes("Greeting Heeloo World"));
table.put(put);

Problem i am facing is, i am not sure how to iterate inside the RDD.

My code is written in scala and i am running it on Google Cloud Spark Cluster.

Any help or pointers would be appreciated.

Possible duplicate of [How do I iterate RDD's in apache spark (scala)](https://stackoverflow.com/questions/25914789/how-do-i-iterate-rdds-in-apache-spark-scala) — OneCricketeer, Jul 15 '17 at 14:48
@cricket_007 That does not solve my scenario of having MAP inside an RDP. — Tushar Kesarwani, Jul 15 '17 at 15:07
Why not? For example, a foreach loop over the partitions with a case matching on (String, Map[String, String]) — OneCricketeer, Jul 15 '17 at 15:12
Alternatively, there's an hbase spark connector library that exists. Any reason not to use it? — OneCricketeer, Jul 15 '17 at 15:13
@cricket_007 Google cloud does not provide any documentation do use that.. So using this method. Can you help with the foreach code ? — Tushar Kesarwani, Jul 15 '17 at 15:17
GCP should allow custom Spark Packages, but try to use `rdd.foreachPartition` — OneCricketeer, Jul 15 '17 at 15:22
@cricket_007 I am very new to spark. If not an issue, can you give the example to write foreachPartition for [(String, Map[String, String])]. Thanks a lot for your help. — Tushar Kesarwani, Jul 15 '17 at 15:40

How to iterate on spark RDD and insert into HBASE

0 Answers0