how to add a Incremental column ID for a table in spark SQL

Question

I'm working on a spark mllib algorithm. The dataset I have is in this form

Company":"XXXX","CurrentTitle":"XYZ","Edu_Title":"ABC","Exp_mnth":.(there are more values similar to these)

Im trying to raw code String values to Numeric values. So, I tried using zipwithuniqueID for unique value for each of the string values.For some reason I'm not able to save the modified dataset to the disk. Can I do this in any way using spark SQL? or what would be the better approach for this?

Sorry..I figured out with this thread http://stackoverflow.com/questions/33102727/primary-keys-with-apache-spark — KM-Yash, Jul 14 '16 at 19:11
Can you please delete your question (since it's a duplicate)? Thanks. — Jacek Laskowski, Jan 29 '17 at 23:23
Possible duplicate of [Primary keys with Apache Spark](http://stackoverflow.com/questions/33102727/primary-keys-with-apache-spark) — Lykathia, Feb 27 '17 at 18:35

score 20 · Accepted Answer · edited Sep 16 '22 at 23:02

20

Scala

import org.apache.spark.sql.functions.monotonically_increasing_id
val dataFrame1 = dataFrame0.withColumn("index",monotonically_increasing_id())

Java

 Import org.apache.spark.sql.functions;
Dataset<Row> dataFrame1 = dataFrame0.withColumn("index",functions.monotonically_increasing_id());

edited Sep 16 '22 at 23:02

Community

1
1

answered Aug 08 '17 at 12:59

Yugerten

878
1
11
30

how to add a Incremental column ID for a table in spark SQL

1 Answers1

Linked