Spark sql row_number or sequence number?

Question

random number generator SparkSQL ?

For example:

Thanks.

http://stackoverflow.com/q/32760888/1560062 – zero323 Dec 07 '15 at 15:50 — zero323, Dec 07 '15 at 15:50
Sorry, but I really don't understand what he is asking for. – Alberto Bonsanto Dec 07 '15 at 15:52 — Alberto Bonsanto, Dec 07 '15 at 15:52

score 2 · Answer 1 · answered Dec 08 '15 at 12:09

2

Sequence in spark sql is in spark 1.6 its select monotonically_increasing_id() from table , spark 1.6 is due to get released

answered Dec 08 '15 at 12:09

here an example which works for Spark 2.2 sine_signale_no_ts.createOrReplaceTempView("signal") sine_signale_my_ts = sql("select (monotonically_increasing_id()+1483228800)*1000 as ts, sine from signal") – Romeo Kienzler Aug 25 '17 at 12:41

score 1 · Answer 2 · edited Jun 20 '20 at 09:12

1

Spark Sql already have random functions there is one blog.

Or for number of rows spark sql also have row_number() function.

edited Jun 20 '20 at 09:12

Community

answered Dec 07 '15 at 15:52

Kaushal

Hi Kaushal, I tried row_number function but spark sql gave an error – sri hari kali charan Tummala Dec 07 '15 at 16:41
sqlContext.sql("select * from (select column1,column2,column3,column4,column5 ,row_number() OVER (PARTITION BY column1 ORDER BY column1 DESC) rank as random from TestTable ) a") – sri hari kali charan Tummala Dec 07 '15 at 16:42
what exception you are getting – Kaushal Dec 08 '15 at 05:46
Hi Kaushal, I got what I am looking for in spark 1.6 its select monotonically_increasing_id() from table , spark 1.6 is due to get released – sri hari kali charan Tummala Dec 08 '15 at 12:08

2 Answers2