How to rank rows in DataFrame?

Question

I have this DataFrame df:

+-----------+--------------+----------+------------------+----+
|country_pk |        cat_pk|   item_pk|          valrank |rank|
+-----------+--------------+----------+------------------+----+
|     325100|      13531631|  69707980|  7.52451608671448|   1|
|     325100|      13531631|  69708020| 7.429137657465015|   2|
|     325100|      13531631|  69712040| 7.228689734451295|   3|
|     325100|      13531631|  69711180| 6.832206807237015|   4|
|     325100|      13531631|  69710300| 6.525382182769996|   5|
|     325100|      13531631|  69707960| 6.243003852010675|   6|
|     325100|      13531631|  69711150|6.0890434468906385|   7|
|     325100|      13531631|  69711060|3.5615308250377256|   8|
|     325100|      13531631|  69711130| 3.390146091450524|   9|
|     325100|      13531631|  69711330|3.1940345789723783|  10|
|     325100|      13531631|  69711190|2.4280125684249114|  11|
|     325100|      13531631|  69707920| 1.529893080297972|  12|
|     325100|      13531631|  69711370|               0.0|  13|
|     325100|      13531631|  69711740|               0.0|  13|
|     325100|      13531631|  69711140|               0.0|  13|
|     325100|      13531631|  69711170|               0.0|  13|
|     325100|      13531631|  69708010|               0.0|  13|
|     325100|      13531631|  69710260|               0.0|  13|
|     325100|      13531631|  69707970|               0.0|  13|
|     325100|      13531631|  69707930|               0.0|  13|
|     325100|      13531631|  69711160|               0.0|  13|
|     325100|      13531631|  69711110|               0.0|  13|
|     325100|      13531631|  69708000|               0.0|  13|
|     325100|      13531631|  69711420|               0.0|  13|
+-----------+--------------+----------+------------------+----+

I create the column rank as follows

val df = spark.sqlContext
              .sql("SELECT country_pk, cat_pk, item_pk, valrank, RANK() OVER (partition by cat_pk ORDER BY valrank desc) AS rank FROM df")

However, in this case, the value 13 gets duplicated. How can I get 13,14-..., even if valrank is equal?

row_number is a function that you can use with withColumn. :) I had used it with your other question in using window function https://stackoverflow.com/questions/48112589/how-to-drop-duplicates-using-conditions — Ramesh Maharjan, Jan 05 '18 at 15:44
@RameshMaharjan: Yes, I already obtained the correct result. Thanks. — Markus, Jan 05 '18 at 15:45
you can upvote the other answer if that helped you :) and you can delete this one :) — Ramesh Maharjan, Jan 05 '18 at 15:47
oh yes :) you've already upvoted :) its somebody stupid who downvoted. thanks @Markus. I guess you can delete this question — Ramesh Maharjan, Jan 05 '18 at 15:50
@RameshMaharjan: Well, it was not me, sorry. Some of my questions are also downvoted without any visible sense. In some cases, I can understand the downvoting, while in others it just does not make any sense. — Markus, Jan 05 '18 at 15:52

How to rank rows in DataFrame?

0 Answers0