1

I'm trying to come up with a way in Java to dynamically/programatically (is that word?) build a WindowSpec so that I can pass in parameters for the partitioning and order by clauses. I've got it partially working. I can pass in a simple comma separated string for each clause and build the window spec. The only thing I can't figure out is how to specifiy the order by direction, if I don't want ascending.

For ex, let's assume I've got a simple string for sortKeys like table1.a, table1.b. I can build my expression like so:

List<String> sorts = Arrays.asList(sortKeys.split(","));
Column[] sortCols = new Column[sorts.size()];

for (int i = 0; i < sorts.size(); i++) {
    sortCols[i] = new Column(sorts.get(i).trim() );
}

Then I can pass that into my WindowSpec:

WindowSpec w = org.apache.spark.sql.expressions.Window.partitionBy(partCols).orderBy(sortCols);

This all works fine. But how do I specify the direction of the order by for each column?

Andrew
  • 8,445
  • 3
  • 28
  • 46
  • 1
    the columns you want to sort by descending, use either `org.apache.spark.sql.functions.desc` function or `desc` method from `org.apache.spark.sql.Column` class. See this post https://stackoverflow.com/questions/30332619/how-to-sort-by-column-in-descending-order-in-spark-sql – C.S.Reddy Gadipally May 07 '19 at 03:43
  • Wow, I was making this waaaaay too hard, thanks! – Andrew May 07 '19 at 15:44
  • Also, if you are using multiple columns in `orderBy`, use `sort`. see https://stackoverflow.com/questions/55133532/get-the-highest-price-with-smaller-id-when-two-id-have-the-same-highest-price-in/55133699#55133699 – C.S.Reddy Gadipally May 07 '19 at 15:49
  • @Andrew How did you make it work to pass the columns dynamically in both partitionBy and OrderBy to make your solution work – Nikunj Kakadiya Aug 03 '22 at 04:47

0 Answers0