How does spark handle aggregate max for non numeric values?

Question

I have a dataframe which has the following data

DF1

|value|condition|
+-----+---------+
| 1   |   Y     |
| 2   |   Y     |
| 3   |   Y     |
| 3   |   N     |
| 3   |   N     |
+---------------+

I want to understand what will the result of the dataframe if i have max on an aggregation

DF1.groupby(DF1).max(condition) does it give the max count of the strings which is Y, if so how do i get the max values according to the alphabetical order ?

Edit--

This is not for date or any other datatype i want it exclusively for string

i want it exclusively for string where the link provided is for date @user10465355 — Sundeep Pidugu, Nov 15 '18 at 13:53

score 2 · Accepted Answer · edited Nov 15 '18 at 15:13

2

Try this,

scala> val df1 = Seq((1,"Y"),(2,"Y"),(3,"N"),(3,"Z")).toDF("value","condition")
df1: org.apache.spark.sql.DataFrame = [value: int, condition: string]

scala> df1.show
+-----+---------+
|value|condition|
+-----+---------+
|    1|        Y|
|    2|        Y|
|    3|        N|
|    3|        Z|
+-----+---------+


scala> df1.agg(max("condition")).show
+--------------+
|max(condition)|
+--------------+
|             Z|
+--------------+

edited Nov 15 '18 at 15:13

Sundeep Pidugu

2,377
2
21
43

answered Nov 15 '18 at 13:32

Sathiyan S

1,013
6
13

so when applying for max it will automatically give the highest order alphabet ? instead of occurrences of the alphabet ? – Sundeep Pidugu Nov 15 '18 at 13:55
yes! Is this not what you wanted? – Sathiyan S Nov 16 '18 at 06:18
Yeah ! what if want to get the number of occurrences of the alphabet to be calculated ? – Sundeep Pidugu Nov 16 '18 at 08:30
1

`df1.groupBy("condition").agg(count("condition")).show` – Sathiyan S Nov 16 '18 at 08:48

How does spark handle aggregate max for non numeric values?

1 Answers1