Pyspark: Get maximum value from a column [performance improvement]

Question

I have a spark dataframe as follows:

+-----+------+
|A    | count|
+-----+------+
|dummy|23    |
|ABC  |157   |
|abc  |15    |
+-----+------+

I am trying to find the max value out of this column [157 in the example above] and this is what I have done:

max_value = df.agg({"count": "max"}).collect()[0][0]

I am new to spark programming. Although the solution above works, for large data [say a few million rows], I am unsure how efficient this solution is going to be [as it involves a reduction component]. Are there more efficient solutions available to get the max value out of a column?

PS: I have gone through many solutions on the internet [such as: https://stackoverflow.com/questions/33224740/best-way-to-get-the-max-value-in-a-spark-dataframe-column] and have not come across one dealing with the performance.

EDIT 1: The dataframe I am dealing with has multiple columns of large data.

EDIT 2: There are the transformations being performed on the data before the max value is to be fetched:

a) I get my input data from Google Cloud Platform (in Parquet).

b) This data is converted into a pyspark dataframe.

c) I then add a "count" column to this dataframe.

d) Then, from the "count" column, I would like to fetch the max value.

Answer 1

-- Not an answer --

A quick POC for @BennyElgazar.
Executed on Azure Databricks using a single VM cluster, with 4 cores.

import pyspark.sql.functions as F

df = spark.range(1000000000).orderBy(F.rand())

---

df.agg({"id":"max"}).first()[0]

Out[34]: 999999999 Command took 0.64 seconds

df.sort(F.col("id").desc()).first().asDict()['id']

Out[35]: 999999999 Command took 55.38 seconds

---

P.S.
There is no need for asDict()

Any of the following works just fine:

df.sort(F.col("id").desc()).first()[0]
df.sort(F.col("id").desc()).first().id
df.sort(F.col("id").desc()).first()['id']

Answer 2

As suggested by @DavidדודוMarkovitz above, the best possible solution to reduce execution time is to make use of spark caching or persistence mechanisms. This is elaborated below:

• Store the dataframe to cache and/or to disk.
• Run a simple dataframe action [such as count()] on this dataframe.
• Post this, run the actual [in this case, get the max value] transformation.

The intermediary step [count()] would take some time but post this [due to the lazy loading nature of spark], all the other transformations on the dataframe would take minimal time to complete.

Answer 3

What is the motivation for aggregation and not sorting ? If your interest is only in the max count number which you already have according to your example.

so why not just DF.sort(f.col("count").desc()).first().asDict()['count'] ?