How to query for the maximum / highest value in an field with PySpark

Question

The following dataframe will produce values 0 to 3.

df = DeltaTable.forPath(spark, '/mnt/lake/BASE/SQLClassification/cdcTest/dbo/cdcmergetest/1').history().select(col("version"))

Can someone show me how to modify the dataframe such that it only provides the maximum value i.e 3?

I have tried

df.select("*").max("version")

And

df.max("version")

But no luck

Any thoughts?

Does this answer your question? [Best way to get the max value in a Spark dataframe column](https://stackoverflow.com/questions/33224740/best-way-to-get-the-max-value-in-a-spark-dataframe-column) — Lamanus, Mar 19 '23 at 04:37

score 1 · Accepted Answer · answered Mar 19 '23 at 04:27

Use Max function, This should work:

df.select(F.max("version").alias("max_version")).show()

or

df.agg(F.max("version").alias("max_version")).show()

Input:

+-------+
|version|
+-------+
|      0|
|      1|
|      3|
|      2|
+-------+

Output:

+-----------+
|max_version|
+-----------+
|          3|
+-----------+

1 Answers1