In "DataFrame API" , how show all rows?

Question

my code (only example)

bikedf.groupBy("Bike #").agg(
    count("Trip ID").alias("number")).\
sort(desc("number")).show()

I use jupiter notebook.

In situation, result only showing top 20 rows.

How show Not 20 rows, but all rows?

I am tempted to close this as duplicate of [Is there better way to display entire Spark SQL DataFrame?](https://stackoverflow.com/questions/30264373/is-there-better-way-to-display-entire-spark-sql-dataframe/30266712) because if you can show all the rows, then you probably shouldn't be using spark to begin with. — pault, Nov 29 '20 at 01:50

score 0 · Answer 1 · answered Nov 28 '20 at 16:58

Use collect instead of show:

print(bikedf.groupBy("Bike #")
            .agg(count("Trip ID").alias("number")
            .sort(desc("number"))
            .collect())

If you want to pretty print, you can do

output = bikedf.groupBy("Bike #") \
               .agg(count("Trip ID").alias("number") \
               .sort(desc("number"))
output.show(output.count())

score 0 · Answer 2 · answered Nov 28 '20 at 22:24

You cans et the number of rows you want to show. Also you can set to not truncate the output setting False in show function. To show 200 columns:

bikedf.groupBy("Bike #").agg(
    count("Trip ID").alias("number")).\
sort(desc("number")).show(200, False)

In "DataFrame API" , how show all rows?

2 Answers2