How to select only a specific range of rows in Pyspark?

Question

When I try to create a new column in my pyspark dataframe with the number of each row. Now I need to select only a range of rows and it just keep bringing the same error

The code i'm using is:

from pyspark.sql.window import Window
w = Window().partitionBy(lit('a')).orderBy(lit('a'))
df1 = my_dataframe.withColumn("row_num", row_number().over(w))

display(df1.filter(col("row_num").between(1,40)))

And the error is:

TypeError: 'int' object is not callable

The snippet I'm using can be found here in this stackoverflow post.

Thanks!

Yes, the "cols" was just a typo. I was thinking about another thing when I wrote that, sorry — Marcos Dias, Jul 18 '22 at 14:42
if you're still getting the error, try debugging `my_dataframe`. does `my_dataframe.show()` execute succesfully? — samkart, Jul 18 '22 at 14:48

score 0 · Answer 1 · answered Jul 18 '22 at 07:21

from pyspark.sql.functions import lit,row_number,col
from pyspark.sql.window import Window

my_dataframe = spark.createDataFrame([('a',),('b',),('c',),('d',),('e',)],'item : string')

w = Window().partitionBy(lit('a')).orderBy(lit('a'))
df1 = my_dataframe.withColumn("row_num", row_number().over(w))

df1.show()

# +----+-------+
# |item|row_num|
# +----+-------+
# |   a|      1|
# |   b|      2|
# |   c|      3|
# |   d|      4|
# |   e|      5|
# +----+-------+

df1.filter(col('row_num').between(2,4)).show()

+----+-------+
# |item|row_num|
# +----+-------+
# |   b|      2|
# |   c|      3|
# |   d|      4|
# +----+-------+

How to select only a specific range of rows in Pyspark?

1 Answers1