How to select or drop a designated row in pyspark dataframe

Question

How to select or drop a designated row in pyspark dataframe? such as drop third row in dataframe

Possible duplicate of [How take a random row from a PySpark DataFrame?](https://stackoverflow.com/questions/34003314/how-take-a-random-row-from-a-pyspark-dataframe) — Gábor Bakos, Nov 17 '19 at 16:03

score 0 · Answer 1 · edited Jun 20 '20 at 09:12

0

You can use where or filter functions to achieve this as shown below:-

df.filter($"age" > 15)
df.where($"age" > 15)

Update to drop by column index

val col = df.columns
val n   = df.columns.length 
val toBeDropped = n-1 // to drop last column  and so on..
val oldDf = df.drop(col(ToBeDropped ))

edited Jun 20 '20 at 09:12

Community

answered Nov 17 '19 at 15:28

I want drop a column by index – wa007 Nov 18 '19 at 07:33
I have updated the answer, if this was helpful pls do accept the answer. – Jayadeep Jayaraman Nov 18 '19 at 07:49
That is not possible without getting all the rows in the driver which will lead OOM errors if the data is large. The best way to do would be to use the `filter` statement and programmatically determining the condition – Jayadeep Jayaraman Nov 18 '19 at 08:13