How to iterate over DataFrame columns and drop NaN value

Question

I have this example DataFrame:

d = {'col1': [1, 2, np.NaN], 'col2': [3, np.NaN, 4], 'col3': [np.NaN, 5, 6]}
df = pd.DataFrame(data=d)

and want to drop NaN values in that way to get the resulting dataFrame in this shape

I tried to iterate over columns and use dropna() function on each column but that didn't work:

for i in columns:
    df3[i].dropna()

Next when I tried :

df = df.dropna(subset=['col1', 'col2','col3'])

I ended up with df with all rows dropped:

score 1 · Answer 1 · edited Jan 30 '21 at 21:15

1

You do not need to iterate. You can use the dropna command:

df = df.dropna(subset=['col1', 'col2'])

edited Jan 30 '21 at 21:15

Yevhen Kuzmovych

answered Jan 30 '21 at 20:14

gtomer

2

Applying this to his example and adding 'col3' will drop all rows. – Stefan Jan 30 '21 at 20:23
Yes, this solution dropped all rows when I added the 'col3' – madmax80 Jan 30 '21 at 20:28

score 1 · Accepted Answer · answered Jan 30 '21 at 20:20

1

You can do this in one line using 'dropna' of pandas. No need to iterate. Already asked here: How to remove blanks/NA's from dataframe and shift the values up

df = df.apply(lambda x: pd.Series(x.dropna().values))

answered Jan 30 '21 at 20:20

Stefan

2 Answers2