how to delete nan in df from row without losing the whole row?

Question

G'day, how can I drop the nan values without losing the whole row? This is how my df looks like.

I have tried the pandasdf = schule.dropna() But this deleted everything in the rows.

But I have to keep the values, because at the end, I want them to go upwards.

Do you want to replace them by None? Because it is not clear what you want to achieve, there will always be a value (NaN, None or useful values) — sampers, Feb 17 '20 at 11:15
My guess is that they don't want to replace `NaN`, but instead combine all rows in the df into a single row, replacing `NaN` with the value obtained from a different row. — Wesley Bowman, Feb 17 '20 at 11:24
This looks as if fixing the parser that created the dataframe would make more sense... — MSpiller, Feb 17 '20 at 11:29
Take a look at this: https://stackoverflow.com/questions/25941979/remove-nan-cells-without-dropping-the-entire-row-pandas-python3 — Karthick Mohanraj, Feb 17 '20 at 11:30
@KarthickMohanraj thank you, this is very helpful! I will try this. If it won't work for me, I will slice the columns and do regular cleaning. — Noel Harris, Feb 17 '20 at 11:33

score 3 · Accepted Answer · answered Feb 17 '20 at 11:57

3

it might help. you can try it !!!!

answered Feb 17 '20 at 11:57

Mohit Sharma

590
3
10

score 1 · Answer 2 · edited Apr 02 '20 at 22:25

As per your problem statement, you want to give less priority to nan values and bring the non-nan values to the top.

import numpy as np
import pandas as pd
import functools

def drop_and_roll(col, na_position='last', fillvalue=np.nan):
    result = np.full(len(col), fillvalue, dtype=col.dtype)
    mask = col.notnull()
    N = mask.sum()
    if na_position == 'last':
        result[:N] = col.loc[mask]
    elif na_position == 'first':
        result[-N:] = col.loc[mask]
    else:
        raise ValueError('na_position {!r} unrecognized'.format(na_position))
    return result

df = pd.read_table('data', sep='\s{2,}')

print(df.apply(functools.partial(drop_and_roll, fillvalue='')))

score 0 · Answer 3 · answered Feb 17 '20 at 12:17

Assuming you're wanting to backfill values and then drop any duplicates that appear in any of the columns this example works:

import pandas as pd
import numpy as np

data = [
    ['POINT_1.1', 'POINT_1.2', pd.NA],
    [pd.NA, pd.NA, 'POINT_1.3'],
    ['POINT_2.1', 'POINT_2.2', pd.NA],
    [pd.NA, pd.NA, 'POINT_2.3']

]

df = pd.DataFrame(data)
df

#            0          1          2
# 0  POINT_1.1  POINT_1.2       <NA>
# 1       <NA>       <NA>  POINT_1.3
# 2  POINT_2.1  POINT_2.2       <NA>
# 3       <NA>       <NA>  POINT_2.3

t = df.T.bfill().T.bfill()
t

#            0          1          2
# 0  POINT_1.1  POINT_1.2  POINT_1.3
# 2  POINT_2.1  POINT_2.2  POINT_2.3

for column in t.columns:
    t = t.drop_duplicates(column)
t

#            0          1          2
# 0  POINT_1.1  POINT_1.2  POINT_1.3
# 2  POINT_2.1  POINT_2.2  POINT_2.3

how to delete nan in df from row without losing the whole row?

3 Answers3