Checking NAN & Spaces for a given column list in Pandas Data frames

Asked Dec 22 '22 at 14:40

Active Dec 22 '22 at 15:16

Viewed 28 times

I am a beginner to Python and pandas ,I need to filter rows when a given columns/column in the list is spaces or Nulls or Nan

My result should be like this

Rejection rows:-

I am storing the keys in a list for this data frame in cols_list.

cols_list = ['Product_ID','Vendor_ID']

To remove Nans:

df2 = df.dropna(axis = 0, subset = cols_list, how = 'all', inplace = True)

but I would like to remove rows if both keys have spaces or either one has spaces too. also i would like to capture the dropped rows in to another data frame .

df2 = df[df[cols_list].isna()]

Can any one please advise to check for spaces as well as Nan and remove those rows and capture the same . Since isna is not getting any of rows.Please advise and help.

edited Dec 22 '22 at 15:16

asked Dec 22 '22 at 14:40

Balaji Krishnamoorthy

First `replace('', float('nan'))`. Or use boolean indexing `m = df[cols].isna().any(axis=1) | df[cols].eq('').any(axis=1) ; print(df[~m]) ; print(df[m])` – mozway Dec 22 '22 at 15:13
I am trying to apply this – Balaji Krishnamoorthy Dec 22 '22 at 15:29
Sir, I am getting empty dataframe. – Balaji Krishnamoorthy Dec 22 '22 at 15:31
Please provide a reproducible input (not images) – mozway Dec 22 '22 at 15:39
Hi @mozway , could you please specific , i couldn't understand your question – Balaji Krishnamoorthy Dec 23 '22 at 06:14
https://stackoverflow.com/questions/20109391/how-to-make-good-reproducible-pandas-examples – mozway Dec 23 '22 at 06:19
Hi @mozway, please find the code which i tried but having said , i couldn't able to select rows which has Nan in respective columns in cols_list – Balaji Krishnamoorthy Dec 23 '22 at 07:12
df_nonull[cols_list] = df_nonull[cols_list].replace([' '],float('Nan')) – Balaji Krishnamoorthy Dec 23 '22 at 07:13
df_suc_rec=df_nonull.dropna(subset=cols_list,how='any') – Balaji Krishnamoorthy Dec 23 '22 at 07:13
df_rej_rec=df[df_nonull[cols_list].isnull()] - not giving expected rows , it just display all records with Nan – Balaji Krishnamoorthy Dec 23 '22 at 07:13
df_rej_rec=df[df_nonull['Product_ID'].isnull() | df_nonull['Vendor_ID'].isnull()] but i have this columns in the list , i need to pass this list in the dataframe to select all rows which has both columns or either one of columns is Nan Please advise. – Balaji Krishnamoorthy Dec 23 '22 at 07:22
Hi @mozway, i just got the result , thanks .. – Balaji Krishnamoorthy Dec 23 '22 at 10:21
m = df_nonull[cols_list].isna().any(axis=1) | df_nonull[cols_list].eq('').any(axis=1) df[m] – Balaji Krishnamoorthy Dec 23 '22 at 10:21

Checking NAN & Spaces for a given column list in Pandas Data frames

0 Answers0