fillna() not allowing floating values

Question

I'm testing a simple imputation method on the side using a copy of my dataset. I'm essentially trying to impute missing values with categorical means grouped by the target variable.

df_test_2 = train_df.loc[:,['Survived','Age']].copy()  #copy of dataset for testing

#creating impute function
def impute(df,variable):
    if 'Survived'==0: df[variable] = df[variable].fillna(30.7)
    else: df[variable] = df[variable].fillna(28.3)

#imputing
impute(df_test_2,'Age')

The output is that the imputation is successful, but the values added are 30 and 28 instead of 30.7 and 28.3. 'Age' is float64.

Thank you

Edit: I simply copied the old code for calling the function here and corrected it now. Wasn't the issue in my original code; problem persists.

You've defined `impute` but are calling `impute_age`. Is it possible `impute_age` refers to some other older version of the function that's still in memory and filled with 30 and 28? — ALollz, Nov 17 '20 at 20:22
Besides calling the wrong function you test `'Survived' == 0` which is always `False`. Your question is not reproducible. — Michael Szczesny, Nov 17 '20 at 20:23
Your function fills every `na` with `28.3`. It does not impute `28`. Please provide a [reproducible example](https://stackoverflow.com/help/minimal-reproducible-example). — Michael Szczesny, Nov 17 '20 at 20:37

score 2 · Answer 1 · answered Nov 17 '20 at 20:54

Have a look at this to see what may be going on

To test it I set up a simple case

import pandas as pd
import numpy as np

data = {'Survived' : [0,1,1,0,0,1], 'Age' :[12.2,45.4,np.nan,np.nan,64.3,44.3]}
df = pd.DataFrame(data)
df

This got the data set

    Survived    Age
0   0           12.2
1   1           45.4
2   1           NaN
3   0           NaN
4   0           64.3
5   1           44.3

I ran your function exactly

def impute(df,variable):
    if 'Survived'==0: df[variable] = df[variable].fillna(30.7)
    else: df[variable] = df[variable].fillna(28.3)

and this yielded this result

    Survived    Age
0   0.          12.2
1   1           45.4
2   1           28.3
3   0           28.3
4   0           64.3
5   1           44.3

As you can see on the index 3 the row age got filled with the wrong value. The problem is this 'Survived'==0. This is always going to be false. You are checking to see if the string is 0 and it is not.

What you may want is

df2 = df[df['Survived'] == 0].fillna(30.7)
df3 = df[df['Survived'] == 1].fillna(28.3)
dfout = df2.append(df3)

and the output is

    Survived    Age
0   0           12.2
3   0           30.7
4   0           64.3
1   1           45.4
2   1           28.3
5   1           44.3

score 0 · Answer 2 · answered Nov 17 '20 at 20:33

0

Anish

I think is better to use the method apply() available in pandas. This method applies (in rows or in columns) a custom function over a dataframe.

I let you one post: Stack Question

Documentation pandas: Doc Apply df

regards,

answered Nov 17 '20 at 20:33

sebashc3712

65
7

fillna() not allowing floating values

2 Answers2