Function to remove outliers in python

Question

I am trying to write a function to update all the outliers in all the columns in a dataset with the interquartile range. It is working when I pass a column as input but if I add another loop to iterate through all the columns its not working.

df2ColumnNames=df2.columns

def fixoutliers(x):

for i in df2ColumnNames:
    print("colnames ",i)
    xy=x[i]    
    print(xy)
    updated=[]
    Q1,Q3=np.percentile(xy,[25,75])
    IQR=Q3-Q1
    #print(IQR)
    minimum=Q1-1.5*IQR
    maximum=Q3+1.5*IQR
    print("maximum",maximum)
    for i in xy:
        if(i>maximum):
            i=maximum
            updated.append(i)
        elif(i<minimum):
            i=minimum
            updated.append(i)
        else:
            print("In else")
            updated.append(i)
    return updated

You may want to add the `numpy` and `pandas` tags as well. – Dillon Davis Jul 04 '18 at 19:15 — Dillon Davis, Jul 04 '18 at 19:15

score 1 · Answer 1 · answered Jul 05 '18 at 07:42

Thanks all for your suggestions. With a bit of struggle I managed to create the function which I was after. Posting the solution if it helps someone

#####Define a function which takes input a dataframe(x) that can contain both numeric and categorical columns######

def fixoutliers(x):

##Get all the column name from the input dataframe x
xColumnNames=x.columns
print(xColumnNames)
#for j in df2ColumnNames:

for j in xColumnNames:
    try:
        print("colnames ",j)
        xy=x[j]    
        mydata=pd.DataFrame()
        #print(xy)
        updated=[]
        Q1,Q3=np.percentile(xy,[25,75])
        IQR=Q3-Q1
        minimum=Q1-1.5*IQR
        maximum=Q3+1.5*IQR
        for i in xy:
            if(i>maximum):
                print("Entering maxim")
                i=maximum
                updated.append(i)
            elif(i<minimum):
                print("enterinf minimum")
                i=minimum
                updated.append(i)
            else:
                updated.append(i)
        x[j]=updated
    except:
        continue
return x

score 0 · Answer 2 · answered Jul 05 '18 at 05:11

0

Since boxplot is also using the same theory 'inter-quartile range' to detect outliers, you can use it directly to find outliers on your dataframe.

import pandas as pd

_, bp = pd.DataFrame.boxplot(df2, return_type='both')
outliers = [flier.get_ydata() for flier in bp["fliers"]]
out_liers = [i.tolist() for i in outliers]

answered Jul 05 '18 at 05:11

Surani Matharaarachchi

565
2
13

Percentile-based outlier detection is problematic. [There is for instance this great post explaining it.](https://stackoverflow.com/a/22357811/8881141) – Mr. T Jul 05 '18 at 07:18
Thanks for the link...I will check my distribution properly before treating the outliers – Tinku Jul 05 '18 at 08:23

Function to remove outliers in python

2 Answers2