7

I have dataset with three columns in Python notebook. It seems there are too many outliers out of 1.5 times IQR. I'm think how can I count the outliers for all columns?

If there are too many outliers, I may consider to remove the points considered as outliers for more than one feature. If so, how I can count it in that way?

Thanks!

enter image description here

Chasen Li
  • 205
  • 1
  • 3
  • 12
  • 1
    As a starting point take a look at: http://stackoverflow.com/questions/34782063/how-to-use-pandas-filter-with-iqr – albert Aug 21 '16 at 19:47

1 Answers1

24

Similar to Romain X.'s answer but operates on the DataFrame instead of Series.

Random data:

np.random.seed(0)
df = pd.DataFrame(np.random.randn(100, 5), columns=list('ABCDE'))
df.iloc[::10] += np.random.randn() * 2  # this hopefully introduces some outliers
df.head()
Out: 
          A         B         C         D         E
0  2.529517  1.165622  1.744203  3.006358  2.633023
1 -0.977278  0.950088 -0.151357 -0.103219  0.410599
2  0.144044  1.454274  0.761038  0.121675  0.443863
3  0.333674  1.494079 -0.205158  0.313068 -0.854096
4 -2.552990  0.653619  0.864436 -0.742165  2.269755

Quartile calculations:

Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1

And these are the numbers for each column:

((df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))).sum()
Out: 
A    1
B    0
C    0
D    1
E    2
dtype: int64

In line with seaborn's calculations:

enter image description here

Note that the part before the sum ((df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))) is a boolean mask so you can use it directly to remove outliers. This sets them to NaN, for example:

mask = (df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))
df[mask] = np.nan
Community
  • 1
  • 1
ayhan
  • 70,170
  • 20
  • 182
  • 203