How to remove Outliers in Python?

Question

I want to remove outliers from my dataset "train" for which purpose I've decided to use z-score or IQR.

I'm running Jupyter notebook on Microsoft Python Client for SQL Server.

I've tried for z-score:

from scipy import stats
train[(np.abs(stats.zscore(train)) < 3).all(axis=1)]

for IQR:

Q1 = train.quantile(0.02)
Q3 = train.quantile(0.98)
IQR = Q3 - Q1
train = train[~((train < (Q1 - 1.5 * IQR)) |(train > (Q3 + 1.5 * 
IQR))).any(axis=1)]

...which returns...

for z-score:

TypeError: unsupported operand type(s) for /: 'str' and 'int'

for IQR:

TypeError: unorderable types: str() < float()

My train dataset looks like:

# Number of each type of column
print('Training data shape: ', train.shape)
train.dtypes.value_counts()

Training data shape: (300000, 111) int32 66 float64 30 object 15 dtype: int64

Help would be appreciated.

Probably missing a few casts. But you would need to show more about your data sets in order to know — Attersson, Jan 28 '19 at 09:10
What does your train data look like? Does it contain any categorical features? — DJanssens, Jan 28 '19 at 09:11

Sergey Bushmanov · Accepted Answer · 2019-01-28T11:53:21.783

6

You're having trouble with your code because you're trying to calculate zscore on categorical columns.

To avoid this, you should first separate your train into parts with numerical and categorical features:

num_train = train.select_dtypes(include=["number"])
cat_train = train.select_dtypes(exclude=["number"])

and only after that calculate index of rows to keep:

idx = np.all(stats.zscore(num_train) < 3, axis=1)

and finally add the two pieces together:

train_cleaned = pd.concat([num_train.loc[idx], cat_train.loc[idx]], axis=1)

For IQR part:

Q1 = num_train.quantile(0.02)
Q3 = num_train.quantile(0.98)
IQR = Q3 - Q1
idx = ~((num_train < (Q1 - 1.5 * IQR)) | (num_train > (Q3 + 1.5 * IQR))).any(axis=1)
train_cleaned = pd.concat([num_train.loc[idx], cat_train.loc[idx]], axis=1)

Please let us know if you have any further questions.

PS

As well, you might consider one more approach for dealing with outliers with pandas.DataFrame.clip, which will clip outliers on a case-by-case basis instead of dropping a row altogether.

edited Jan 28 '19 at 11:53

answered Jan 28 '19 at 10:06

Sergey Bushmanov

23,310
7
53
72

Thank you for your answer. I've tried to use z-score and got an error: TypeError: include and exclude must both be non-string sequences. – Stanislav Jirák Jan 28 '19 at 11:34
This worked for me: num_train = train.select_dtypes(include=['number']) cat_train = train.select_dtypes(exclude=['number']) – Stanislav Jirák Jan 28 '19 at 11:36
Have you gotten `idx` filled with booleans `True` and `False`? If so, use it to index `num_train` and `cat_train`. It worked on my laptop perfectly. – Sergey Bushmanov Jan 28 '19 at 11:46
I did not. However, the IQR way seems to work fine and is more sutaible for my purpose otherwise. Thanks. – Stanislav Jirák Jan 28 '19 at 13:21

score 0 · Answer 2 · answered Apr 02 '22 at 07:36

0

you can use autooptimizer module.

pip install autooptimizer

from autooptimizer.process import outlier_removal

answered Apr 02 '22 at 07:36

mohamadreza barghi

11
2

How to remove Outliers in Python?

2 Answers2

Linked