0

I need to remove outliers from a variable which contains several NANs in it. It looks like this:

 X-velocity

1   0.0345
2   0.0222
3   0.0034
4   0.5604
5   0.4326
6    NaN
7   0.0333
8   0.3635
9   0.3345
10  0.3468
11  0.4573
12  0.7985
13  0.9359
14  NAN
15  0.4635
16  0.6857
17  0.4239
18  NAN
19  0.3849
20  0.3726
21  0.4637
22  0.3647
23  NAN
24  0.2938
25  0.5227

I need to remove the outlier from the variable without deleting or changing the value or position of the NAN. I don't mean NAN is the outlier, I mean the outlier of the continuous numbers. for example, I want to remove all the numbers that out of the range of mean +/- 3 * standard deviation. When I'm doing the outlier detection and removing, I don't want to affect the NANs, I want them to be there (since I need to perform other operation based on the NANs later).

Is there any possible way to do it? I appreciat any help.

Leran
  • 39
  • 7
  • Depends, what you consider an outlier. Please define it in your question. What would you do if there were no NaN values? You should also provide a [Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve). – Mr. T Nov 05 '18 at 18:57
  • How about creating another 'column' (X_velocity_is_NaN), fill it with 0s and 1s (depending on the NaN values in the original column), and then doing outlier and/or NaN removal in the original column? – moe Nov 05 '18 at 19:01
  • @ moe Your solution is close to the answer I accepted. So thanks as well! – Leran Nov 06 '18 at 05:58

1 Answers1

2

If you have a method to determine whether something is an outlier or not (I imagine you have some threshold) you can create a new column that stores this flag.

For example:

# [True or False] is this more than 3 standard deviations away from the mean
df['is_outlier'] = abs(df['X-velocity'] - (df['X-velocity'].mean())/df['X-velocity'].std() > 3

You can then select values by using both this outlier flag OR whether the value is null:

# Select rows that contain non-outliers or null values 
filtered = df[(~df.is_outlier) | df['X-velocity'].isnull()]
jfbeltran
  • 1,808
  • 3
  • 13
  • 17