6

I wanted to apply median-absolute-deviation (MAD) based outlier detection using the answer from @Joe Kington as given below:

Pythonic way of detecting outliers in one dimensional observation data

However, what's going wrong with my code, I could not figure out how to assign the outliers as nan values for MY DATA:

import numpy as np
data = np.array([55,32,4,5,6,7,8,9,11,0,2,1,3,4,5,6,7,8,25,25,25,25,10,11,12,25,26,27,28],dtype=float)
median = np.median(data, axis=0)
diff = np.sum((data - median)**2, axis=-1)
diff = np.sqrt(diff)
med_abs_deviation = np.median(diff)
modified_z_score = 0.6745 * diff / med_abs_deviation
data_without_outliers = data[modified_z_score < 3.5]
?????
print data_without_outliers
Community
  • 1
  • 1

2 Answers2

1

What is the problem with using:

data[modified_z_score > 3.5] = np.nan

Note that this will only work if data is a floating point array (which it should be if you are calculating MAD).

ebarr
  • 7,704
  • 1
  • 29
  • 40
  • 1
    You may want to be more explicit and set the `dtype` to either `"float32"` or `np.float32`. – ebarr Apr 02 '14 at 07:57
1

The problem might be line:

diff = np.sum((data - median)**2, axis=-1)

Applying np.sum() will collapse the result to scalar. Remove top-level sum, and your code will work. Other way around it to ensure that that data is at least 2d array. You can use numpy.atleast_2d() for that.

In order to assign NaNs, follow answer from https://stackoverflow.com/a/22804327/4989451

tstanisl
  • 13,520
  • 2
  • 25
  • 40