Replacing Defined Outliers Using Apply/Tapply R

Question

Good Afternoon R wizards,

I searched through a few posts on replacing outliers in data set - two that came closest to answering my questions were Changing outliers for NA in all columns in a dataset in R and Replace outliers by quantiles in R

The code in the 2nd reference works great if you want to update a column or two, but I have 40+ and would like to be able to use apply function to hit all the columns at once.

I want to set a threshold "max" of quantile(probs = .75) for each column, and replace any x>"max" with "max"

set.seed(1)
x = matrix(rnorm(20), ncol = 2)
x[2, 1] = 100
x[4, 2] = 200
colnames(x) <- c("a","b")
#apply(x,2,quantile,probs = .75)

Winsor75 <- function(x) {
  Max <- quantile(x, probs = .75)

    return(Max)
}
y <- as.data.frame(x)

y$a[y$a > Winsor75(x)] <- Winsor75(x)

The last line of code effectively replaces any defined outliers (in my case values above 75%) but uses the 75% for the entire matrix "x" where as I would like (a) the quantile to be attributable to each column and for (b) the ability to use the function in apply/tapply etc so I can perform the operation on all columns efficiently.

Any suggestions?

Thanks!

Use `lapply` i.e. `y[] <- lapply(y, function(u) replace(u, u > Winsor75(u), Winsor75(u)))` — akrun, Mar 29 '18 at 17:14
I also have a lot of NA's in data so would need to be done with incomplete data — HowdyDude, Mar 29 '18 at 17:15

ngm · Accepted Answer · 2018-03-29T17:23:11.177

1

as.data.frame(lapply(y, function(x) pmin(x, quantile(x, 0.75, na.rm = TRUE))))

As a function:

df_winsor <- function(df, p) {
  as.data.frame(lapply(df, 
                       function(x) pmin(x, quantile(x, probs = p, na.rm = TRUE))))
}

Statistician's Disclaimer: I've solved the programming problem you asked. This should not be taken as an endorsement of the idea of automatically checking for, or doing anything with, so-called "outliers".

edited Mar 29 '18 at 17:23

answered Mar 29 '18 at 17:18

ngm

2,539
8
18

Upvoting for the disclaimer. Agree emphatically. – IRTFM Mar 29 '18 at 17:22

score 0 · Answer 2 · answered Mar 29 '18 at 17:42

0

One option is to use mutate_all with custom function and apply rules to all columns.

Approach:

I have crated an replaceOutlier function (based on OPs function) which calculatesMaxand then replaces any item which is more thanMaxbefore returning vector.replaceOutlieris applied over all columns usingdplyr::mutate_all`.

library(tidyverse)

replaceOutlier <- function(x) {
  Max <- quantile(x, probs = .75)
  x[x>Max] <- Max
  return(x)
}

x %>% as_tibble() %>% mutate_all(funs(replaceOutlier))

#Results
# # A tibble: 10 x 2
#     a       b
#   <dbl>   <dbl>
# 1 -0.626  1.08  
# 2  0.698  0.390 
# 3 -0.836 -0.621 
# 4  0.698  1.08  
# 5  0.330  1.08  
# 6 -0.820 -0.0449
# 7  0.487 -0.0162
# 8  0.698  0.944 
# 9  0.576  0.821 
# 10 -0.305  0.594 
#

Data

set.seed(1)
x = matrix(rnorm(20), ncol = 2)
x[2, 1] = 100
x[4, 2] = 200
colnames(x) <- c("a","b")

answered Mar 29 '18 at 17:42

MKR

19,739
4
23
33

Thanks guys, I appreciate the responses – HowdyDude Mar 29 '18 at 18:26
@ALMDude You are welcome. I appreciate your efforts towards providing details along with question. Also you can accept an answer by clicking on `tick` symbol in left of the answer. It will be helpful to future users in that way. – MKR Mar 29 '18 at 18:38
No problem, all answers worked for the ones I could run, for some reason I am having tidyverse/dplyr/pillar issues in regards to packages and could not test out your option yet - looks like a flexible and viable option though! – HowdyDude Mar 30 '18 at 12:16

Replacing Defined Outliers Using Apply/Tapply R

2 Answers2