2

I did look up a similar example which used 

## Some sample data
set.seed(0)
dat <- matrix(1:100, 10, 10)
dat[sample(1:100, 50)] <- NA
dat <- data.frame(dat)
## Remove columns with more than 50% NA
dat[, -which(colMeans(is.na(dat)) > 0.5)]

But I am not sure how to convert it into a number and not a percentage.

NelsonGon
  • 13,015
  • 7
  • 27
  • 57
Dianna Li
  • 121
  • 6
  • This example does not have 2000 NAs, how many should be used then? – NelsonGon Feb 23 '20 at 06:43
  • You are using R. RStudio is just an IDE that helps you write code more efficiently. – Roman Luštrik Feb 23 '20 at 06:57
  • Related: [Subset dataframe based on number of observations in each column](https://stackoverflow.com/questions/53646975/subset-dataframe-based-on-number-of-observations-in-each-column) – markus Feb 23 '20 at 07:58

4 Answers4

3

One base R option could be:

dat[, colMeans(is.na(dat)) <= 0.5]

   X1 X2 X4 X5 X6 X8 X10
1  NA 11 NA NA NA 71  NA
2  NA 12 32 NA 52 72  NA
3   3 NA 33 NA 53 73  93
4   4 14 NA 44 NA NA  94
5   5 15 35 NA 55 75  95
6  NA NA 36 46 NA 76  NA
7  NA NA NA 47 57 NA  97
8   8 18 NA 48 NA 78  98
9   9 NA 39 NA 59 79  99
10 NA NA 40 50 NA 80 100

Or using a specified number:

dat[, colSums(is.na(dat)) <= 5]

Or using half of the rows as a criteria:

dat[, colSums(is.na(dat)) <= nrow(dat)/2]

And the same idea with dplyr:

dat %>%
 select_if(~ mean(is.na(.)) <= 0.5)

Or using a specified number:

dat %>%
 select_if(~ sum(is.na(.)) <= 5)

Similarly, using half of the rows as a criteria:

dat %>%
 select_if(~ sum(is.na(.)) <= length(.)/2)
tmfmnk
  • 38,881
  • 4
  • 47
  • 67
3

Or you could also count them: 

dat[, -which(colSums(is.na(dat)) > 2000)]
Thomas L.
  • 524
  • 1
  • 5
  • 17
2

Using purrr:

purrr::discard(dat, ~sum(is.na(.x)) > 5)
   X1 X2 X3 X5 X6 X7 X8
1  NA 11 NA 41 NA 61 71
2  NA 12 NA NA 52 62 NA
3   3 13 23 NA 53 63 NA
4   4 NA NA NA NA NA NA
5   5 15 NA NA 55 65 NA
6  NA 16 26 46 56 66 76
7  NA 17 27 47 57 67 77
8   8 NA NA 48 58 NA 78
9   9 19 29 49 NA NA NA
10 10 NA 30 50 60 NA 80

Alternatively:

purrr::keep(dat, ~sum(is.na(.x)) <= 5)
   X1 X2 X3 X5 X6 X7 X8
1  NA 11 NA 41 NA 61 71
2  NA 12 NA NA 52 62 NA
3   3 13 23 NA 53 63 NA
4   4 NA NA NA NA NA NA
5   5 15 NA NA 55 65 NA
6  NA 16 26 46 56 66 76
7  NA 17 27 47 57 67 77
8   8 NA NA 48 58 NA 78
9   9 19 29 49 NA NA NA
10 10 NA 30 50 60 NA 80
NelsonGon
  • 13,015
  • 7
  • 27
  • 57
0

I multiplied it for 100 to keep it as percentage. For you should look like this:

##Keep only the columns that their NA values are not greater than 50%

dat<-dat[(colMeans(is.na(dat)))*100 <= 50]
Marco
  • 2,368
  • 6
  • 22
  • 48
Dave_Giaco
  • 1
  • 2