Removing NA observations with dplyr::filter()

Question

My data looks like this:

library(tidyverse)

df <- tribble(
    ~a, ~b, ~c,
    1, 2, 3, 
    1, NA, 3, 
    NA, 2, 3
)

I can remove all NA observations with drop_na():

df %>% drop_na()

Or remove all NA observations in a single column (a for example):

df %>% drop_na(a)

Why can't I just use a regular != filter pipe?

df %>% filter(a != NA)

Why do we have to use a special function from tidyr to remove NAs?

because any comparison with `NA`, including `NA==NA`, will return `NA` — Marat Talipov, Mar 04 '15 at 15:06
@Metrics: it's weird (to me) that that works. Can you explain why it does? — Ben Bolker, Mar 04 '15 at 15:08
By the way, this has nothing specifically to do with `dplyr/filter`. — Ben Bolker, Mar 04 '15 at 15:09
related but not quite identical: http://stackoverflow.com/questions/25100974/na-matches-na-but-is-not-equal-to-na-why — Ben Bolker, Mar 04 '15 at 15:12
@user3731467 I don't have the `diamonds` data, but on an example data, the suggestion by Metrics worked — akrun, Mar 04 '15 at 15:13
This is also a good, related answer http://stackoverflow.com/questions/16822426/r-dealing-with-true-false-na-and-nan — bjoseph, Mar 04 '15 at 15:18
@BenBolker If you convert your comment to an answer I will accept. I did not realize that NAs behaved this way! — emehex, Mar 04 '15 at 15:22
Which comment? Looks to me like @MaratTalipov's comment might be the one you think is the answer? — Ben Bolker, Mar 04 '15 at 15:24

score 88 · Answer 1 · edited Aug 08 '17 at 21:25

88

For example:

you can use:

df %>% filter(!is.na(a))

to remove the NA in column a.

edited Aug 08 '17 at 21:25

Petter Friberg

21,252
9
60
109

answered Aug 08 '17 at 20:54

JeffZheng

1,277
1
10
13

This takes far too long for large data (dataframe with 7.3m rows and 290 cols).. any faster methods? – MJimitater Aug 10 '22 at 07:46

score 45 · Answer 2 · answered Apr 13 '20 at 19:54

45

If someone is here in 2020, after making all the pipes, if u pipe %>% na.exclude will take away all the NAs in the pipe!

answered Apr 13 '20 at 19:54

shacke

592
4
11

score 38 · Accepted Answer · edited May 23 '17 at 10:30

From @Ben Bolker:

[T]his has nothing specifically to do with dplyr::filter()

From @Marat Talipov:

[A]ny comparison with NA, including NA==NA, will return NA

From a related answer by @farnsy:

The == operator does not treat NA's as you would expect it to.

Think of NA as meaning "I don't know what's there". The correct answer to 3 > NA is obviously NA because we don't know if the missing value is larger than 3 or not. Well, it's the same for NA == NA. They are both missing values but the true values could be quite different, so the correct answer is "I don't know."

R doesn't know what you are doing in your analysis, so instead of potentially introducing bugs that would later end up being published an embarrassing you, it doesn't allow comparison operators to think NA is a value.

score 1 · Answer 4 · edited Dec 24 '21 at 01:30

1

I always use this and it is working perfectly

cool$day[cool$day==''] <- NA  
cool$day[is.na(cool$day)] <- "NA"

cool <- cool[!cool$day == "NA", ]

edited Dec 24 '21 at 01:30

Henry Ecker

34,399
18
41
57

answered Dec 23 '21 at 22:07

Anya Sti

131
2
5

score 0 · Answer 5 · answered Mar 26 '23 at 12:51

Another option could be using complete.cases in your filter to for example remove the NA in the column A. Here is some reproducible code:

library(dplyr)
df %>%
  filter(complete.cases(a))
#> # A tibble: 2 × 3
#>       a     b     c
#>   <dbl> <dbl> <dbl>
#> 1     1     2     3
#> 2     1    NA     3

^{Created on 2023-03-26 with reprex v2.0.2}

Removing NA observations with dplyr::filter()

5 Answers5

Linked

Related