Unable to remove irrelevant floating point value

Question

Sample of dataset Full sample can be download via this link

Date/Time,Hs,Hmax,Tp,Tz,Peak Direction,SST
1/01/2018 0:00,-99.9,-99.9,-99.9,-99.9,-99.9,-99.9
1/01/2018 0:30,0.513,0.81,10.315,4.748,-99.9,-99.9
1/01/2018 1:00,0.566,0.93,10.778,5.003,92,26.4
1/01/2018 1:30,0.557,0.85,9.984,4.99,91,26.4

Read in via this method, and all columns except date.time are numeric.

maloolaba.waves <- read.csv(file = "./data/mooloolaba_2018-01-01t00_00-2018-10-31t23_30.csv", header = T)

Function to remove rows containing -99.9.

maloo.RM.outlier <- maloolaba.waves[!(apply(maloolaba.waves, 1, 
                             function(y) any(y == -99.9) )),]

Now when I do summary after removing value -99.9 I get this.

summary(maloo.RM.outlier)

          Date.Time           Hs               Hmax        
 1/01/2018 1:00 :    1   Min.   :-99.900   Min.   :-99.900  
 1/01/2018 1:30 :    1   1st Qu.:  0.805   1st Qu.:  1.350  
 1/01/2018 10:00:    1   Median :  1.112   Median :  1.870  
 1/01/2018 10:30:    1   Mean   :  1.234   Mean   :  2.089  
 1/01/2018 11:00:    1   3rd Qu.:  1.608   3rd Qu.:  2.700  
 1/01/2018 11:30:    1   Max.   :  4.257   Max.   :  7.262  
 (Other)        :14543                                      
       Tp                Tz          Peak.Direction      SST       
 Min.   :-99.900   Min.   :-99.900   Min.   :  5    Min.   :19.80  
 1st Qu.:  7.529   1st Qu.:  5.035   1st Qu.: 91    1st Qu.:21.00  
 Median :  9.146   Median :  5.568   Median :105    Median :23.00  
 Mean   :  9.245   Mean   :  5.679   Mean   :103    Mean   :23.43  
 3rd Qu.: 10.903   3rd Qu.:  6.257   3rd Qu.:119    3rd Qu.:26.00  
 Max.   : 21.121   Max.   : 10.146   Max.   :358    Max.   :28.65 

Yet when I look at the dataset for maloo.RM.outlier, there are no values -99.9, so I then searched.

which(maloo.RM.outlier$Hs == -99.9, arr.ind = T)

[1] 11501 13775

I have tried looking at the row numbered 11501 and 13775, no -99.9 values there. I have tried, clearing the global environment data, restarting the R session and nothing seems to fully get rid of the value -99.9 and the summary still says the minimum = -99.9. Does anyone know how to remove floating point values?

Answer 1

Because of R's FAQ 7.31, you can't really test for floating-point equality, just an approximation. There are several ways to do it, but a popular (and my favorite) way is to subtract my limit/equality and look for something below a threshold.

Because the actual value might be on either side (pos/neg) of my comparison value, we can use the absolute value to take that into account. The resulting code changes your

any(y == -99.9)

to

any( abs(y + 99.9) < 1e-9 )

Which, coincidentally, is precisely what Marius' suggested function (dplyr::near) is doing:

dplyr::near
# function (x, y, tol = .Machine$double.eps^0.5) 
# {
#     abs(x - y) < tol
# }
# <bytecode: 0x000000002506d7b8>
# <environment: namespace:dplyr>

though it is using a slightly more robust way of finding something of just-barely-above a "near-zero" magnitude by using .Machine$double.eps^0.5.

I chose 1e-9 for code-golf, though if you are programming something, you should probably name it something meaningful so it isn't a "magic constant". Perhaps tol <- 1e-9 or eps <- 1e-9 (for epsilon, a variable frequently used to indicate an arbitrarily small, positive number).