3

Let me start by saying I am rather new to R and generally consider myself to be a novice programmer...so don't assume I know what I'm doing :)

I have a large matrix, approximately 300,000 x 14. It's essentially a 20-year dataset of 15-minute data. However, I only need the rows where the column I've named REC.TYPE contains the string "SAO " or "FL-15".

My horribly inefficient solution was to search the matrix row by row, test the REC.TYPE column and essentially delete the row if it did not match my criteria. Essentially...

   j <- 1
   for (i in 1:nrow(dataset)) {
      if(dataset$REC.TYPE[j] != "SAO  " && dataset$RECTYPE[j] != "FL-15") {
        dataset <- dataset[-j,]  }
      else {
        j <- j+1  }
   }

After watching my code get through only about 10% of the matrix in an hour and slowing with every row...I figure there must be a more efficient way of pulling out only the records I need...especially when I need to repeat this for another 8 datasets.

Can anyone point me in the right direction?

Jilber Urbina
  • 58,147
  • 10
  • 114
  • 138
MattB
  • 31
  • 3
  • 1
    your question would be much more meaningful to others if you provides a small reproducible example. Right now we have to guess what `dataset` is. This makes the question much more localized to you and less helpful to future searchers. – Tyler Rinker Mar 03 '13 at 07:29
  • 2
    What's *really* killing you here is the fact that you're rewriting the dataset all the time. Don't do that! – Glen_b Mar 03 '13 at 08:30

3 Answers3

4

You want regular expressions. They are case sensitive (as demonstrated below).

x <- c("ABC", "omgSAOinside", "TRALAsaoLA", "tumtiFL-15", "fl-15", "SAOFL-15")
grepl("SAO|FL-15", x)
[1] FALSE  TRUE FALSE  TRUE FALSE  TRUE

In your case, I would do

subsao <- grepl("SAO", x = dataset$REC.TYPE)
subfl <- grepl("FL-15", x = dataset$RECTYPE)
#mysubset <- subsao & subfl # will return TRUE only if SAO & FL-15 occur in the same line
mysubset <- subsao | subfl # will return TRUE if either occurs in the same line
dataset[mysubset, ]
Roman Luštrik
  • 69,533
  • 24
  • 154
  • 197
  • Roman, I think OP's problem is more fundamental. Instead of using a `for` loop, he needs to be using `[` to extract the rows he wants. As far as his question goes, you can assume that he's trying to match either of the strings "SAO " or "FL-15" exactly. – Wilduck Mar 03 '13 at 06:33
4

I couldn't tell from the code you posted but if your data is already in a data.frame, you can do this directly. If not, first run dataset <- data.frame(dataset).

From there:

dataset[dataset$REC.TYPE == "SAO  " | dataset$RECTYPE == "FL-15",]

should return what you're looking for. For loops are horribly inefficient in R. Once you've read through the R tutorial, the R inferno will tell you how to avoid some common pitfalls.

The way this particular line works is to filter the data frame, by only returning rows that match the criteria. You can type ?[ into your R interpeter for more information.

Wilduck
  • 13,822
  • 10
  • 58
  • 90
  • 2
    For loops are just fine, you just have to set everything up. Real benefits come from vectorization, though. See http://stackoverflow.com/a/3131278/322912 – Roman Luštrik Mar 03 '13 at 06:32
  • For loops and apply take similar amounts of time, yes. But vectorization is much faster than either. The version of the code that I've posted is equivalent to vectorization. Instead of checking for equality in each row, you check for equality as a vector operation. – Wilduck Mar 03 '13 at 06:35
  • In any case, +1 for reference to R inferno. OP is smart enough to appreciate its message. – Roman Luštrik Mar 03 '13 at 20:19
3

As other posters have said, repeating the subset [ operation is slow. Instead, functions that operate over the entire vector are preferable.

I assume that both your criteria affect REC.TYPE. My solution uses the function %in%:

dataset <- dataset[dataset$REC.TYPE %in% c("SAO","FL-15"),]
Blue Magister
  • 13,044
  • 5
  • 38
  • 56