Replacement of empty strings in data frame takes very long

Question

I have a dataframe in R with 11M rows and 46 columns. Some of the fields contain empty strings (""). I need to replace those empty strings with NAs because write.dta (in the foreign package) can't handle empty strings.

My for-loop, however, takes very long (around 15 minutes per column; sometimes R/ entire system crashes). I'm running RStudio (R 3.0.2) on a 8GB RAM Mac. Does anyone know of a faster way?

for (i in 1:46){

   if (length(which(myDF[,i]==""))!=0) {

    myDF[,i][which(myDF[,i]=="")]<-NA 

   }
}

`for`-loops in R are notorious slow. Avoid when possible. You can instead look at `apply`, `sapply`, `lapply`. Or see solution of @Zbynek. — MrGumble, Feb 06 '14 at 09:41

score 10 · Accepted Answer · answered Feb 06 '14 at 09:25

10

This should work:

myDF[myDF==''] <- NA

answered Feb 06 '14 at 09:25

Zbynek

5,673
6
30
52

Do you have a suggestion to make this work with data frames that have posix columns? I like your solution but it produces error "Error in as.POSIXlt.character(x, tz, ...): character string is not in a standard unambiguous format" and I would like to preserve data types. – trisaratops Aug 07 '17 at 19:24
@triSaratops I think you can not have empty string in POSIXlt column - `as.POSIXct('')` produces error. Could you please post example of your data? – Zbynek Aug 16 '17 at 08:55

score 2 · Answer 2 · answered Feb 06 '14 at 12:46

2

You can also use the is.na<- function:

is.na(myDF) <- myDF == ''

answered Feb 06 '14 at 12:46

Sven Hohenstein

80,497
17
145
168

Replacement of empty strings in data frame takes very long

2 Answers2