0

I'm trying to remove numeric columns with sd=0, and this error "Error: memory exhausted (limit reached?)" pops up, my "data" was read with fread.

data2 <- data[ - as.numeric(which(apply(data, 2, var) == 0))]
Error: memory exhausted (limit reached?)

Is there a way around this?

SecretAgentMan
  • 2,856
  • 7
  • 21
  • 41
Sue
  • 57
  • 1
  • 7
  • You can try to simplify this expression using several variables – HubertL Nov 07 '20 at 01:15
  • If you're really trying to remove columns, you should put a comma in front on your internal code, alongside `with=F`. It'll become `data[, - as.numeric(which(apply(data, 2, var) == 0)), with=F]`. – Cainã Max Couto-Silva Nov 07 '20 at 05:10

2 Answers2

1

Try using the data.table syntax:

library(data.table)
idx <- data[, sapply(.SD, function(col) var(col)==0)]
data2 <- data[, -idx, with=FALSE]
Cainã Max Couto-Silva
  • 4,839
  • 1
  • 11
  • 35
0

You can use colVars from matrixStats package which is usually fast.

data2 <- data[matrixStats::colVars(as.matrix(data)) != 0, ]
Ronak Shah
  • 377,200
  • 20
  • 156
  • 213
  • Tried matrixStats, and had this error>>Error: cannot allocate vector of size 1.4 Gb – Sue Nov 07 '20 at 02:02