0

I am trying to import a 15.51 GB dataset into R on my macbook air:

enter image description here

The file is currently a .sav and my code for importing is:

Sys.setenv(R_MAX_VSIZE = 16e9)

one <-  read.spss('NIS.sav', reencode='utf-8')

I get the vector memory exhausted limit reached error message when I try to import.

Is there a way for my to import this dataset? I am happy to just select my specific columns of interest for import instead, if that is a possibility.

davidk
  • 133
  • 2
  • 11
  • You should try to set the memory.limit up. See here for an example: [link]https://stackoverflow.com/questions/5171593/r-memory-management-cannot-allocate-vector-of-size-n-mb EDIT: I see its only for windows PCs. Do you have 32 bit or 64bit Rstudio? – pbraeutigm May 16 '21 at 15:47
  • 7
    You're trying to read a 16G file on a machine with 8G of memory? That seems like it's going to problematic no matter what. There are tools that allow you to use data sets larger than your memory (see the CRAN "high performance task view"), but applying them to an SPSS-format file could be tricky ... – Ben Bolker May 16 '21 at 15:50
  • See also [this thread](https://stackoverflow.com/questions/67425654/what-is-a-good-trick-for-loading-a-db-file-into-r-that-exhausts-memory). – tpetzoldt May 16 '21 at 15:51
  • I don't know where to even begin on this ... 15.51GB of data in 8GB of RAM, might be *possible* if you have sufficient *virtual memory* (i.e., a swap partition or swap-file), but once you start *thrashing* from ram to disk to ram to disk, there's nothing R or anything else can do to really improve the abhorrent performance that will ensue. Even if 15.51GB is in an inefficient format and it can be compressed somewhat in memory ... but that's not really likely, and much less son on a 2-to-1 ratio. – r2evans May 17 '21 at 00:58

0 Answers0