0

I have a large csv file (14GB, 3.6M rows) and its delimiter is (sep = '\u0007').

df <- read.csv("test.csv", header = TRUE, sep='\u0007')
Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec,  : 
  could not allocate memory (2048 Mb) in C function 'R_AllocStringBuffer'

I've already added "R_MAX_VSIZE=150Gb" in the .Renviron file. Can someone help to check?

And if I removed the sep='\u0007', the data can be loaded but the formate is wrong.

My laptop details:

macOS Catalina
Processor: 2.9 GHz 6-Core Intel Core i9
Memory: 32 GB 2400 MHz DDR4
    —Memory Used: 15.13GB, Cached files: 2.92GB, Swap Used: 1.5 MB
Macintosh HD: 270.08GB available of 499.96GB
CPU: idle 94.72%
coco
  • 871
  • 1
  • 7
  • 15
  • I would think that anyone trying to help would need more specifics about OS and how many other processes ae running that might be competing for space. R objects need contiguous memory space. – IRTFM Apr 27 '22 at 03:15
  • @IRTFM Thanks for the reminder, I just updated the laptop and system info – coco Apr 27 '22 at 04:06
  • The error suggests that you have lots of other processes active. Shut down all your active programs except R. Do NOT keep an `.Rdata` file because it may be part of the problem. If you don't know what that is, then do a search on SO and delete the one that's probably there. Reboot and only start R. Retry. You should have plenty of Ram. – IRTFM Apr 27 '22 at 04:16
  • https://stackoverflow.com/questions/1358003/tricks-to-manage-the-available-memory-in-an-r-session?r=SearchResults&s=1|112.8473 – IRTFM Apr 27 '22 at 04:18

0 Answers0