6

I am running some basic data manipulation on a Macbook Air (4GB Memory, 120GB HD with 8GB available). My input file is about 40 MB, and I don't write anything to the disk until end of the process. However, in the middle of my process, my Mac says there's no memory to run. I checked hard drive and found there's about 500MB left. So here are my questions:

  1. How is it possible that R filled up my disk so quickly? My understanding is that R store everything in memory (unless I explicitly write something out to disk).
  2. If R does write temporary files on the disk, how can I find these files to delete them?

Thanks a lot.

Update 1: error message I got: 

Force Quit Applications: Your Mac OS X startup disk has no more space available for 
application memory

Update 2: I checked tempdir() and it shows "var/folders/k_xxxxxxx/T//Rtmpdp9GCo". But I can't locate this director from my Finder

Update 3: After unlink(tempdir(),recursive=TRUE) in R and restarting my computer, I got my disk space back. I still would like to know if R write on my hard drive to avoid similar situations in the future.

Update 4: My main object is about 1GB. I use Activity Monitor to track process, and while Memory usage is about 2GB, Disk activity is extremely high: Data read: 14GB, data write, 44GB. I have no idea what R is writing.

AdamNYC
  • 19,887
  • 29
  • 98
  • 154

2 Answers2

7

R writes to a temporary per-session directory which it also cleans up at exit.

It follows convention and respects TMP and related environment variables.

Dirk Eddelbuettel
  • 360,940
  • 56
  • 644
  • 725
  • Is there any reason that disks should be involved here? Might just be running out of physical RAM...? – Gavin Simpson Nov 26 '12 at 15:12
  • Thanks Dirk. If I force quit R, is there any way to clean it up manually? – AdamNYC Nov 26 '12 at 15:36
  • @AdamNYC [this](https://stackoverflow.com/a/56715074/5783745) shows how to completely remove all temporary files in the current session – stevec Jun 22 '19 at 13:39
2

What makes you think that disk space has anything to do with this? R needs all objects held in memory, not off disk (by default; there are add-on packages that allow a subset of operations on on-disk stored files too big to fit into RAM).

One of the steps in the "process" is causing R to request a chunk of RAM from the OS to enable it to continue. The OS could not comply and thus R terminated the "process" that you were running with the error message you failed to give us. [Hint, it would help if you showed the actual error not your paraphrasing thereof. Some inkling of the code you were running would also help. 40MB on-disk sounds like a reasonably large file; how many rows/columns etc.? How big is the object within R; object.size()?

Gavin Simpson
  • 170,508
  • 25
  • 396
  • 453
  • Hi Gavin, please see my update. I think it is disk space b/c at the beginning of the process I had 8GB in my harddrive, and in the middle of the process, I got only 500MB (I ran no other applications). I have about 3 million rows with around 30 columns. I can't test my object size anymore b/c of the above mentioned error. :-( – AdamNYC Nov 26 '12 at 15:30
  • ?? So it bails out just loading that data into R? Hint, can you break this "process" down into individual chunks? If you can't even read the data in then see `?read.table` (assuming this is tabular/spreadsheet data), as passing in the data types may help. – Gavin Simpson Nov 26 '12 at 15:33
  • Though I don't have MacOS X on any machine here, that sounds like you've exhausted all the physical RAM and available SWAP space. – Gavin Simpson Nov 26 '12 at 15:34
  • Is SWAP space part of hard drive? Sorry for a basic question :) – AdamNYC Nov 26 '12 at 15:35
  • 1
    Yes, on Linux it is a separate partition of the disk that the OS can treat as "memory", but it is slow so it pages memory addressed from RAM out to the SWAP space for only those addresses not being used. That said, the SWAP space is usually already allocated so won't show up in the un-used space on your disk; it is already in-use, even if there is nothing currently in the SWAP. – Gavin Simpson Nov 26 '12 at 15:37