0

Is it possible to partially generate data.tabel bigger than RAM and partially save it to the hard drive? Then in code subsequently load parts of the file to RAM, and do calculations on those parts.

Thanks

eod
  • 449
  • 3
  • 17
  • 3
    You could have a look at [disk.frame](https://diskframe.com/index.html) which allows [data.table syntax](https://diskframe.com/articles/data-table-syntax.html) – Waldi Jan 12 '22 at 07:51
  • 1
    An example : https://stackoverflow.com/q/63851782/13513328, not sure [issue with data.table](https://github.com/xiaodaigh/disk.frame/issues/239) is resolved – Waldi Jan 12 '22 at 08:03
  • None of `data.frame`, `data.table`, nor `tbl_df` are capable of dealing with objects that are larger than RAM supports. `dbplyr` provides `tibble`-like functionality that allows you to work with data in a database, which could easily be larger-than-RAM as a whole, so that's possible. `disk.frame` works as partial-in-memory and can support (among other things) `data.table`-like syntax, if that's your preference. Alternatively, though, you may want to consider a different approach: *use a database*. With docker it's not hard to do so on a PC with modest capabilities. – r2evans Jan 21 '22 at 21:13

0 Answers0