Processing without entirely loading a table

Question

I have a file of approximately 40Gb and i need to perform some analytics activity on it but my ram is limited to 4gb any suggestions how to proceed?

I've voted to close this answer because it is too broad. You will get a better answer if you show what you have tried to do and why it didn't work. — Mark Lakata, Nov 22 '16 at 17:54
What kind of analytics are you trying to do? Maybe you could process it in chunks? — William, Nov 22 '16 at 18:25
see the "Large Memory and Out-of-Memory Data" section of the [High-Performance Computing Task View](https://cran.r-project.org/web/views/HighPerformanceComputing.html) — Ben Bolker, Nov 22 '16 at 19:57

score 1 · Answer 1 · edited May 23 '17 at 12:24

1

Check out the LaF package. It allows you to do column selection and filtering on files that would not otherwise fit into memory.

Documentation and examples are a little thin on the ground which is a shame given that the syntax is a little idiosyncratic. This may help:

edited May 23 '17 at 12:24

Community

answered Nov 22 '16 at 19:44

Jacob

score 0 · Answer 2 · answered Nov 22 '16 at 17:54

0

Increase Your Virtual Memory: increase the size of the paging file

answered Nov 22 '16 at 17:54

Taufiq Rahman

2 Answers2