0

I have a few hundred large model outputs (8 Gb), each saved as a list containing many elements (and sub elements). To do further work on all these outputs it is not possible to load all 100 8 Gb files at once into the environment and I came across lazyLoad as a possible solution here, here, and here. But have not succeeded in making the code work for me yet!

I suspect the problem might be coming from the data being saved using saveRDS() instead of save() (if I try load() a file saved with saveRDS() it gives an error) but I'm not certain.

I have two questions:

1) Is lazyLoad the correct way to deal with large outputs for indexing and loading individual levels of the data structure when called rather than storing everything in memory?

2) What am I doing wrong?

I have made several attempts along the lines of this:

e = local({readRDS("path/to/file.RData"); environment()})
tools:::makeLazyLoadDB(e, "path/to/file")
lazyLoad("path/to/file")

But the final line (lazyLoad("path/to/file") gives NULL as a result.

Since this is a general problem I haven't created a fake data structure as a reproducible example but can do so if required.

I thank you!

QAsena
  • 603
  • 4
  • 9

1 Answers1

0

1) readRDS requires assignment unlike load, so this works:

e = local({df <- readRDS("path/to/file.RData"); environment()})
tools:::makeLazyLoadDB(e, "path/to/file")
lazyLoad("path/to/file")

I should be saving my files according to convention using .rds as an extension when saving data using saveRDS.

2) Probably not the right approach in this case as indexing any element of the "promise" loads the whole file not just the indexed element.

QAsena
  • 603
  • 4
  • 9