rmarkdown::render() in a loop - cannot allocate vector of size

Question

I have a bunch of directories with Rmd files to be compiled into HTML. Each call to rmarkdown::render includes a large list of params. Everything works fine when I render the Rmd files individually.

However, when I execute a function that loops over the directories and builds the HTML files it crashes after it has completed nine (it is not a specific Rmd causing the problem, it crashes on the 9th regardless of the order).

The relevant part of the error traceback is:

Error: cannot allocate vector of size 38.4 Gb

8. knitr::knit_meta_add(old_knit_meta, attr(old_knit_meta, "knit_meta_id")) 

7. rmarkdown::render(input = RMDfile, output_file = RMDfileout, 
        output_format = output, output_dir = dir, param = params, 
        quiet = quiet)

The problem appears to be caused by this line in rmarkdown::render:

 on.exit({
    knit_meta_reset()
    if (length(old_knit_meta)) {
      knitr::knit_meta_add(old_knit_meta, attr(old_knit_meta, 
                                               "knit_meta_id"))
    }
  }, add = TRUE)

It appears to me that the params from the previous calls are being saved in the meta-data, and is getting too large after rmarkdown::render is called 9 times in the function.

rmarkdown::knitr has a knit_meta argument, but the help documentation says "(For expert use) Meta data generated by knitr" and I'm no knitr expert.

The Rmd files are completely independent from each other. Is there a way to reset the knitr meta-data for each call to rmarkdown::render?

I'm hoping someone can offer a quick fix or workaround for this. If not, I'll develop an example to reproduce the problem.

Can you share the code which is used to create the loop over the files? — Michael Harper, Apr 17 '18 at 22:14

score 15 · Accepted Answer · answered Apr 18 '18 at 20:16

15

After digging around in rmarkdown::render code for a while I've found a solution.

Adding knitr::knit_meta(class=NULL, clean = TRUE) before rmarkdown::render(input=file, etc) seems to do the trick..

answered Apr 18 '18 at 20:16

Adrian

684
3
20

1

Beautiful. I added some lines to remove any unnecessary data between loop iterations; that didn't work. I would have to clear the workspace every 8 reports and try again. This solution worked for me (all 20 reports can be rendered at once). THANK YOU! – Nova Oct 01 '18 at 15:41

rmarkdown::render() in a loop - cannot allocate vector of size

1 Answers1

Linked