R markdown to html via knitr + pandoc: error 137

Question

I have the following issue: I have an .Rmd file which I compile to html via knitr + pandoc using the Rstudio button. In this .Rmd file I pass json data to the js layer following the method described here: http://livefreeordichotomize.com/2017/01/24/custom-javascript-visualizations-in-rmarkdown/

This is because I want to use the data for some custom d3 visuals. This seems to work well for modest amounts of data, however when I try to pass in larger data I have a problem compiling from .Rmd to html; the problem seems to reside with pandoc, as I get:

pandoc document conversion failed with error 137

I attempted to look everywhere online for an explanation of this error message without any luck. Does anyone have any idea what this error means?

Not the same error message but this may be helpful: https://stackoverflow.com/questions/34687030/pandoc-document-conversion-failed-with-error-127 — Stedy, Jul 18 '18 at 22:00
It’s probably an Out of Memory error. See https://stackoverflow.com/questions/1041182/why-does-my-perl-script-exit-with-137. — HappyFace, Jul 16 '19 at 14:24

score 1 · Accepted Answer · answered Jul 30 '18 at 16:31

After researching the error I am still not able to fix it. However, I have a workaround that allows one to inject arbitrary json data into an html report produced with R markdown without going through pandoc at all; the latter does not seem to like the injection of a lot of json using e.g. the method described in

http://livefreeordichotomize.com/2017/01/24/custom-javascript-visualizations-in-rmarkdown/

since this 137 error appears to me to be related to pandoc killing the conversion process to html as it takes to long to terminate.

The workaround is pretty simple: rather than injecting the json data during the knitr compilation step by including it in a chunk with the 'asis' option (as described in the above link), it is better to append the json data at the end of the in the html file that is produced by the compilation through knitr and pandoc. In other words, inject the json data in the output html file of the kitr + pandoc steps.

Assuming that the rmd file to be compiled into html resides in an installed package, and following the suggestions at

Best practice for embedding arbitrary JSON in the DOM?

for embedding json data in the DOM, one can achieve this goal using the xml2 package with a function such as:

create_report <- function(file, 
                          subdir, 
                          package, 
                          parameters, 
                          output_file, 
                          data_arr = NULL){
  #check file exists
  if (system.file(package = package, subdir, file) == ''){
    stop('Cannot find the .rmd file')
  }
  #first generate report
  address <- rmarkdown::render(system.file(package = package,
                                       subdir,
                                       file),
                           output_dir = getwd(),
                           intermediates_dir = getwd(),
                           output_file = output_file,
                           params = parameters,
                           clean = FALSE)
  #then append the data in the json files located in the named list
  #data_arr directly into the 
  #html after compilation if needed:
  if (!is.null(data_arr)){
  report <- xml2::read_html(address)
  report_body <- xml2::xml_find_first(report, "body")
  #adding the data
  for (i in 1:length(data_arr)){
    xml2::xml_add_child(report_body, "script", type = 'application/json', id = 
  names(data_arr[i]), data_arr[[i]])
  }
  #Then add a script that takes the data from the script tags added above and 
  #initializes the variables.
  varnames <- paste0("var ", names(data_arr), " = 
  JSON.parse(document.getElementById('", names(data_arr), "').innerHTML);",
                   collapse = " ")
  xml2::xml_add_child(report_body, "script", type = 'text/javascript', id = 
  'external_json_init', varnames)
  xml2::write_html(report, address)
}
return(address)
}

In the above function, data_arr is supposed to be a named list whose elements are json strings obtained, e.g., by converting R objects using jsonlite::toJSON, and whose names are the variable names to be used to store the data in the javascript layer. In this manner, arbitrary sized json can be injected into an R markdown generated html, to be manipulated by javascript.

R markdown to html via knitr + pandoc: error 137

1 Answers1