1

I want to log certain events to a .txt file in my R script. Part of the script is parallelized. Is it possible to log events with cat() or similar inside a parallelized function?

This does not work.


chunks <- list(1:3, 4:6)

foreach(i = 1:cores) %:%
  foreach(x = chunks[[i]]) %dopar% {
    cat("Working on chunk, ", i, "number ", x, "...\n\n")
  }

This works, but it is system specific (not ideal)

foreach(i = 1:cores, .packages = "glue") %:%
  foreach(x = chunks[[i]]) %dopar% {
    system(glue("echo 'Working on chunk {i}, number {x}' >> output.txt"))
  }

Example output of the second (working) code block:

Working on chunk 1, number 1
Working on chunk 1, number 2
Working on chunk 1, number 3
Working on chunk 2, number 4
Working on chunk 2, number 6
Working on chunk 2, number 5
Brigadeiro
  • 2,649
  • 13
  • 30
  • 1
    Does this answer your question? [No standard output received inside foreach loop](https://stackoverflow.com/questions/45070987/no-standard-output-received-inside-foreach-loop) – F. Privé Apr 17 '20 at 20:06
  • @F.Privé this looks really great and is very closed, but it has unintended consequences. For example, when loading packages for each thread (.packages = c("package", "names") I get all the package loading text for each thread which is really excessive! Any tips? – Brigadeiro Apr 17 '20 at 23:17

1 Answers1

2

You can use doSNOW package in R. You can specify the file where you want the output logs too. Hope this works for you.

library(doSNOW)

cl <- makeCluster(2, outfile = "abc.out")
registerDoSNOW(cl)

chunks <- list(1:3, 4:6)

foreach(i = 1:2) %:%
  foreach(x = chunks[[i]]) %dopar% {
    cat("Working on chunk, ", i, "number ", x, "...\n\n")
    print("ongoing")
    # your statements
  }

stopCluster(cl)
Akshit
  • 424
  • 4
  • 15