R: Printing Individual Iterations From Loops?

Question

I am working with the R programming language.

Normally, I print iterations from loops - this helps me keep track of things. For example:

  my_list = list()

for (i in 1:1000)
    
{ 
    {tryCatch({
        
        frame_i = rnorm(1,1,1)
        my_list[[i]] = frame_i
        ifelse(i%% 10 == 0 ,  cat("\f"), print(i))
        
    }, error = function(e){})
    }}


# desired output 
[1]  1.0000000 -0.9298258

Now, I am trying to do the same thing - but using the "doParallel" libraries in R:

library(doParallel)
registerDoParallel(cl <- makeCluster(3))

test = foreach(i = 1:1000, .combine = "rbind") %dopar% {

 
    {tryCatch({

        frame_i = rnorm(1,1,1)
        my_list[[i]] = frame_i
        ifelse(i%% 10 == 0 ,  cat("\f"), print(i))

    }, error = function(e){})
    }}

This loop seems to have run, but nothing seems to be printing.

I tried consulting the following posts:

Print outputs in foreach loop in R (e.g. sink("Report.txt") does not appear to create a file on my computer)
No standard output received inside foreach loop
R foreach and print behavior
Is it possible to create a text log within a foreach() %dopar% call in R?

But I still can't seem to figure out how to print the (intermittent) results from the "foreach loop" - can someone please help me figure out how to do this?

Thanks!

Note: I am interested in printing out the actual iterations, and not creating a progress bar

The comments on https://stackoverflow.com/questions/58473626/r-doparallel-progress-bar-to-monitor-finished-jobs suggest {pbapply} or {log4r} might be an option. — I_O, Dec 11 '22 at 12:50
Thank you so much for your reply! If you have time, could you please show me how to use this function? Thanks! — stats_noob, Dec 11 '22 at 16:52
I put an example as an answer - admittedly, this still does not inform about the progress *within* any of the distributed workloads. — I_O, Dec 11 '22 at 20:02

score 2 · Answer 1 · answered Dec 11 '22 at 19:26

(Disclaimer: I'm the author)

The Futureverse combined with progressr will give you near-live progress updates also when running in parallel. An example adopted from https://progressr.futureverse.org/#foreach-with-dofuture is:

library(doRNG) ## %dorng%
library(doFuture)
plan(multisession, workers = 3)
registerDoFuture()

library(progressr)
handlers(global = TRUE)

test <- local({
   p <- progressor(1000/10)
   ## %dorng% instead of %dopar% because of rnorm()
   res <- foreach(i = 1:1000, .combine = "rbind") %dorng% {
     frame_i <- rnorm(1,1,1)
     if (i %% 10 == 0) p(sprintf("i = %d", i))
     frame_i
   }
     
   ## Drop %dorng% records (not needed with %dopar%)
   attr(res, "rng") <- NULL  
   attr(res, "doRNG_version") <- NULL
   res
})

score 1 · Answer 2 · answered Dec 11 '22 at 20:00

As suggested here, package {pbapply} offers a progress bar which can also be applied for calls to parallel processing. Example:

library(pbapply)
library(doParallel)
registerDoParallel(cl <- makeCluster(3))

my_fun <- function(repetitions = 10){  
  foreach( i = 1:repetitions, .combine = 'rbind' ) %dopar% {
    Sys.sleep(.5)
  }
}

tmp <- pbreplicate(3, my_fun(10))

## > tmp <- pbreplicate(3, my_fun(10))
##  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100%

However, the progress bar increases only when a packet is finished, it does not increase with progress inside a packet.

R: Printing Individual Iterations From Loops?

2 Answers2

Linked