How to make R foreach threads write to same log file

Question

I have a number of lengthy jobs that I want to parallelize with foreach-dopar so that each thread works on a job independent of others. I want to track the status of each thread (some may fail while others succeed) by writing to a log file using sink. The following apparently doesn't work; the log file has only one entry.

library(foreach)
library(doParallel)
library(doSNOW)

cl = makeCluster(2, type="SOCK")
registerDoSNOW(cl)
dl = file("runlog.Rout", open="wt")
sink(dl, type="output",  append=TRUE)
sink(dl, type="message", append=TRUE)
dump <- foreach(i=1:5, 
            .errorhandling = "stop",
            .verbose=TRUE) %dopar% 
{
    beg.time = Sys.time()
    cat(as.character(beg.time), " I am running....\n", file="mylog.txt")
    # do something here.....
    end.time = Sys.time()
    del.tm = difftime(end.time, beg.time, units="mins")  
    cat("....saving output to file......\n\n", file="mylog.txt")
    save(del.tm, file = paste("I:/Rhome/H", i, ".RData", sep=""))
    return(i)
}
stopCluster(cl)
sink(type="output")
sink(type="message")

The log file has just one line :

....saving output to file......

What went wrong ?

score 3 · Accepted Answer · edited May 23 '17 at 12:34

3

Although I don't really trust having multiple processes write to the same file, you may have success by using the append=TRUE option:

cat("...\n", file="mylog.txt", append=TRUE)

Without setting this option, cat will overwrite the previous contents of "mylog.txt" each time it's called.

For other approaches, see my answer here.

edited May 23 '17 at 12:34

Community

1
1

answered Sep 20 '16 at 21:32

Steve Weston

19,197
4
59
75

That's exactly what's missing! – horaceT Sep 20 '16 at 22:29
Could please elaborate on the issue of multiple processes writing to the same file? Do you think this could lead to unexpected outputs in the file or even a crash of the run? – Janina Nov 20 '20 at 10:07

score 2 · Answer 2 · answered May 18 '18 at 15:10

You can also call makeCluster with the argument outfile. From the documentatino, outfile is

Where to direct the stdout and stderr connection output from the workers. "" indicates no redirection (which may only be useful for workers on the local machine). Defaults to ‘/dev/null’ (‘nul:’ on Windows). The other possibility is a file path on the worker's host. Files will be opened in append mode, as all workers log to the same file.

How to make R foreach threads write to same log file

2 Answers2