R parallel system call on files

Question

I have to convert a large number of RAW images and am using the program DCRAW to do that. Since this program is only using one core I want to parallelize this in R. To call this function I use:

system("dcraw.exe -4 -T image.NEF")

This results in outputting a file called image.tiff in the same folder as the NEF file, which is totally fine. Now I tried multiple R packages to parallelize this but I only get nonsensical returns (probably caused by me). I want to run a large list (1000+ files) through this system call in r , obtained by list.files()

I could only find info on parallel programming for variables within R but not for system calls. Anybody got any ideas? Thanks!

score 2 · Accepted Answer · answered Dec 09 '15 at 15:43

2

It doesnt' matter if you use variables or system. Assuming you're not on Windows (which doesn't support parallel), on any decent system you can run

parallel::mclapply(Sys.glob("*.NEF"),
  function(fn) system(paste("dcraw.exe -4 -T", shQuote(fn))),
  mc.cores=8, mc.preschedule=F)

It will run 8 jobs in parallel. But then you may as well not use R and use instead

ls *.NEF | parallel -u -j8 'dcraw.exe -4 -T {}'

instead (using GNU parallel).

answered Dec 09 '15 at 15:43

Simon Urbanek

13,842
45
45

Hello Simon, your assumption was wrong but that is because I was wrong. It is indeed on Windows and I was not aware that parallel was not possible on windows. Thank you for your answer though. – jonas van duijvenbode Dec 09 '15 at 15:53
For Windows there seems to be `wxargs` which allows parallel execution - I didn't test it, thought, since I don't run Windows - see http://www.pirosa.co.uk/demo/wxargs/wxargs.html – Simon Urbanek Dec 11 '15 at 02:08

score 1 · Answer 2 · edited May 23 '17 at 12:23

1

On Windows I use a modification of this solution (the top voted one) to run many commands with no more than, say, 4 or 8 simultaneously:

Parallel execution of shell processes

It's not an R solution, but I like it.

edited May 23 '17 at 12:23

Community

1
1

answered Dec 09 '15 at 16:39

Andrey Shabalin

4,389
1
19
18

I am unfortunately at a total loss in the windows command line so I'm just going to take the time to switch to Linux, which should make life easier! Thanks though. – jonas van duijvenbode Dec 10 '15 at 08:55

R parallel system call on files

2 Answers2