Stack Overflow
Stack Overflow

Questions tagged [snow]

DO NOT USE FOR SNOW ANIMATION.The R package SNOW (acronym for Simple Network Of Workstations) provides a high-level interface for using a cluster of workstations for parallel computations. Use with the [r] tag.

The package snow (acronym for Simple Network Of Workstations) provides a high-level interface for using a cluster of workstations for parallel computations in R.

snow implements an interface to three different low-level mechanisms for creating a virtual connection between processes:

  • Socket
  • PVM (Parallel Virtual Machine)
  • MPI (Message Passing Interface)

The snowfall package provides a more recent alternative to snow. Functions can be used in sequential or parallel mode.

Resources:

127 questions
37
votes
5 answers

Error calling serialize R function

I am loading the following packages into R: library(foreach) library(doParallel) library(iterators) I "parallelize" code for a long time, but lately I am getting INTERMITTENT stops while code is running. The error is: Error in serialize(data,…
Marcelo Sardelich
  • 952
  • 2
  • 13
  • 28
16
votes
3 answers

Using Rcpp within parallel code via snow to make a cluster

I've written a function in Rcpp and compiled it with inline. Now, I want to run it in parallel on different cores, but I'm getting a strange error. Here's a minimal example, where the function funCPP1 can be compiled and runs well by itself, but…
Vincent
  • 15,809
  • 7
  • 37
  • 39
11
votes
5 answers

error: object '.doSnowGlobals' not found?

I'm trying to parallelize a code on 4 nodes(type = "SOCK"). Here is my code. library(itertools) library(foreach) library(doParallel) library(parallel) workers <- ip address of 4 nodes cl = makePSOCKcluster(workers, master="ip address of…
Rajendra Kumar
  • 323
  • 1
  • 3
  • 12
11
votes
1 answer

Difference between "SOCK", "PVM", "MPI", and "NWS" for the R SNOW package

The makeCluster function for the SNOW package has the different cluster types of "SOCK", "PVM", "MPI", and "NWS" but I'm not very clear on the differences among them, and more specifically which would be best for my program. Currently I have a queue…
jpd527
  • 1,543
  • 1
  • 14
  • 30
9
votes
1 answer

Parallel R on a Windows cluster

I've got a Windows HPC Server running with some nodes in the backend. I would like to run Parallel R using multiple nodes from the backend. I think Parallel R might be using SNOW on Windows, but not too sure about it. My question is, do I need to…
Manolete
  • 3,431
  • 7
  • 54
  • 92
8
votes
2 answers

R foreach: from single-machine to cluster

The following (simplified) script works fine on the master node of a unix cluster (4 virtual cores). library(foreach) library(doParallel) nc = detectCores() cl = makeCluster(nc) registerDoParallel(cl) foreach(i = 1:nrow(data_frame_1), .packages =…
Antoine
  • 1,649
  • 4
  • 23
  • 50
8
votes
2 answers

Parallel processing of big rasters in R (windows)

I'm using the doSNOW package and more specifically the parLapply function to perform reclassification (and subsequently other operations) on a list of big raster datasets (OS: Windows x64). The code looks a little like this minimalistic…
Val
  • 6,585
  • 5
  • 22
  • 52
7
votes
1 answer

register PSOCK cluster with foreach - R 2.14

How would I register a PSOCK cluster (created using package Parallel) with foreach? I'm currently using registerDoSNOW in the doSNOW package to register a snow SOCK cluster.
Suraj
  • 35,905
  • 47
  • 139
  • 250
7
votes
1 answer

"Cannot open the connection" - HPC in R with snow

I'm attempting to run a parallel job in R using snow. I've been able to run extremely similar jobs with no trouble on older versions of R and snow. R package dependencies prevent me from reverting. What happens: My jobs terminate at the parRapply…
Sarah
  • 1,614
  • 1
  • 23
  • 37
5
votes
2 answers

Resetting R random number generator (rlecuyer) for inner loops using Snow/doSNOW

I have an outer foreach/dopar parallel loop containing an inner loop. Every instance of the inner loop should work on the same set of random numbers. The rest, i.e. the remaining parts of the outer body and the parallel instances should work as…
g g
  • 304
  • 2
  • 13
5
votes
1 answer

Print progress to windows cmd within clusterApply or clusterMap

I am calling a python script on multiple cores using the snow package in R. What I want is to print the progress to the console. Using cat(), message() or print() inside my function is not giving any output. This makes it difficult to track the…
Wilmar van Ommeren
  • 7,469
  • 6
  • 34
  • 65
5
votes
1 answer

How to make a object available to node in the snow package for R parallel computing

This is something I find difficult to understand: cl = makeCluster(rep("localhost", 8), "SOCK") # This will not work, error: dat not found in the nodes pmult = function(cl, a, x) { mult = function(s) s*x parLapply(cl, a, mult) } scalars =…
qed
  • 22,298
  • 21
  • 125
  • 196
4
votes
2 answers

R, dplyr and snow: how to parallelize functions which use dplyr

Let's suppose that I want to apply, in a parallel fashion, myfunction to each row of myDataFrame. Suppose that otherDataFrame is a dataframe with two columns: COLUNM1_odf and COLUMN2_odf used for some reasons in myfunction. So I would like to write…
enneppi
  • 1,029
  • 2
  • 15
  • 33
4
votes
2 answers

R Parallel Processing - Node Choice

I am attempting to process a large amount of data in R on Windows using the parallel package on a computer with 8 cores. I have a large data.frame that I need to process row-by-row. For each row, I can estimate how long it will take for that row to…
Michael Barrowman
  • 953
  • 7
  • 14
4
votes
1 answer

makeCluster function in R snow hangs indefinitely

I am using makeCluster function from R package snow from Linux machine to start a SOCK cluster on a remote Linux machine. All seems settled for the two machines to communicate succesfully (I am able to estabilish ssh connections between the two).…
Aaron Iemma
  • 43
  • 1
  • 3
1
2 3
…
8 9