how to speed up an R for loop?

Question

I am running the following for loop for the gwr.basic function in the GWmodel package in R. What I need to do is to collect the mean of estimate parameter for any given bandwidth.

the code looks like:

library(GWmodel)
data("DubVoter")
#Dub.voter


LARentMean = list()
for (i in 20:21)
{
gwr.res <- gwr.basic(GenEl2004 ~ DiffAdd + LARent + SC1 + Unempl + LowEduc + Age18_24 + Age25_44 + Age45_64, data = Dub.voter, bw = i,  kernel = "bisquare", adaptive = TRUE, F123.test = TRUE)
a <- mean(gwr.res$SDF$LARent)
LARentMean[i] <- a
}
outcome = unlist(LARentMean)

> outcome
[1] -0.1117668 -0.1099969

However it is terribly slow at returning the result. I need a much wider range such as 20:200. Is there a way to speed the process up? If not, how to have a stepped range let's say 20 to 200 with steps of 5 to reduce the number of operations?

I am a python user new to R. I read on SO that R is well known for being slow at for loops and that there are more efficient alternatives. More clarity on this point would be welcomed.

Your question seems to suggest that **each iteration** of the `for` loop is very expensive, not the `for` loop itself. In that case `for` loop alternatives (`*apply` or `dplyr`) may not help much. Or are you looking for solutions which parallelize the code? — musically_ut, Aug 22 '15 at 14:56
I only have two cores. I don't know if parallel computations would make any improvement — Blue Moon, Aug 22 '15 at 15:07
GWR models take awhile to computer. 200 of them take longer. Your bottleneck isn't in your code design. — John, Aug 22 '15 at 15:22
Can you define _"terribly slow"_? A `microbenchmark` of one iteration (`bw=20`) on my MacBook Pro says it takes ~9s for it to run. I don't know if changing the `bw` value will cause the model to be slower. 2 cores _might_ help if you had more iterations since memory is probably not a factor as `DubVoter` is just 1MB (it will get copied for each parallel task), but for that few it may only get you a 25-33% speedup (it could be more, I know not of your architecture). You can do `seq(20,200,5)` to go by 5's and I'd suggest installing the `beep` pkg to notify you audibly when the task is done. — hrbrmstr, Aug 22 '15 at 15:28
As an aside, an iteration takes ~7s on my Mac Pro. Even with Matt's answer, you may not like the time it takes on your system. A parallel run of the models on an beefy EC2 instance would take almost no time and cost you about $5.00 USD. — hrbrmstr, Aug 22 '15 at 15:36

Matt Bannert · Accepted Answer · 2015-08-22T15:50:28.140

4

I got the same impression like @musically_ut. The for loop and the traditional for-vs.apply debate is unlikely to help you here. Try to go for parallelization if you got more than one core. There are several packages like parallel or snowfall. Which package is ultimately the best and fastest depends on your machine and operating system.

Best does not always equal fastest here. A code that works cross-platform and can be worth more than a bit of extra performance. Also transparency and ease of use can outweigh maximum speed. That being said I like the standard solution a lot and would recommend to use parallel which ships with R and works on Windows, OSX and Linux.

EDIT: here's the fully reproducible example using the OP's example.

library(GWmodel)
data("DubVoter")

library(parallel)

bwlist <- list(bw1 = 20, bw2 = 21)


cl <- makeCluster(detectCores())

# load 'GWmodel' for each node
clusterEvalQ(cl, library(GWmodel))

# export data to each node
clusterExport(cl, varlist = c("bwlist","Dub.voter"))

out <- parLapply(cl, bwlist, function(e){
 try(gwr.basic(GenEl2004 ~ DiffAdd + LARent + SC1 +
 Unempl + LowEduc + Age18_24 + Age25_44 +
 Age45_64, data = Dub.voter,
 bw = e,  kernel = "bisquare",
 adaptive = TRUE, F123.test = TRUE  ))

} )


LArent_l <- lapply(lapply(out,"[[","SDF"),"[[","LARent")
unlist(lapply(LArent_l,"mean"))

# finally, stop the cluster
stopCluster(cl)

edited Aug 22 '15 at 15:50

answered Aug 22 '15 at 15:02

Matt Bannert

27,631
38
141
207

how would you built the for loop there? – Blue Moon Aug 22 '15 at 15:10
I didn't even realize you were looping over bandwidth cause it was so far to the right, but you could use a list like `list(bw1 = 20, bw2 = 21)` and the parLapply that list since the dataset appears to be the unchanged in all iterations. plus I would extract the means outside the loop. – Matt Bannert Aug 22 '15 at 15:22
This sounds great. Would you mind to wrap it all together in an answer? Since this is a common topic (slow R for Loops) it would great to see a full example – Blue Moon Aug 22 '15 at 15:31
hold on, gotta set up an example. Kinda difficult to make something useful since you don't provide an reproducible example. – Matt Bannert Aug 22 '15 at 15:34
the few line of codes i wrote are all the code you need. Data is already in package. You just need to install GWmodel – Blue Moon Aug 22 '15 at 15:36
Just to clarify, there is no "debate" regarding for vs. apply. They are both for loops at their core and neither is appreciably faster if they're well written. If you look at the code for any of the apply functions they are just for loops with some extra logic. If you can vectorize your operations you do much better than for loops, and that's the only thing that new R users typically get caught out on. In the OP case though, parallelization here is indeed what the OP needs, combined with some decent pre-allocation practices. Now, whether you should even be using GWR is a separate issue... ;) – Forrest R. Stevens Aug 22 '15 at 15:42
Awesome. I tried with 40 different bandwidth values and took around 6 minutes. – Blue Moon Aug 22 '15 at 16:31
@ForrestR.Stevens agreed. – Matt Bannert Aug 23 '15 at 18:22

score 4 · Answer 2 · edited Aug 22 '15 at 16:22

4

Besides using parallelization as Matt Bannert suggests, you should preallocate the vector LARentMean. Often, it's not the for loop itself that is slow but the fact that the for seduces you to do slow things like creating growing vectors.

Consider the following example to see the impact of a growing vector as compared to preallocating the memory:

library(microbenchmark)

growing <- function(x) {
  mylist <- list()
  for (i in 1:x) {
    mylist[[i]] <- i
  }
}

allocate <- function(x) {
  mylist <- vector(mode = "list", length = x)
  for (i in 1:x) {
    mylist[[i]] <- i
  }
}

microbenchmark(growing(1000), allocate(1000), times = 1000)
# Unit: microseconds
#          expr      min       lq      mean   median       uq       max neval
# growing(1000) 3055.134 4284.202 4743.4874 4433.024 4655.616 47977.236  1000
# allocate(1000)  867.703  917.738  998.0719  956.441  995.143  2564.192  1000

The growing list is about 5 times slower than the version that preallocates the memory.

edited Aug 22 '15 at 16:22

sunny

3,853
5
32
62

answered Aug 22 '15 at 15:32

CL.

14,577
5
46
73

1

True, but this is a case where the model run is the bottleneck not the `for` / `list` operations, and the OP wld only be growing the numeric list by 180. – hrbrmstr Aug 22 '15 at 15:34
1

Yes. Therefore I wrote that parallelization is what the OP actually needs. However, he also asked for some clarification on what's wrong with `for` loops in R and why they might (not) be slow. As he's using a growing list I tried to elaborate on this aspect a bit. – CL. Aug 22 '15 at 15:36
Microbenchmark is a good idea to finally see which solution helps the most here. I think we see kind of a minimal example here, cause @john red's code did not really take long on my machine. – Matt Bannert Aug 22 '15 at 15:53

how to speed up an R for loop?

2 Answers2

Linked