5

I have an R For loop that downloads data from a server and adds result to table however, I sometimes get an error stopping the loop. If I tell it to redo the last download and continue, it works for another while before the next error. The error isn't with code or data, but is random; sometimes it runs for 2.5 hours, other times it stops after 45 minutes downloading the same data.
Is there a way I could get my loop to take a step back if there is an error and retry? eg. in

for (i in 1:1000){
    table[i,] <- downloadfnc("URL", file = i)
}

lets say I get an error while it was downloading i=500, all I do to fix is:

for (i in 500:1000){
    i <- i + 499     #since i starts at 1, 499+1=500
    table[i,] <- downloadfnc("URL",file = i)
}

then it downloads file"500" even though it got an error last time. is there a way I could automate it, so that if there is an error, it takes a step back (i-1) and retry it (perhaps with a few seconds delay)?

(been using R for only several weeks, so basic talk please)

Ron Crash
  • 53
  • 1
  • 5
  • Seems like a memory issue. – N8TRO Aug 13 '15 at 23:03
  • what exactly does downloadfnc return? – rawr Aug 14 '15 at 00:15
  • downloadfnc (i made the name, it's actually 'DODSGrab') returns a Formal class SpatioTemporalDataFrame (coordinates with some data). but I simplified my code for the question. -- N8TRO, I don't think it's a memory issue since I only download several kb of data, it takes several seconds to receive the data, and I overwrite previous download. I think it is something with the server. – Ron Crash Aug 14 '15 at 05:29

1 Answers1

13

You could throw a try-catch combo.

for (i in 1:1000){
    while(TRUE){
       df <- try(downloadfnc("URL", file = i), silent=TRUE)
       if(!is(df, 'try-error')) break
    }
    table[i,] <- df
}

This will continue within the while loop until the file is successfully downloaded, and only move on when it is successfully downloaded.

philchalmers
  • 727
  • 7
  • 19
  • I will try this over night and see how it goes. Will update. – Ron Crash Aug 14 '15 at 05:46
  • Is there a chance this loop is making too many requests and overloading the server? (I'm getting some weird symptoms).... so it seems to work, it's been running for about 8.5 hours. The first 3 were great (did 60% of task). but then after that, the rate went down dramatically (6% in next 5 hours). i checked speed of server around 6 hours in, and it went down for me and for on to Dallas (test) the same amount (used webpagetest.org). 8 hours in, my speed dropped to 5 minutes/load, and Dallas was 300ms. now my speed fluctuates between several seconds and 5min. – Ron Crash Aug 14 '15 at 16:11
  • Not entirely sure, but those symptoms don't really sound like R related issues. Servers should be able to get pinged quite a bit, so 1000 jobs shouldn't be taxing at all. Are you witnessing any RAM related problems; how large are these downloads? It's possible your local comp is getting bogged down and you may want to write the files locally rather than store them all in RAM, but if not there's something up with the server. – philchalmers Aug 14 '15 at 16:17
  • the files are no more than a few kbs (10 by 4 DataFrame). it seems to be ok in the morning (back to 3s/ download). I was thinking, maybe if the While loop was looping during connection before the download failed or passed the servers might be perceiving it as many users (I don't know if this is even plausible). but last several hours were good. – Ron Crash Aug 14 '15 at 18:07