Jsonlite and For loop issue

Question

I am downloading data from google trends. Sometimes the connection drops and I have to restart the loop from the beginning (1-726). Suppose error is on 721th attempt still, I have to restart it from 1 again.

library(jsonlite)
baseurl <- "https://www.googleapis.com/trends/v1beta/graph?&terms="
for(i in 1:726){
mydata <- fromJSON(paste0(baseurl, file.list$query[i]), flatten=TRUE)
message("Retrieving query ", i)
pages[[i]] <- mydata$lines
}

The error looklikes this

Retrieving query 1
Retrieving query 2
Retrieving query 3
Error in open.connection(con, "rb") : HTTP error 503.

Then I have to restart the whole process repeatedly to get the desired data. Can there be a method that it retries itself without losing the progress?

I think the free google APIs may have some limitation in the number of requests per seconds... try adding `Sys.sleep(0.5)` inside the loop in order to perform 2 requests per second max... — digEmAll, May 22 '17 at 07:25
Or you can wrap the reading in a `try` clause and store the success in a vector. Then you would know which requests were succesful — ekstroem, May 22 '17 at 07:31

score 2 · Answer 1 · edited May 23 '17 at 11:47

This answer on SO describes useful snippet to retry iteration in case of error/exception. Your code (it's impossible to reproduce BTW, as you didn't assign anything to file.list) with such a snippet would look like this:

library(jsonlite)
baseurl <- "https://www.googleapis.com/trends/v1beta/graph?&terms="
for(i in 1:726){
  mydata<-NULL
  attempt <- 1
  while( is.null(mydata) && attempt <= 3 ) {
    attempt <- attempt + 1
    try(
      mydata <- fromJSON(paste0(baseurl, file.list$query[i]), flatten=TRUE)
    )
  } 
    message("Retrieving query ", i)
  pages[[i]] <- mydata$lines
}

Feel free to change number of attempts in while condition. Also, the reason can be the limit for number of requests per second - as digEmAll suggested, try adding Sys.sleep().

Jsonlite and For loop issue

1 Answers1