How to handle API error in a foreach loop R?

Question

FYI, based on some comments I added more information.

I created the following function that is making a call to an API:

keyword_checker <- function(keyword, domain, loc, lang){

  keyword_to_check <- as.character(keyword)

  api_request <- paste("https://script.google.com/blabalbalba",
                       "?kw=",keyword,
                       "&domain=",domain,
                       "&loc=",loc,
                       "&lang=",lang,sep="")
  api_request <- URLencode(api_request, repeated = TRUE)  
  source <-fromJSON(file = api_request)#Json file into Data Frame
  return(data.frame(do.call("rbind", source$data$result))) ##in order to extract only the "results" data

I am using the R package foreach() with %dopar% and doSNOW to do many API calls (more than 120k calls). Unfortunately, it happens that there are some errors (usually time out connection), so it makes the script stops. In order to avoid this problem I used the .errorhandling = 'pass'. Now, the script doesn't stop but I would like to know if there is a way to make the API call until I get an answer?

Here is my script:

cl <- makeCluster(9)
registerDoSNOW(cl)

final_urls_checker <- foreach(i = 1:length(mes_urls_to_check), .combine=rbind, .errorhandling = 'pass', .packages='rjson') %dopar% {
  test_keyword <- as.character(mes_urls_to_check[i])
  results <- indexed_url(test_keyword)}  ##name of my function

##Stop cluster
stopCluster(cl)

I basically want my script to continue (without stopping the whole process) until I get the answer from the API call

Do I need to incorporate the TryCatch function within the foreach, OR is that better to "upgrade" the function that I created by adding something like "if the API doesn't give the answer, then wait until it gets it?"

I hope this is clearer.

May be by adding `try catch` block inside `foreach` loop? – Rushabh Patel Mar 15 '19 at 12:05 — Rushabh Patel, Mar 15 '19 at 12:05

snair.stack · Answer 1 · 2019-03-18T10:21:29.300

Try using tryCatch inside the foreach function to catch the expected error messages (here failed API call due to time out). Below is a sample code snippet for the given function keyword_checker, based on my understanding.

library(foreach)
 cl <- makeCluster(9)
registerDoSNOW(cl)

final_urls_checker <- foreach(i = 1:length(mes_urls_to_check), .combine=rbind, .errorhandling = 'pass', 
    .packages='rjson') %dopar% {
  test_keyword <- as.character(mes_urls_to_check[i])
  #results <- keyword_checker(test_keyword)}  ##name of my function
  results <- function(test_keyword){
    dmy <- tryCatch(
    {
        keyword_checker(test_keyword)
    },
    error = function(cond){
        message = "Timeout error! Calling again..."
        dmy2 <- keyword_checker(test_keyword)
        return(dmy2) 
    }
    warning = function(cond){
        message("Warning message:")
        message(cond)
        return(NULL)
    }
    finally = {
        message(paste("Succesfully called API ", test_keyword))
    }
        )
    return(dmy)
  }

##Stop cluster
stopCluster(cl)

Here's a link which explains how to write tryCatch. Note, this snippet may not exactly work since I didn't run the code block. But calling the API caller again, when it fails should do the job.

Check this link, for a discussion on similar issue.

thanks for your help. I tried to give more information about my question. hope this will be easier to understand — Franck, Mar 15 '19 at 15:28
thank you for your answer, I'll try it asap. I'll also update my answer because I tried `trycatch` directly in the function I created and this is also working — Franck, Mar 18 '19 at 09:52

score 1 · Answer 2 · answered Mar 18 '19 at 09:55

Here is an updated script including the TryCatch directly in the function.

indexed_url <- function(url){

  url_to_check <- as.character(url)

  api_request <- paste("https://script.google.com/macros/blablabalbalbaexec",
                       "?page=",url_to_check,sep="")
  api_request <- URLencode(api_request, repeated = TRUE)  
  source <- tryCatch({
    fromJSON(file = api_request)#Convertir un Json file en Data Frame  
  }, error = function(e) {
    cat(paste0("Une erreur a eu lieu :",e))
    Sys.sleep(1)
    indexed_url(url)
  })  
  return(data.frame(do.call("rbind", source)))
}

Then running the foreach just the way it was is working perfectly. No more errors, and I have the full analysis.

How to handle API error in a foreach loop R?

2 Answers2