How to recover from timeout error in MODISTools

Question

I am grateful to be using MODISTools to download image matches in time and space. Specifically, MODISTools is looking up leaf area index for a million related observations made by a different satellite. As warned, the NASA server times out regularly.

When that happens I would like r to catch the error, wait briefly (if even needed), then try again. I know the wait does not need to be more than a few seconds because I am running multiple instances, and only one or two of them time out at once. Once the error is detected, processing could resume by retrying the same step, or, with essentially the same effect, by restarting the sequence pasted below. Is there a straightforward way to do this? If it involves invoking server commands from r, please write your answer for an ignoramus!

Here's an example, though without hitting a time-out it won't illustrate much. For simplicity I show fixed location and dates. In the real run they derive from other fields in the my.chunk dataframe.

 look.these.up <- which(is.na(my.chunk$lai))
 for (m.obs in look.these.up) {
      junk <- mt_subset(product = "MOD15A2H", lon = 5, lat = 5,
           start = "2017-01-01", end = "2017-01-08")
      my.chunk$lai[m.obs] <- junk$data[, "data"][5] 
 }

score 0 · Answer 1 · answered Jun 25 '18 at 15:59

Here's what worked for me. I looked up at .How can I view the source code for a function? how to view the code behind the mt_subset command and modified it.

Some of the simplifications, like creating all.dates outside of the for loop and dropping most of the input error checking, were possible because of the limited subset of MODIS data I need.

 for (m.obs in 1:n.obs) {
    modis.date <- all.dates[which(all.dates$calendar_date <= 
       ret.date & all.dates$calendar_date >= (ret.date - 7)), 1][1]

 ## Retrieve the MODIS values for the obs time & place
 m.response <- httr::GET(
    url = "https://modis.ornl.gov/rst/api/v1/MOD15A2H/subset", 
    query = list(latitude = my.chunk$lat[m.obs], 
       longitude = my.chunk$lon[m.obs],   
       startDate = modis.date, endDate = modis.date,
       kmAboveBelow = 0, kmLeftRight = 0),
    httr::write_memory())
    ## If connection fails, try it again up to 10 times.   
    attempts = 0
    while (httr::http_error(m.response)) {
       attempts = attempts + 1
       print(paste0("retry # ", attempts))
       if (attempts >= 10) {
          stop()
          m.response <- NULL
       } 
       Sys.sleep(1) # seconds
       m.response <- httr::GET(
          url = "https://modis.ornl.gov/rst/api/v1/MOD15A2H/subset", 
          query = list(latitude = my.chunk$lat[m.obs], 
             longitude = my.chunk$lon[m.obs],   
             startDate = modis.date, endDate = modis.date,
             kmAboveBelow = 0, kmLeftRight = 0),
          httr::write_memory())
    }   
   modis.vals <- jsonlite::fromJSON(httr::content(m.response,
      "text", encoding = "UTF-8"), 
       simplifyVector = TRUE)$subset[, "data"]  # decodes returned data
   my.chunk$fpar[m.obs] <- modis.vals[[6]] # fPar

   etc.
}

How to recover from timeout error in MODISTools

1 Answers1