1

I'm using R to make an API call to a weather data provider to download some weather forecasts. I'm using a free key that allows me to make no more than 10 calls per minute. I've tried using Sys.sleep() to ensure I don't go over the threshold but the API resource monitor tells me that I've exceeded the number of calls.

For example, if I'm making 6 calls, a time interval of 10 seconds between the calls ought to be sufficient (not taking into account the time R would need). 

dat <- list()
for(i in 1:6){
  dat[[i]] <- getWeatherData(web_url, api_key, history_date, data_format)
  Sys.sleep(10)
  web_url <- gsub(i-1, i, url)
}

The getWeatherData function does the following:

  1. makes the API call (only one API call is made each time the function is invoked. Uses httr::GET() to get the data),
  2. parses the XML output to get desired variables (regulat expressions),
  3. performs some clean-up (for missing/garbage values),
  4. converts strings to R date-time objects (POSIXct), and
  5. rounds values to the nearest hour (lubridate::round_date()).

Function inputs:

  1. web_url is a custom url,
  2. api_key is my personal key,
  3. history_date is a string (formatted as "%d/%m/%Y %H:%M:%S"), and
  4. data_format specifies if I want an .XML or .json file as output.

I can not share the url/key for obvious reasons. As soon as I run this, I get a notification from the data provider that I've exceeded the allowable calls per minute (10). I don't get a notification every time - not sure why that is either.

Any help is appreciated!

Gautam
  • 2,597
  • 1
  • 28
  • 51
  • What does the `getWeatherData` function do? Are you sure it only makes one API call on your behalf? – MrFlick Dec 12 '17 at 20:05
  • Why don't you first try with `Sys.sleep(70)`? Just less than once a minute. I'm wondering if there is any buffering is causing delay and your calls are exceeding threshold. – MKR Dec 12 '17 at 20:15
  • @MrFlick Yes, the function only makes the API call - uses `httr` library to get the .XML output and clean it up. I know it makes only one call because I call `httr::GET()` only once. @MKR `Sys.sleep(70)` works. I can set a high value for the sleep-time but I have to make a bunch of calls to get all the data I want (1 month per site, 30 sites) and waiting that long would take quite some time. I'm trying to get another free key so I can cycle the keys instead but figuring this out would be useful as well. – Gautam Dec 12 '17 at 20:19
  • Can you time your for loop to check how long 6 calls are taking? Alternatively, could you try making calls in batches of 10 and waiting a minute between each of the calls? – Walker in the City Dec 12 '17 at 22:40

1 Answers1

0

This solution should be helpful for you if Sys.sleep doesn't do the trick.

Basically, this replaces the use of Sys.sleep with while logic.

dat <- list()
delay_seconds<-10

for(i in 1:6){
  dat[[i]] <- getWeatherData(web_url, api_key, history_date, data_format)
  date_time<-Sys.time()
  while((as.numeric(Sys.time()) - as.numeric(date_time))<delay_seconds){}
  web_url <- gsub(i-1, i, url)
}

Here, we are:

  1. defining a number of seconds to wait ( delay_seconds<-10 )
  2. defining a start time for comparison ( date_time<-Sys.time() )
  3. using a while loop that checks the present time in comparison to our comparison time and seeing if this is less than our chosen delay interval ( (as.numeric(Sys.time()) - as.numeric(date_time)<delay_seconds )
  4. doing nothing until the wait time is over( {} )

Not knowing if you need/want to, but in the case that you're hoping to get your data out of the lists and into a longer combined form, I recommend the dplyr function bind_rows().

dat2<-bind_rows(dat)

Thanks to an answer by rbtj to this question: How to make execution pause, sleep, wait for X seconds in R?

Pake
  • 968
  • 9
  • 24