Nested function to retrieve data from multiple URLs (with authentication) in R

Question

My code below is designed to retrieve data (and its metadata) with authentication through an API endpoint, and return all metadata into a dataframe. I want to create a nested function to repeat this same process for another API endpoint with the same authentication and append the result from the second endpoint to the first, into a single dataframe (they both have the same data structure and headers). I don't know where in this process should I put link_to_endpoint2, and how to do the nesting, and append the results, etc.

get_data <- function(uid, credentials, root_url) {

  cookie <- credentials$cookie
  token <- credentials$token

  start_time <- Sys.time()
  print (start_time)
  url <- paste0(root_url, 'link_to_endpoint1', uid)
  resp <- httr::GET(url,
                    httr::add_headers(.headers = c(`Content-Type` = "application/json", 
                                                   Cookie = cookie, `X-CSRF-Token` = token)),
                    body = body, 
                    encode = "json")

  httr::warn_for_status(resp)
  resources <- httr::content(resp)
  access_check <- resources$result$error
  assertthat::assert_that(is.null(access_check),
                          msg = 'Access denied')

  resources <- resources$result[[1]]$resources
  res_ids <- purrr::map_chr(resources, 'id')
  res_urls <- purrr::map_chr(resources, 'url')
  res_desc <- purrr::map_chr(resources, 'description')
  res_format <- purrr::map_chr(resources, 'format')
  res_rclass <- purrr::map_chr(resources, 'data_classification_of_file')
  res_rtype <- purrr::map_chr(resources, 'mimetype')
  res_rname <- purrr::map_chr(resources, 'name')
  out <- data.frame(res_ids,
                    res_urls,
                    res_desc,
                    res_format,
                    res_rclass,
                    res_rtype,
                    res_rname,
                    stringsAsFactors = FALSE)
  out$id <- uid
  end_time <- Sys.time()
  print (end_time)
  process_time <- (end_time - start_time)
  print(process_time)
  return(out)
}

Hi Susie, if my solution worked for you please consider accepting it as an answer by clicking on the check mark to the left. This lets me know it worked for you and the community know it worked as well. — CPak, Sep 09 '17 at 01:58

score 0 · Accepted Answer · answered Aug 07 '17 at 21:48

You'll have to test this solution but you can try

Changing these lines to

# Old line
New line

# get_data <- function(uid, credentials, root_url) {
get_data <- function(uid, credentials, root_url, link) {

# url <- paste0(root_url, 'link_to_endpoint1', uid)
url <- paste0(root_url, link, uid)

Then you could call this function with (assuming you have already loaded purrr)

links <- c('link_to_endpoint1', 'link_to_endpoint2')
desired <- map_df(links, ~get_data(uid, credentials, root_url, .x))

Nested function to retrieve data from multiple URLs (with authentication) in R

1 Answers1