Split columns or vector to pass to Purrr Function

Question

I have a web scrape function that I created that gets data from an API. I pass a df column I have to one of the function arguments in the web scrape function. The issue I'm having is that the URL takes up to 500 numbers in one of the parameters, and my df has 2000 rows.

How would I split the rows by 500 in order to pass the values into the function?

I've created a very basic reprex that shows the workflow of what I am looking to do. I want to pass the split df column to the parse function. I'm guessing I would need to wrap the JSON parse with map_dfr

library(tidyverse)

sample_df <- tibble(id = 1:20,
             col_2 = rnorm(1:20)) 

# parse function
parse_people <- function(ids = c("1", "10"), argument_2 = NULL){
  # Fake Base Url
  base_url <- "https://www.thisisafakeurl.com/api/people?Ids="

  # fix query parameters to collapse Ids to pass to URL
  ids<- stringr::str_c(ids, collapse = ",")

  url <- glue::glue("{base_url}{ids}")

  # Get URL
  resp <- httr::GET(url)

  # Save Response in JSON Format
  out <- httr::content(resp, as = "text", encoding = "UTF-8")

  # Read into JSON format.  
    jsonlite::fromJSON(out, simplifyDataFrame = TRUE, flatten = TRUE)

}


sample_parse <- parse_people(sample_df$id)

I think I probably need to create 2 functions. 1 function that parses the data, and one that uses map_dfr based off of the splits.

Something like:

# Split ID's from DF here.  I want blocks of 500 rows to pass below

# Map Split ID's over parse_people
ids %>% 
map_dfr(parse_people)

Can you please check my solutiion – akrun Mar 12 '20 at 17:29 — akrun, Mar 12 '20 at 17:29

David Ranzolin · Answer 1 · 2020-03-12T17:21:05.717

0

Possible duplicate here.

In the meantime, you can split your 20 row dataframe into 5 dataframes of 4 rows each via:

sample_df <- tibble(id = 1:20,
                    col_2 = rnorm(1:20)) 

split(sample_df, rep(1:5, each = 4))

Then you can pass the resulting list of dataframes to a purrr function.

Edit: If you don't know the total rows in advance, want to split by a given number, but also include all rows, there's another solution in the link:

chunk <- 3
n <- nrow(sample_df)
r  <- rep(1:ceiling(n/chunk),each=chunk)[1:n]
d <- split(sample_df,r)

Here I want chunks of 3, but it will include all rows (the last data frame in the list has 2 rows)

edited Mar 12 '20 at 17:21

answered Mar 12 '20 at 17:08

David Ranzolin

913
1
6
18

What if I have an uneven number of rows to split in my df? I want to split by 500, but include all rows. – Jazzmatazz Mar 12 '20 at 17:16
I amended the answer with another solution. – David Ranzolin Mar 12 '20 at 17:21
I updated my question/example. Basically I'm trying to split the rows from the ids in order to pass to map_dfr. – Jazzmatazz Mar 12 '20 at 17:49

akrun · Accepted Answer · 2020-03-12T18:02:41.643

0

If we need to split the data.frame into a list of data.frame, an option is group_split with gl

library(dplyr)
n <- 3
lst1 <- sample_df %>% 
          group_split(grp = as.integer(gl(n(), n, n())), keep = FALSE) %>%
          map(pull, id)

and pass it to the function as

map(lst1, ~ parse_people(ids = .x))

edited Mar 12 '20 at 18:02

answered Mar 12 '20 at 17:24

akrun

874,273
37
540
662

I updated my question/example. Basically I'm trying to split the rows from the ids in order to pass to map_dfr. – Jazzmatazz Mar 12 '20 at 17:49
How would I pass the values to the map_dfr function as ids? I.E. parse_people(ids =) in the parse function. – Jazzmatazz Mar 12 '20 at 18:00
@Jazzmatazz Try `map(lst1, ~parse_people(ids = .x))` – akrun Mar 12 '20 at 18:02

Split columns or vector to pass to Purrr Function

2 Answers2