create dataframe from list of lists of data.frames

Question

I have a list of lists of data.frames, which I would like to convert to a data.frame. The structure is as follows:

l_of_lists <- list(
  year1 = list(
    one = data.frame(date = c("Jan-10", "Jan-22"), type = c("type 1", "type 2")),
    two = data.frame(date = c("Feb-1", "Feb-28"), type = c("type 2", "type 3")),
    three = data.frame(date = c("Mar-10", "Mar-15"), type = c("type 1", "type 4"))
    ),
  year2 = list( # dates is used here on purpose, as the names don't perfectly match
    one = data.frame(dates = c("Jan-22"), type = c("type 2"), another_col = c("entry 2")),
    two = data.frame(date = c("Feb-10", "Feb-18"), type = c("type 2", "type 3"), another_col = c("entry 2", "entry 3")),
    three = data.frame(date = c("Mar-10", "Mar-15"), type = c("type 1", "type 4"), another_col = c("entry 4", "entry 5"))
    ),
  year3 = list( # this deliberately only contains two data frames
    one = data.frame(date = c("Jan-10", "Jan-12"), type = c("type 1", "type 2")),
    two = data.frame(date = c("Feb-8", "Jan-28"), type = c("type 2", "type 3"))
  ))

The data frame has two particularities I tried to mimic above:

the column names differ by 1-2 characters (e.g. date vs. dates)
some columns are only present in some data frames (e.g. another_col)

I now would like to convert this to a data frame (I tried different calls to rbind and also do.call, as described e.g. here unsuccessfully) and would like to - match on column names tolerantly (if the column names are similar to 1-2 characters, I want them to be matched and - fill non-existent columns with NA in other columns.

I want a data frame similar to the following

year  level       date        type  another_col                    
   1    one    "Jan-10"    "type 1"           NA
   1    one    "Jan-22"    "type 2"           NA
   1    two     "Feb-1"    "type 2"           NA
   1    two    "Feb-28"    "type 3"           NA
   1  three    "Mar-10"    "type 1"           NA
   1  three    "Mar-15"    "type 4"           NA
   2    one    "Jan-22"    "type 2"     "entry 2"
   2    two     "Feb-1"    "type 2"     "entry 2"
   2    two    "Feb-28"    "type 3"     "entry 3"
   2  three    "Mar-10"    "type 1"     "entry 4"
   2  three    "Mar-15"    "type 4"     "entry 5"
   3    one    "Jan-10"    "type 1"           NA
   3    one    "Jan-12"    "type 2"           NA
   3    two     "Feb-8"    "type 2"           NA
   3    two    "Feb-28"    "type 3"           NA

Can someone point out if rbind is the correct path here - and what I am missing?

MDEWITT · Accepted Answer · 2019-10-28T16:57:38.290

You could do something like the following using purrr and dplyr:

l_of_lists <- list(
  year1 = list(
    one = data.frame(date = c("Jan-10", "Jan-22"), type = c("type 1", "type 2")),
    two = data.frame(date = c("Feb-1", "Feb-28"), type = c("type 2", "type 3")),
    three = data.frame(date = c("Mar-10", "Mar-15"), type = c("type 1", "type 4"))
  ),
  year2 = list( # dates is used here on purpose, as the names don't perfectly match
    one = data.frame(dates = c("Jan-22"), type = c("type 2"), another_col = c("entry 2")),
    two = data.frame(date = c("Feb-10", "Feb-18"), type = c("type 2", "type 3"), another_col = c("entry 2", "entry 3")),
    three = data.frame(date = c("Mar-10", "Mar-15"), type = c("type 1", "type 4"), another_col = c("entry 4", "entry 5"))
  ),
  year3 = list( # this deliberately only contains two data frames
    one = data.frame(date = c("Jan-10", "Jan-12"), type = c("type 1", "type 2")),
    two = data.frame(date = c("Feb-8", "Jan-28"), type = c("type 2", "type 3"))
  ))

# add libraries
library(dplyr)
library(purrr)

# Map bind_rows to each list within the list
l_of_lists %>% 
  map_dfr(~bind_rows(.x, .id = "level"), .id = "year")

This will yield:

     year level   date   type  dates another_col
1  year1   one Jan-10 type 1   <NA>        <NA>
2  year1   one Jan-22 type 2   <NA>        <NA>
3  year1   two  Feb-1 type 2   <NA>        <NA>
4  year1   two Feb-28 type 3   <NA>        <NA>
5  year1 three Mar-10 type 1   <NA>        <NA>
6  year1 three Mar-15 type 4   <NA>        <NA>
7  year2   one   <NA> type 2 Jan-22     entry 2
8  year2   two Feb-10 type 2   <NA>     entry 2
9  year2   two Feb-18 type 3   <NA>     entry 3
10 year2 three Mar-10 type 1   <NA>     entry 4
11 year2 three Mar-15 type 4   <NA>     entry 5
12 year3   one Jan-10 type 1   <NA>        <NA>
13 year3   one Jan-12 type 2   <NA>        <NA>
14 year3   two  Feb-8 type 2   <NA>        <NA>
15 year3   two Jan-28 type 3   <NA>        <NA>

Then of course you can do some regex parsing to keep only the numeric year:

l_of_lists %>% 
  map_dfr(~bind_rows(.x, .id = "level"), .id = "year") %>% 
  mutate(year = substring(year, regexpr("\\d", year)))

If you know that date and dates are the same, you can always use mutate to changed then to those values that are not missing (i.e.mutate(date = ifelse(!is.na(date), date, dates)))

create dataframe from list of lists of data.frames

1 Answers1