What is the most efficient way to cast a list as a data frame?

Question

Very often I want to convert a list wherein each index has identical element types to a data frame. For example, I may have a list:

> my.list
[[1]]
[[1]]$global_stdev_ppb
[1] 24267673

[[1]]$range
[1] 0.03114799

[[1]]$tok
[1] "hello"

[[1]]$global_freq_ppb
[1] 211592.6


[[2]]
[[2]]$global_stdev_ppb
[1] 11561448

[[2]]$range
[1] 0.08870838

[[2]]$tok
[1] "world"

[[2]]$global_freq_ppb
[1] 1002043

I want to convert this list to a data frame where each index element is a column. The natural (to me) thing to go is to is use do.call:

> my.matrix<-do.call("rbind", my.list)
> my.matrix
     global_stdev_ppb range      tok     global_freq_ppb
[1,] 24267673         0.03114799 "hello" 211592.6       
[2,] 11561448         0.08870838 "world" 1002043

Straightforward enough, but when I attempt to cast this matrix as a data frame, the columns remain list elements, rather than vectors:

> my.df<-as.data.frame(my.matrix, stringsAsFactors=FALSE)
> my.df[,1]
[[1]]
[1] 24267673

[[2]]
[1] 11561448

Currently, to get the data frame cast properly I am iterating over each column using unlist and as.vector, then recasting the data frame as such:

new.list<-lapply(1:ncol(my.matrix), function(x) as.vector(unlist(my.matrix[,x])))
my.df<-as.data.frame(do.call(cbind, new.list), stringsAsFactors=FALSE)

This, however, seem very inefficient. Is there are better way to do this?

As of 2017 you should use `your_list %>% reduce(bind_rows)` from `purrr` — Zafar, Mar 06 '17 at 21:32

score 50 · Accepted Answer · answered Dec 22 '10 at 18:47

I think you want:

> do.call(rbind, lapply(my.list, data.frame, stringsAsFactors=FALSE))
  global_stdev_ppb      range   tok global_freq_ppb
1         24267673 0.03114799 hello        211592.6
2         11561448 0.08870838 world       1002043.0
> str(do.call(rbind, lapply(my.list, data.frame, stringsAsFactors=FALSE)))
'data.frame':   2 obs. of  4 variables:
 $ global_stdev_ppb: num  24267673 11561448
 $ range           : num  0.0311 0.0887
 $ tok             : chr  "hello" "world"
 $ global_freq_ppb : num  211593 1002043

`plyr::rbind.fill` is often a little faster than `rbind.fill`, and the whole operation is equivalent to `plyr::ldply(my.list, data.frame)` — hadley, Dec 23 '10 at 00:54

Gavin Simpson · Answer 2 · 2010-12-23T15:16:02.587

Another option is:

data.frame(t(sapply(mylist, `[`)))

but this simple manipulation results in a data frame of lists:

> str(data.frame(t(sapply(mylist, `[`))))
'data.frame':   2 obs. of  3 variables:
 $ a:List of 2
  ..$ : num 1
  ..$ : num 2
 $ b:List of 2
  ..$ : num 2
  ..$ : num 3
 $ c:List of 2
  ..$ : chr "a"
  ..$ : chr "b"

An alternative to this, along the same lines but now the result same as the other solutions, is:

data.frame(lapply(data.frame(t(sapply(mylist, `[`))), unlist))

[Edit: included timings of @Martin Morgan's two solutions, which have the edge over the other solution that return a data frame of vectors.] Some representative timings on a very simple problem:

mylist <- list(list(a = 1, b = 2, c = "a"), list(a = 2, b = 3, c = "b"))

> ## @Joshua Ulrich's solution:
> system.time(replicate(1000, do.call(rbind, lapply(mylist, data.frame,
+                                     stringsAsFactors=FALSE))))
   user  system elapsed 
  1.740   0.001   1.750

> ## @JD Long's solution:
> system.time(replicate(1000, do.call(rbind, lapply(mylist, data.frame))))
   user  system elapsed 
  2.308   0.002   2.339

> ## my sapply solution No.1:
> system.time(replicate(1000, data.frame(t(sapply(mylist, `[`)))))
   user  system elapsed 
  0.296   0.000   0.301

> ## my sapply solution No.2:
> system.time(replicate(1000, data.frame(lapply(data.frame(t(sapply(mylist, `[`))), 
+                                               unlist))))
   user  system elapsed 
  1.067   0.001   1.091

> ## @Martin Morgan's Map() sapply() solution:
> f = function(x) function(i) sapply(x, `[[`, i)
> system.time(replicate(1000, as.data.frame(Map(f(mylist), names(mylist[[1]])))))
   user  system elapsed 
  0.775   0.000   0.778

> ## @Martin Morgan's Map() lapply() unlist() solution:
> f = function(x) function(i) unlist(lapply(x, `[[`, i), use.names=FALSE)
> system.time(replicate(1000, as.data.frame(Map(f(mylist), names(mylist[[1]])))))
   user  system elapsed 
  0.653   0.000   0.658

Hrm.. the `replicate()` usage in this answer is a bit weird. You're testing how efficient it is to convert a small list to a data frame lots of times. That seems like it would only rarely be useful. Wouldn't it make more sense to test the efficiency of the conversion of a *large* list of lists? — naught101, Jul 08 '14 at 02:22
@naught101 possibly; you have the code, try it out ;-) (Report back findings --- you can edit them into my Answer if you like) — Gavin Simpson, Jul 12 '14 at 16:34
@ naught101 I have a [means](http://stackoverflow.com/questions/26534438/intradataframe-analysis-creating-a-derivative-data-frame-from-another-data-fram/26535386) of creating a such a list given someone has a large data frame to crunch the numbers on. — jxramos, Oct 23 '14 at 19:59

score 18 · Answer 3 · answered Dec 22 '10 at 18:48

I can't tell you this is the "most efficient" in terms of memory or speed, but it's pretty efficient in terms of coding:

my.df <- do.call("rbind", lapply(my.list, data.frame))

the lapply() step with data.frame() turns each list item into a single row data frame which then acts nice with rbind()

score 16 · Answer 4 · answered Sep 20 '13 at 07:02

Although this question has long since been answered, it's worth pointing out the data.table package has rbindlist which accomplishes this task very quickly:

library(microbenchmark)
library(data.table)
l <- replicate(1E4, list(a=runif(1), b=runif(1), c=runif(1)), simplify=FALSE)

microbenchmark( times=5,
  R=as.data.frame(Map(f(l), names(l[[1]]))),
  dt=data.frame(rbindlist(l))
)

gives me

Unit: milliseconds
 expr       min        lq    median        uq       max neval
    R 31.060119 31.403943 32.278537 32.370004 33.932700     5
   dt  2.271059  2.273157  2.600976  2.635001  2.729421     5

score 13 · Answer 5 · edited Dec 23 '10 at 10:41

13

This

f = function(x) function(i) sapply(x, `[[`, i)

is a function that returns a function that extracts the i'th element of x. So

Map(f(mylist), names(mylist[[1]]))

gets a named (thanks Map!) list of vectors that can be made into a data frame

as.data.frame(Map(f(mylist), names(mylist[[1]])))

For speed it's usually faster to use unlist(lapply(...), use.names=FALSE) as

f = function(x) function(i) unlist(lapply(x, `[[`, i), use.names=FALSE)

A more general variant is

f = function(X, FUN) function(...) sapply(X, FUN, ...)

When do the list-of-lists structures come up? Maybe there's an earlier step where an iteration could be replaced by something more vectorized?

edited Dec 23 '10 at 10:41

Gavin Simpson

170,508
25
396
453

answered Dec 22 '10 at 22:09

Martin Morgan

45,935
7
84
112

2

+1 for the illustration of `Map`. I need to incorporate `Map`, `Reduce` et. al into my every-day routines... – Joshua Ulrich Dec 22 '10 at 22:45
How does one use these things? The `as.data.frame(Map(f(mylist), names(mylist)))` version doesn't work for me for the sorts of data @DrewConway and I used as the lists don't have names; I get this returned instead `data frame with 0 columns and 0 rows`. Even with names, I can't get this to work for `mylist` in my answer. I'm genuinely curious as I haven't used `Map` et al at all, so am interested in how they work, what they do, when best to deploy etc. – Gavin Simpson Dec 22 '10 at 23:23
Oops, should have been names(mylist[[1]]), i.e., get the names of the sub-elements from the first element. – Martin Morgan Dec 22 '10 at 23:32
And the fastest solutions thus far (added some timings to my answer for comparison). – Gavin Simpson Dec 23 '10 at 10:49

Yi Li · Answer 6 · 2017-04-07T05:21:02.620

3

The dplyr package's bind_rows is efficient.

one <- mtcars[1:4, ]
two <- mtcars[11:14, ]
system.time(dplyr::bind_rows(one, two))
   user  system elapsed 
  0.001   0.000   0.001

edited Apr 07 '17 at 05:21

answered Apr 07 '17 at 03:58

Yi Li

49
4

score 0 · Answer 7 · answered Jul 14 '18 at 12:07

Not sure where they rank as far as efficiency, but depending on the structure of your lists there are some tidyverse options. A bonus is that they work nicely with unequal length lists:

l <- list(a = list(var.1 = 1, var.2 = 2, var.3 = 3)
        , b = list(var.1 = 4, var.2 = 5)
        , c = list(var.1 = 7, var.3 = 9)
        , d = list(var.1 = 10, var.2 = 11, var.3 = NA))

df <- dplyr::bind_rows(l)
df <- purrr::map_df(l, dplyr::bind_rows)
df <- purrr::map_df(l, ~.x)

# all create the same data frame:
# A tibble: 4 x 3
  var.1 var.2 var.3
  <dbl> <dbl> <dbl>
1     1     2     3
2     4     5    NA
3     7    NA     9
4    10    11    NA

And you can also mix vectors and data frames:

library(dplyr)
bind_rows(
  list(a = 1, b = 2),
  data_frame(a = 3:4, b = 5:6),
  c(a = 7)
)

# A tibble: 4 x 2
      a     b
  <dbl> <dbl>
1     1     2
2     3     5
3     4     6
4     7    NA

What is the most efficient way to cast a list as a data frame?

7 Answers7

Linked

Related