Mass rbind.fill for many data frames

Question

I am attempting to row bind many data frames together into a single massive data frame. The data frames are named sequentially with the first named df1, the second named df2, the third named df3, etc. Currently, I have bound these data frames together by explicitly typing the names of the data frames; however, for a very large number of data frames (roughly 10,000 total data frames are expected) this is suboptimal.

Here is a working example:

# Load required packages
library(plyr)

# Generate 100 example data frames
for(i in 1:100){
   assign(paste0('df', i), data.frame(x = rep(1:100),
                                      y = seq(from = 1,
                                              to = 1000,
                                              length = 100)))
  }
}

# Create a master merged data frame
 df <- rbind.fill(df1, df2, df3, df4, df5, df6, df7, df8, df9, df10,
             df11, df12, df13, df14, df15, df16, df17, df18, df19, df20,
             df21, df22, df23, df24, df25, df26, df27, df28, df29, df30,
             df31, df32, df33, df34, df35, df36, df37, df38, df39, df40,
             df41, df42, df43, df44, df45, df46, df47, df48, df49, df50,
             df51, df52, df53, df54, df55, df56, df57, df58, df59, df60,
             df61, df62, df63, df64, df65, df66, df67, df68, df69, df70,
             df71, df72, df73, df74, df75, df76, df77, df78, df79, df80,
             df81, df82, df83, df84, df85, df86, df87, df88, df89, df90,
             df91, df92, df93, df94, df95, df96, df97, df98, df99, df100)

Any thoughts on how to optimize this would be greatly appreciated.

Sorry about that -- I updated the code above for others who may have this question at a later date. — Entropy, Dec 08 '14 at 15:21

score 6 · Accepted Answer · answered Dec 07 '14 at 23:01

Or with data.table::rbindlist . Set fill to true to take care of the missing values, if any.

rbindlist(mget(ls(pattern="df")), fill=TRUE)

         x          y
    1:   1    1.00000
    2:   2   11.09091
    3:   3   21.18182
    4:   4   31.27273
    5:   5   41.36364
   ---               
 9996:  96  959.63636
 9997:  97  969.72727
 9998:  98  979.81818
 9999:  99  989.90909
10000: 100 1000.00000

score 4 · Answer 2 · answered Dec 07 '14 at 22:21

4

do.call comes handy. The function you specify works on a list of arguments.

library(plyr)
df.fill <- lapply(ls(pattern = "df"), get)
df <- do.call("rbind.fill", df.fill)

> str(df)
'data.frame':   10000 obs. of  2 variables:
 $ x: int  1 2 3 4 5 6 7 8 9 10 ...
 $ y: num  1 11.1 21.2 31.3 41.4 ...

answered Dec 07 '14 at 22:21

Roman Luštrik

69,533
24
154
197

Yep, though `mget(ls(pattern="df"))` would shorten the creation of the list slightly. – thelatemail Dec 07 '14 at 22:43
Yeah but then you have names of names – Rich Scriven Dec 07 '14 at 22:52
I'm not entirely sure what you mean, Richard? – Roman Luštrik Dec 08 '14 at 09:03
Sorry, I was refering to thelatemail's comment because `mget` returns a named list. But I suppose it doesn't matter. – Rich Scriven Dec 08 '14 at 22:05

score 0 · Answer 3 · answered Mar 27 '17 at 08:37

0

We can use bind_rows from dplyr

library(dplyr)
res <- bind_rows(mget(paste0("df", 1:100)))

answered Mar 27 '17 at 08:37

akrun

874,273
37
540
662

score 0 · Answer 4 · answered Jan 17 '20 at 04:25

0

If you already have your df as a list which contains all your datasets from 1 to 100 you could have used a for loop as follows:

new_list <- list() for(i in 1:100){ new_list <- rbind.fill(newlist, data.frame(df[[i]])) }

answered Jan 17 '20 at 04:25

A.K

1
1

Mass rbind.fill for many data frames

4 Answers4

Linked

Related