Aggregate data from different files into data structure

Question

I noticed I encounter this task quite often when programming in R, yet I don't think I implement it "pretty".

I get a list of file names, each containing a table or a simple vector. I want to read all the files into some construct (list of tables?) so I can later manipulate them in simple loops.

I know how to read each file into a table/vector, but I do not know how to put all these objects together in one structure (list?).

Anyway, I guess this is VERY routine so I'll be happy to hear about your tricks.

this really is a repeat question with the one Shane points out below: http://stackoverflow.com/questions/2209258/merge-several-data-frames-into-one-data-frame-with-a-loop/2209371#2209371 — JD Long, Aug 06 '10 at 16:18

score 3 · Answer 1 · answered Aug 06 '10 at 15:27

3

Do all the files have the same # of columns? If so, I think this should work to put them all into one dataframe.

library(plyr)
x <- c(FILENAMES)
df <- ldply(x, read.table, sep = "\t", header = T)

If they don't have all the same columns, then use llply() instead

answered Aug 06 '10 at 15:27

JoFrhwld

8,867
4
37
32

That will work even if the files don't have the same number of columns. – hadley Aug 06 '10 at 23:51

score 2 · Answer 2 · edited May 23 '17 at 12:13

2

You can have a look at my answer here: Merge several data.frames into one data.frame with a loop.

edited May 23 '17 at 12:13

Community

1
1

answered Aug 06 '10 at 13:09

Shane

98,550
35
224
217

score 2 · Accepted Answer · answered Aug 06 '10 at 15:44

2

Or, without plyr:

filenames <- c("file1.txt", "file2.txt", "file3.txt")
mydata <- array(list(NULL))

for (i in 1:length(filenames))
    {
    mydata[[i]] <- read.table(filenames[i])
    }

answered Aug 06 '10 at 15:44

nico

50,859
17
87
112

1

but better to pre-allocate the list, and might as well use the filenames: mydata <- vector("list", length(filenames)); names(mydata) <- filenames; ... mydata[[filenames[i]]] <- read.table(filenames[i]) – mdsumner Aug 08 '10 at 22:23
@mdsumner: What's the advantage of preallocating? Also, in this way you can put the `read.table` statement in a `tryCatch` instruction, so if file does not exist/is corrupted etc, you end up with a NULL in `mydata` which can then be checked against later in the script – nico Aug 09 '10 at 06:55
it saves memory usage - otherwise the entire mydata list is recreated each time it is visited by an incremented [[i]] – mdsumner Aug 09 '10 at 07:51

Aggregate data from different files into data structure

3 Answers3