How can I read multiple files from multiple directories into R for processing?

Question

I am running a simulation study and need to process and save the results from several text files. I have the data organized in such a way where there are sub directories and within each sub directory, I need to process and get individual results for 1000 data files. This is very easy to do in SAS using macros. However, I am new to R and cannot figure out how to do such. Below is what I am trying to accomplish.

DATA Folder-> DC1 -> DC1R1.txt ... DC1R1000.txt
              DC2 -> DC2R1.txt ... DC2R1000.txt

Any help would be greatly appreciated!

Try `list.files`. http://stackoverflow.com/search?q=%5Br%5D+list.files — Roman Luštrik, Sep 11 '11 at 11:11

Tommy · Answer 1 · 2011-09-11T19:44:14.480

I'm not near a computer with R right now, but read the help for file-related functions:

The dir function will list the files and directories. It has a recursive argument. list.files is an alias for dir. The file.info function will tell you (among other things) if a path is a directory and file.path will combine path parts.

The basename and dirname functions might also be useful.

Note that all these functions are vectorized.

EDIT Now at a computer, so here's an example:

# Make a function to process each file
processFile <- function(f) {
  df <- read.csv(f)
  # ...and do stuff...
  file.info(f)$size # dummy result
}

# Find all .csv files
files <- dir("/foo/bar/", recursive=TRUE, full.names=TRUE, pattern="\\.csv$")

# Apply the function to all files.
result <- sapply(files, processFile)

Thank you all for your responses! @Tommy.....Your coding is working so far....fingers crossed....I have 768 data conditions with 1000 simulated files under each condition. It will take awhile to process, so I will give a status once finished. — Stefanie, Sep 11 '11 at 20:59

score 8 · Answer 2 · answered Sep 11 '11 at 13:46

8

If you need to run the same analysis on each of the files, then you can access them in one shot using list.files(recursive = T). This is assuming that you have already set your working directory to Data Folder. The recursive = T lists all files within subdirectories as well.

answered Sep 11 '11 at 13:46

Ramnath

54,439
16
125
152

Thanks Ramnath. I have never list.files but will give it a try. Tommy's coding is working so far. – Stefanie Sep 11 '11 at 21:04

score 5 · Answer 3 · answered Mar 15 '17 at 12:40

5

filenames <- list.files("path/to/files", recursive=TRUE) This will give you all the files residing under one folder and sub folders under it.

answered Mar 15 '17 at 12:40

Aditi

820
11
27

score 0 · Answer 4 · answered Sep 11 '11 at 07:15

0

You can use Perl's glob () function to get a list of files and send it to R using, e.g., RSPerl's interface.

answered Sep 11 '11 at 07:15

Itamar

2,111
13
16

Hi Ltamar. I am not familiar with Perl. I am a statistician and have never had the need to use such. Although....I hear it is quite powerful. – Stefanie Sep 11 '11 at 21:06

How can I read multiple files from multiple directories into R for processing?

4 Answers4

Linked