How do I read in specific files from a huge set of the same?

Question

I was trying to read in some specific csv files from a set of files (332 of them) named ( 001.csv, 002.csv ...). I wrote the following code to extract the files

x <- as.vector(list.files())
for (i in 1:4)
{
mydata <- data.frame(read.csv(x[i]))
}

don't understand what's wrong with the code. It only reads in the last file (i.e. 004.csv)

What's the problem, are you getting an error message or an undesired behavior? — TidyDev, Jan 29 '18 at 09:42
if you give the same name to your file each time, only the last one will be in your environment. You should import your files in a list. — Cath, Jan 29 '18 at 09:49
Imported the files in a list. It still reads in only the last file (i.e. 004.csv) instead of reading in all the files from 001.csv to 004.csv — Pulkit Chaturvedi, Jan 29 '18 at 10:08

Kumar Manglam · Accepted Answer · 2018-01-29T10:49:48.503

From your problem description, I understood that you want to import 4 files and then want to concatenate them into a single data frame. What you are doing wrong is that you are reading a file and then assigning that into a variable. Each time the loop will execute, the variable mydata will be updated, and that's why you are getting only the last file data into your file.

To overcome this, you can use lists. Below is the sample code, which is doing the same.

# create the sample four files in a temp directory
tempDir <- "temp"
dir.create(tempDir)

set.seed(123)
file1 = data.frame(x=rnorm(100), y= rnorm(100))
write.csv(file1, file=file.path(tempDir, "1.csv"), row.names = FALSE)

file2 = data.frame(x=rnorm(100), y= rnorm(100))
write.csv(file2, file=file.path(tempDir, "2.csv"), row.names = FALSE)

file3 = data.frame(x=rnorm(100), y= rnorm(100))
write.csv(file3, file=file.path(tempDir, "3.csv"), row.names = FALSE)

file4 = data.frame(x=rnorm(100), y= rnorm(100))
write.csv(file4, file=file.path(tempDir, "4.csv"), row.names = FALSE)

# Get the name of files in a vector
x <- as.vector(list.files(tempDir))
wd <- getwd() # keep the current directory information
setwd(tempDir) # change working directory to data folder directory

mydata <- list() # use lists to have data
for (i in 1:4) {
  mydata[[i]] <- data.frame(read.csv(x[i]))
} 
mydata1To4 <- do.call(rbind, mydata) # concatenate list data into a single data frame
mydata1To4 <- data.frame(mydata1To4)

head(mydata1To4)
dim(mydata1To4)

setwd(wd) # reset working directory

Hope the solution works, and you are able to understand it.

EDIT : As pointed in the comment, you can use lapply to remove the loop.

How do I read in specific files from a huge set of the same?

1 Answers1