for loop across different folders and years to read in csv data

Question

I am trying to write a simple for loop to read in a number of .csv files. I have looked at list.files(pattern="data.*csv") which I do not think solves my problem.

I have a link to the data which looks like the following;

read.csv("C:/Users/user/Desktop/data/Year1/beer/beer.csv")

I have many years worth of data, I am trying to write something which changes Year1 to (for i in 1:15)...

Secondly I have many products and for now I am only interested in importing data for all years for the beer product, so I am trying to create a separate vector of products i.e. products <- c("beer", "bread", "milk"), which I can load in at a later stage.

The format of the product folders are all the same so, milk may be C:/Users/user/Desktop/data/Year1/milk/milk.csv. Also the file names are the same across all years, so milk.csv in year 1 is also called milk.csv in year 7 for example.

I can paste what I have currently

Len Greski · Answer 1 · 2018-10-07T22:44:51.513

Leveraging the data that you previously posted in Loading Multiple Files into R at the same time with similar file names, here is one way to subset the result of list.files() or dir() for specific products.

We will subset the list to those containing products beer or milk.

aFileList <- c("Year1/beer/beer.csv",
               "Year1/blades/blades.csv",
               "Year1/carbbev/carbbev.csv",
               "Year1/cigets/cigets.csv",
               "Year1/mayo/mayo.csv",
               "Year1/milk/milk.csv",
               "Year1/mustketc/mustketc.csv",
               "Year2/beer/beer.csv",
               "Year2/blades/blades.csv",
               "Year2/carbbev/carbbev.csv",
               "Year2/cigets/cigets.csv",
               "Year2/mayo/mayo.csv",
               "Year2/milk/milk.csv",
               "Year2/mustketc/mustketc.csv")

aFileList[grep("beer|milk",aFileList)]

The grep() function returns a vector of index numbers for elements of the input vector that contain the tokens requested in the regular expression that is the first argument to grep(). This is used to subset the original vector of file names.

...and the output:

> aFileList[grep("beer|milk",aFileList)]
[1] "Year1/beer/beer.csv" "Year1/milk/milk.csv" "Year2/beer/beer.csv" "Year2/milk/milk.csv"
>

If you use this technique, then you can use lapply() to read the files, per my answer to Loading Multiple Files into R at the same time with similar file names, eliminating the need for a for() loop.

Great! Thanks I will give this a go a let you know later today! — user113156, Oct 08 '18 at 09:54

score 0 · Answer 2 · answered Oct 11 '18 at 23:10

There are so many ways to merge multiple CSV files in a folder into one. Here are a few thoughts...

setwd("C:/your_path_here/CSV Files/")
fnames <- list.files()
csv <- lapply(fnames, read.csv)
result <- do.call(rbind, csv)

filedir <- setwd("C:/your_path_here")
file_names <- dir(filedir)
your_data_frame <- do.call(rbind,lapply(file_names,read.csv))

filedir <- setwd("C:/your_path_here")
file_names <- dir(filedir)
your_data_frame <- do.call(rbind, lapply(file_names, read.csv, skip = 1, header = FALSE))

To get every file from multiple folders, into one single folder, try the following....

xcopy *.ext destination /s

where ext identifies the type of file you want to copy, and destination where you want it copied to. For instance, to copy all of your *.docx files to D:\alldocx, type xcopy *.docx d:\alldocx /s.

for loop across different folders and years to read in csv data

2 Answers2