How to operate on multiple files within a folder without having to load them into R

Question

As part of the problem I'm solving, I need to append 50 files into one file. I was hoping if I could do the same using a for loop but I'm unable to do so since list.files() just gives the literal name of the file and not the data within it.

Could someone help me execute a similar logic in R:

c = list.files()
for (i = 1:length(c))
{
  a = c[i]
  % any operation on 'a' since it now contains the data of the first file in c
}

But the above does not seem to work because 'c' has just the literal names of the file and not the data.

Could someone please guide me on how this can be done?

Generally, you _do_ have to load them into R, e.g. https://stackoverflow.com/q/11433432/324364 — joran, Nov 22 '17 at 17:43
See this blog post: https://www.r-bloggers.com/merging-multiple-data-files-into-one-data-frame/ — Peter K, Nov 22 '17 at 17:49

eipi10 · Answer 1 · 2017-11-22T17:55:25.207

As far as I know, there's no way to do it in R without loading the files into R. However there are a number of ways in R to read in the files, combine them and then write a single combined file. An example is below, where we assume we want to read in, combine and save all csv files in the current directory.

library(tidyverse)

map_df(list.files(pattern="csv$"), function(f) {
  read_csv(f)
}) %>% write_csv(., "New file.csv")

score 0 · Answer 2 · answered Nov 22 '17 at 19:15

Store the file names in an object.

> fl_list <- list.files()

You can see the files present in the directory

> fl_list
[1] "credit.csv"  "credit1.csv" "credit2.csv"
[4] "credit3.csv"

I think you want to append the data by combining them by rows, so I have used rbind.

You will have to create an empty data frame.

> credit_main <- data.frame()

Now you can run the good old for loop on the list of the files present and bind them by rows.

> for(i in 1:length(fl_list)){
+   credit <- read.csv(fl_list[i])
+   credit_main <- rbind(credit, credit_main)
+ }

Check the dimension of the new data frame created.

> dim(credit_main)
[1] 4000   10

I was using the same file with different names for this example

> dim(credit)
[1] 1000   10

score 0 · Answer 3 · answered Nov 24 '17 at 02:11

There is a way to concatenate the files without reading them into R. One can use the system() function in R to execute an operating system command that concatenates files, such as the cat command in Linux / OS X or the type command in Windows.

Adapting Alberto Barradas' Pokémon Stats data from kaggle.com, I have a set of 6 CSV files, one for each of the first 6 generations of Pokémon.

Here is a script that reads the names of the CSV files from disk and uses paste() to turn them into a cat command that can be executed in OS X to combine them into a single CSV file, output.csv.

theGenFiles <- list.files(path="~/gitrepos/pokemonData/",
                          pattern="gen0[123456]\\.csv",
                          full.names=TRUE)
fileString <- NULL
for (i in 1:length(theGenFiles)) fileString <- paste(fileString,theGenFiles[i])

theOSCmd <- paste("cat",fileString," > ~/gitrepos/pokemonData/output.csv")

system(theOSCmd)

regards,

Len

How to operate on multiple files within a folder without having to load them into R

3 Answers3