Reading csvs dynamically into R dfs via a list or a vector?

Question

I'm working with some yearly and quarterly data, and I want to write a loop that cycles through and reads four quarters of data from four different CSVs each year, then run a script on them.

#prep strings

beg <- "fmli"
end <- ".csv"

fileList <- c()

for(i in 1996:2012){
  yearString <- toString(i)
  year <- substr(toString(i), 3,4)
  for(i in 1:4){
    quarter <- toString(i)
    fileToRead <- paste(sep="", beg, year, quarter, end)
    #add to list
    #fileList <- append(fileToRead, i)
  }
  a1 <- read.csv(fileList[1])
  a2 <- read.csv(fileList[2])
  a3 <- read.csv(fileList[3])
  a4 <- read.csv(fileList[4])
}

Essentially, I need this loop to end by assigning that year's 4 quarters to a1-a4. So that I can accomplish this:

#I want it to do this, but automated

#1996 quarterly files
a1 <- read.csv("fmli961.csv")
a2 <- read.csv("fmli962.csv")
a3 <- read.csv("fmli963.csv")
a4 <- read.csv("fmli964.csv")

#run the script

#1997 quarterly files
a1 <- read.csv("fmli961.csv")
a2 <- read.csv("fmli962.csv")
a3 <- read.csv("fmli963.csv")
a4 <- read.csv("fmli964.csv")

#run the script again

#etc, etc

I'm at a loss as to how to append to the list within the for loop -- I understand that this is probably not the most efficient way to go about writing this in R, but I wrote the script by assigning the quarters to DFs so I'm kinda married to this approach...

Possible duplicate of [Fast reading and combining several files using data.table (with fread)](http://stackoverflow.com/questions/21156271/fast-reading-and-combining-several-files-using-data-table-with-fread) — MichaelChirico, Mar 25 '16 at 15:31

fanli · Accepted Answer · 2016-03-25T17:44:30.800

3

Use a list:

beg <- "fmli"
end <- ".csv"

for(i in 1996:2012){
  yearString <- toString(i)
  year <- substr(toString(i), 3,4)
  dataList <- list()
  for(j in 1:4){
    quarter <- toString(j)
    fileToRead <- paste(sep="", beg, year, quarter, end)
    #add to list
    dataList[[j]] <- read.csv(fileToRead)
  }
    names(dataList) <- c("a1", "a2", "a3", "a4")
    # then run your script on dataList[["a1"]], dataList[["a2"]], etc.
}

Oops, forgot the double brackets for list. I always seem to do this :/

edited Mar 25 '16 at 17:44

answered Mar 25 '16 at 15:31

fanli

1,069
7
13

For whatever reason, this only appends one column of the data frame to the list, and the list becomes a list of 4 integer vectors. Why is this happening? – skathan Mar 25 '16 at 16:30
Yep. Figured that out. Also, it probably makes a little bit more sense to change that second i in the inner loop to 'j', as others have done. But this worked and was closest to what I was asking for, so I'm gonna go ahead and give it the checkmark :) – skathan Mar 25 '16 at 17:30
Good call, overwriting `i` totally didn't break everything :) – fanli Mar 25 '16 at 17:46

score 1 · Answer 2 · answered Mar 25 '16 at 15:31

Here's how I'd do it. Instead of using fors, just use lapply instead. The object returned by the function inside it will be part of a list.

lapply(setNames(1996:2012, paste0("y", 1996:2012)), function(y) {
  year <- substr(y, 3, 4)
  lapply(setNames(1:4, paste0("a", 1:4)), function(qt) {
    fileToRead <- paste0("fmli", y, qt, ".csv")
    read.csv(fileToRead)
  })
})

Using nested lapplys you'll end up with a list of years, and its elements are lists of data.frames for each quarter. We use setNames to create a named vector, these names will be the names of the list elements (years and quarters).

Note that you don't need to use toString when using a number as a string. On most (all?) functions this coercion is done automatically.

`paste` will attempt to coerce to character, but I think the broader point for the OP is that they have mistaken `toString` for what `as.character` was designed for, I think. — joran, Mar 25 '16 at 15:41
@joran Indeed, I even forgot about that. That's why it looked like I was reading another language... — Molx, Mar 25 '16 at 15:44

Cinnamon Star · Answer 3 · 2016-03-25T17:01:01.463

-1

I always use assign to read my .csv files.

#prep strings

beg <- "fmli"

end <- ".csv"

a <- c("a1", "a2", "a3", "a4");

for(i in 1996:2012){
  year <- substr(toString(i), 3,4)
  for(j in 1:4){
    quarter <- toString(j)
    fileToRead <- paste(beg, year, quarter, end, sep="")
    assign(a[j], read.csv(file=fileToRead));
  }

  # Now use your script on a1-a4
  # a1-a4 will be overwritten in next iteration

}

edited Mar 25 '16 at 17:01

answered Mar 25 '16 at 15:36

Cinnamon Star

24
5

This won't work, as "assign" only works with strings -- this returns the 'invalid first argument' error...any ideas? – skathan Mar 25 '16 at 16:18
Sorry for the inconvenience. Of course the array `a` has to contain strings and it needs to be accessed with `a[j]` and not `a[i]` – Cinnamon Star Mar 25 '16 at 16:42

Reading csvs dynamically into R dfs via a list or a vector?

3 Answers3