R extract a single data from mutliple csv files

Question

I have multiple csv files (more than 100). Each file represents a time period. In each file, there are 29 lines that need to be skiped (text lines). At line 30, I have a matrix of temperature as a function of latitude and longitude coordinates. For example: at latitude 68.80 and longitude 48.40268, the temperature is 5.94.

So, I would like to extract the temperature at a specific combination of latitude and longitude coordinates for every time period I have (for every file).

I can write the code for a single file, but I'm affraid I don't know how to do it in a loop or how to make it faster.

Any help is appreciated, thank you. Sorry if this is similar to other questions, I read what I could find on that topic, but it did not seem to fit for my problem.

The code for one file:

filenames <- list.files(path="E:/Documents...")
fileone <- read.csv(filenames[1], skip=29, header=T, sep=";")
names(fileone) <- c("Lat", "68.88", "68.86", "68.85", "68.83", "68.82", "68.80", "68.79", "68.77", "68.76", "68.74", "68.73", "68.71")
Tempone <- fileone[which(fileone$Lat==48.40268), "68.80"]

for faster check out `fread` in the `data.table` package (but be careful with Date columns if you have any. They will be imported as characters). — Ricardo Saporta, Aug 06 '13 at 15:55
Most if not all are the same. I'll deal with the files where they are different separately. Thanks. — J.VDW, Aug 06 '13 at 16:57

score 3 · Accepted Answer · answered Aug 06 '13 at 16:03

Assuming data size relative to your system are small enough (relative to your system) to fit into memory all together, you can accomplish this in one shot using lists

## Grab the filienames, just like you're doing
filenames <- list.files(path="E:/Documents...")

## Assuming all columns have the same column names
c.nms <- c("Lat", "68.88", "68.86", "68.85", "68.83", "68.82", "68.80", "68.79", "68.77", "68.76", "68.74", "68.73", "68.71")


## Import them all in one shot, as a list of data.frames
AllData <- lapply(filenames, read.table, 
      skip=29, header=TRUE, sep=";", row.names=NULL, col.names=c.nms)

## Then to get all of your rows
PulledRows <- 
lapply(AllData, function(DF) 
    DF[fileone$Lat==48.40268, "68.80"]
  )

If you are pulling different lat/longs per file, you can use mapply with a vector/list of lat/longs

Thank you, it works like a charm! However, I changed the last line of the code by PulledRows <- lapply(AllData, function(DF) DF[8, 4]). I used row and column numbers rather than names. Thanks again! — J.VDW, Aug 06 '13 at 16:55

R extract a single data from mutliple csv files

1 Answers1

Linked