writing groups of rows as matrices to multiple files in R

Question

I'm new to stackoverflow and relatively new to R. Please point out anything I am doing wrong, or could do better, to ask my question on this forum - thanks!

I have a matrix in R with data that has been summarized by year and month, and I want to write out one file per year, with 12 rows per file (one row for each month), along with some additional text into each output file.

I have looked at a couple different solutions on this forum:

this one explains a technique with tapply to create a list of matrices of salaries by name, then write them to a file: Write many files in a for loop
this one shows how to add multiple lines of text to a file: Write lines of text to a file in R

These have elements of what I want to do, but tapply seems to be limited to one column ("salary" in the example), and I want to grab 3 to write out. I'm also mixing writing data and free-form text.

My data in R looks like this:

Month | Year | Precip | Temp | PanEvap
01 | 2018 | 20 | -4 | 12
02 | 2018 | 15 | 0 | 10
03 | 2018 | 60 | 5 | 40
... etc

(sorry for the lack of formatting - not sure how to make a table, or use fixed-width font in the final posting!)

I need to output one file for each year like this (this would be called "18.dat", the "18" to be taken from the year, "2018"):

"Some text up here"
20 -4 12
15 0 10
60 5 40
...etc
"More text down here"

I hope this makes sense. Any guidance and sample code would be very much appreciated.

Thanks!

`a< -by(data,data$year,identity);setNames(a,paste0("Year",names(a)))`??? — Onyambu, Jul 06 '18 at 01:39
Thanks @Onyambu. I'm not confident I understand what the output of this will be, but I get an error when I run this, even after removing the space inside the arrow after the "a": Error in names(IND) <- deparse(substitute(INDICES))[1L] : 'names' attribute [1] must be the same length as the vector [0] — E Maas, Jul 07 '18 at 17:42

TC Zhang · Accepted Answer · 2018-07-08T02:56:49.063

0

Try this

# Read data
tmp <- data.table::fread('
Month | Year | Precip | Temp | PanEvap
01 | 2018 | 20 | -4 | 12
02 | 2018 | 15 | 0 | 10
03 | 2018 | 60 | 5 | 40
')


years = unique(tmp$Year)

for (i in years){

    text1 = "Some text up here"
    # remove month and year column
    tmp_df =tmp[which(tmp$Year == i),c(-1,-2)] 
    text2 = "More text down here"
    # use file connection with append mode

    fileConn <- file(paste0("./",i,".dat"),open="w")
    writeLines(text1, fileConn)
    close(fileConn)

    fileConn <- file(paste0("./",i,".dat"),open="a")     
    write.table(tmp_df, fileConn,col.names = FALSE,row.names = FALSE)
    writeLines(text2,fileConn)
    close(fileConn)
}

and in 2018.dat :

    Some text up here
    20 -4 12
    15 0 10
    60 5 40
    More text down here

edited Jul 08 '18 at 02:56

answered Jul 06 '18 at 02:34

TC Zhang

2,757
1
13
19

Thank you @TC. It works exactly as I specified. The only adjustment I would want is to make it re-runnable so that it opens not appendable initially (so any existing files from previous runs are overwritten), but then append when each subsequent data and line is written. Not sure how to do that with the file connection controlling all the writes. – E Maas Jul 07 '18 at 17:38
Specifically, I get this error when I try to change the fileConn 'open' parameter to open="" : "Error in writeLines(text2, fileConn) : invalid connection" – E Maas Jul 07 '18 at 18:10
I edited the code. By the way you could `help(file)` to learn more about file connections – TC Zhang Jul 08 '18 at 02:53

score 0 · Answer 2 · answered Jul 07 '18 at 18:02

After fiddling more with my code and learning more about other functions, this is what I ultimately came up with, borrowing from the posts I linked to in my initial post (and I also changed one year to 2017 just to prove multiple files could be created):

tmp <- data.table::fread('
                         Month | Year | Precip | Temp | PanEvap
                         01 | 2018 | 20 | -4 | 12
                         02 | 2018 | 15 | 0 | 10
                         03 | 2017 | 60 | 5 | 40
                         ')
lst <- split(tmp[,3:5], tmp$Year)
lapply(names(lst),
       function(x, lst) {cat("some text up here",
                             file=paste(x,".dat", sep=""), sep="\n",append=FALSE)
                         write.table(lst[[x]], file=paste(x,".dat", sep=""),
                            col.names=FALSE, row.names=FALSE, sep="\t", 
                            quote=FALSE, append=TRUE)
                         cat("some text down here", "and more I needed, separated by a tab", 
                             file=paste(x,".dat", sep=""), sep="\t", append=TRUE)
                        },
       lst)

writing groups of rows as matrices to multiple files in R

2 Answers2