how to process multiple txt files while preserving filenames and content?

Question

I have a set of txt files on a folder. I have already listed them on R and added them as separate datasets

file.list <- list.files(path = folder, ignore.case= T, pattern='txt', full.names = TRUE)

file.list = c("test/2021-04-22_1_7787551954_NI_800240882_EPS046_13_IP_2021-04.TXT", 
"test/2021-04-23_1_20313433_NI_800102838_EPS046_86_IP_2021-05.TXT", 
"test/2021-04-28_1_7787988732_NI_800112806_EPS046_13_IP_2021-05.TXT", 
"test/2021-04-30_1_7788094522_NI_900391901_EPS046_19_IP_2021-05.TXT", 
"test/2021-04-30_1_9419310550_NI_890200500_EPS046_84_IP_2021-05.TXT"
)

for (i in 1:length(file.list)) assign(file.list[i], read_csv(file.list[i],col_names = FALSE,locale = readr::locale(encoding = "latin1")))

now i need to recursively apply a function to each newly created dataframe file from that file list and then saving them back again to a different folder, while maintaining original filenames.

this is the function that i need to apply on each dataframe

df.test$X1 <- sub("^(.{366}).{9}$", "\\1", df.test$X1, perl = TRUE)

will appreciate your advise

For so many reasons (not the least of which is that whatever you do to one table here is likely to be done to all tables), it is usually better in situations like this to load them into a [list of frames](https://stackoverflow.com/a/24376207/3358227), using something like `LOF <- lapply(file.list, read.csv, col_names = FALSE, locale=...)`. — r2evans, May 12 '21 at 23:42

score 0 · Answer 1 · answered May 12 '21 at 22:18

To construct new path with same file names you can do this.

paste('newPath/', sapply(strsplit(file.list, "/"), "[", 2), sep = '')

And regarding multiple dfs or a list of dataframes, it all depends upon how you are going to use your data in future. I mean if you are going to use R only, then its better to create a list of dfs and you can easily name the elements with filenames as well and save as .rds

score 0 · Answer 2 · answered May 13 '21 at 04:53

With the help of lapply you can do :

library(readr)

lapply(file.list, function(x) {
  data <- read_csv(x,col_names = FALSE,
                   locale = readr::locale(encoding = "latin1"))

  data <- apply_fun(data)
  write_csv(data, paste0('new_path/', basename(x)))
})

where apply_fun is the function :

apply_fun <- function(df.test) {
  df.test$X1 <- sub("^(.{366}).{9}$", "\\1", df.test$X1, perl = TRUE)
  df.test  
}

how to process multiple txt files while preserving filenames and content?

2 Answers2