I have a data set in txt format having 31968 files and each file contains 365 values in one column. I want to combine each 48 files in sequence and that will bring 17520 values as result.
Inputs as like
a = (X1, x2…………………………………x48)
b =(x49, x50 ………………………………x96)
Expected outputs like as
a = (1, 2, 3, 4,………………………………..17520)
b= (1, 2, 3, …………………………………..17520)
How I can load the bunch of 31968 file and execute this work in R.
setwd('files location')
file_names = list.files()
for(i in 1:length(file_names)){
p = data.frame()
for(i in i:i+47){
setwd('input files location')
d = read.table(file_names[i],col.names=c("id"),strip.white=TRUE)
p = rbind(p,d)
}
setwd('output location')
write.table(p,paste(file_names[i],".txt",sep=""),row.names=F)
}
See that all your input files lie in a single directory and no other files are present in that directory. The output is created with the name of 48th file read in the every combining list of 48 files.
Since 31968/48 gives 666, create a list with 666 vectors, each contain 48 file names.
file_names <- list.files(path=".", pattern="\\.txt") # change the path to the directory where the files are kept
list_of_files <- lapply(1:666, function(x) file_names[((x-1)*48 + 1):((x-1)*48 + 48)])
Read the files into R as list_of_data and use do.call & rbind to convert into a single data.frame.
for(i in 1:666){
list_of_data <- lapply(list_of_files[[i]], read.table, sep="\t") # put in appropriate read.table parameters for the text files
assign(paste0("a", i), do.call(rbind, list_of_data))
}
Alternative:
for(i in 1:666){
list_of_data <- lapply(list_of_files[[i]], read.table, sep="\t")
assign(sprintf("a.%03d", i), do.call(rbind, list_of_data))
}
This should return 666 objects e.g.
"a.001" "a.002" "a.003" "a.004" "a.005" "a.006" "a.007" "a.008" "a.009" "a.010" "a.011"
"a.012" "a.013" "a.014" "a.015" "a.016" "a.017" "a.018" "a.019" "a.020" "a.021" "a.022"
To merge all 666 data.frame:
frames <- grep("a[.]", ls(), value=T)
library(plyr)
output <- ldply(frames, get)
