0

I have 200 CSV data files (their name is xxx.csv) in "C:\Data". I want to convert all these csv files to RData files. However, each of them are very large, and I am not able to read all of them together into R.

How can I read these files into R, save them to RData file and then delete them from R environment one by one using a function?

Franky
  • 53
  • 5

3 Answers3

1

Try this:

csv2rdata <- function(filename) {
  nm <- sub("\\.csv$", "", basename(filename))
  assign(nm, data.table::fread(filename), envir = environment())
  save(list = nm, file = paste0(dirname(filename), "/", nm, ".RData"), envir = environment())
}

dir("C:\\Data", pattern = "*.csv", full.names = TRUE) %>% lapply(csv2rdata)
ekoam
  • 8,744
  • 1
  • 9
  • 22
0

using this link. you can use the data.table package to read in large amounts of data. Keep in mind, R uses in memory processing, so you need a little bit of hardware if you have large data ~5GB-10GB. Then you can save them as .RData files, then use the rm() inside a loop to delete them.

library(data.table)
library(tidyverse)

tbl_fread <- 
    list.files(pattern = "*.csv") %>% 
    map_df(~fread(.))
Daniel_j_iii
  • 3,041
  • 2
  • 11
  • 27
  • 1
    FYI `map_df` is an undocumented alias of `map_dfr` from `purrr`. It also requires `dplyr` to be installed. – bcarlsen Sep 30 '20 at 15:21
  • I have 200 files.....I am not able to read them together – Franky Sep 30 '20 at 15:22
  • but wouldn't `list.files(pattern = "*.csv")` just list all the files within the respective directory, and then you would read them in 1 by 1 in a function/loop – Daniel_j_iii Sep 30 '20 at 15:42
0

I would try the following, suppose you already have a list with the filenames:

library(data.table)

for (i in 1:length(my_files)) {
  df <- fread(myfiles[i])
  save(df, ....) # use your specification for saving including the filename
}
peter
  • 756
  • 5
  • 16
  • what does not work? missing a list with files? or something with the save function? I don't know how you want to save the files. It would be nice if you show your save function – peter Sep 30 '20 at 15:52