0

I have to import an entire folder containing csv files and this takes up to 30min sometimes. At the moment I am using this code:


data <- do.call(rbind, lapply
                 (files, read.csv, as.is=T, sep = "{", skip = 4, header = TRUE, fileEncoding="utf-16", quote = "", fill = FALSE))

Do you know a faster option?

Beginner
  • 37
  • 7
  • 4
    instead of `read.csv` maybe you could try `data.table::fread`, you could also read in only the columns you need, see [here](https://stackoverflow.com/questions/56396770/is-there-a-faster-way-than-fread-to-read-big-data) – user63230 Nov 16 '20 at 14:36
  • 4
    With `data.table`, `rbindlist(lapply(files, fread))` will be **much** faster. With `dplyr` and `readr`, `bind_rows(lapply(files, read_csv))` will be somewhat faster. You could also use `vroom` which may be faster or slower than data.table depending on what you need, [they have custom support for reading multiple files](https://github.com/r-lib/vroom#reading-multiple-files). – Gregor Thomas Nov 16 '20 at 14:41
  • 1
    You question is similar to https://stackoverflow.com/questions/11433432/how-to-import-multiple-csv-files-at-once. – Gopal Srivastava Nov 16 '20 at 15:24
  • Thank you for your inputs. Unfortunatly it's not that easy, because I really need the fileEnconding argument which is not supported by fread. Running read.cvs without that argument lead to Error in read.table(file = file, header = header, sep = sep, quote = quote, : empty beginning of file – Beginner Nov 17 '20 at 10:08

0 Answers0