1

I have a huge .gz file with millions of data. I would like to read this .gz file in R as I want to do some manipulations to each of its line. I tried:

gzf <- gzfile("myfile.gz", open = "r")  
f <- readLines(gzf)

However this takes forever in R and it gives me a memory error. Is there an efficient way to read it? Thanks

user9463814
  • 199
  • 11
  • Take a look at this: https://stackoverflow.com/questions/37727865/how-can-i-use-fread-to-read-gz-files-in-r – NelsonGon Mar 21 '19 at 11:07
  • What format is your data? `readr::read_csv` can directly read a gz file – Sonny Mar 21 '19 at 12:58
  • @NelsonGon it gives a memory error: Error: cannot allocate vector of size 40.9 Gb. Is there a way around this? – user9463814 Mar 21 '19 at 14:24
  • It's really hard to tell but maybe try the suggestions here: https://stackoverflow.com/questions/5171593/r-memory-management-cannot-allocate-vector-of-size-n-mb – NelsonGon Mar 21 '19 at 14:56

0 Answers0