13

I have been trying hard to solve this, but I cannot get my head around how to read zipped .csv files in R. I could first unzip the files and then read them, but since the amount of unzipped data is around 22GB, I guess it is more practical to handle zipped files.

I basically have many .csv files, which I ZIPPED ONE BY ONE into single .7z files. Every file is named like: file1.csv, file2.csv, etc., which zipped became respectively: file1.csv.7z, file2.csv.7z, etc.

If I use the following command:

data <- read.table(unz("substn-20100101.csv.7z", "substn-20100101.csv"), nrows=10, header=T, quote="\"", sep=",")

I get the message: 

Error in open.connection(file, "rt") : cannot open the connection
In addition: Warning message:
In open.connection(file, "rt") : cannot open zip file 'substn-20100101.7z'

Any help would be much appreciated, thank you in advance.

redmode
  • 4,821
  • 1
  • 25
  • 30
bosspe
  • 145
  • 1
  • 1
  • 7
  • This post might be relevant - http://stackoverflow.com/questions/3053833/using-r-to-download-zipped-data-file-extract-and-import-data – zx8754 Feb 17 '14 at 13:59
  • 2
    Have you correctly set your working directory? Your data file needs to be there. https://sites.google.com/site/manabusakamoto/home/r-tutorials/r-tutorial-4 – JosefAssad Feb 17 '14 at 14:02
  • 2
    Hi both! @zx8754 : Thank you, I have seen the post already, but I get the following error: Error in open.connection(file, "rt") : cannot open the connection In addition: Warning message: In open.connection(file, "rt") : cannot open zip file 'substn-20100101.7z:C' @ Josef Assad: yes, I use the following code to set the working directory and it always worked: rm(list=ls(all=TRUE)) Sys.setlocale("LC_TIME", "C") # Windows rm(list=ls()) main_dir <- "C:/Users/.../Forecast-Machine Learning" setwd(main_dir) temp <- tempfile() data <- read.table(unz("substn-20100101.7z",temp)) unlink(temp) – bosspe Feb 17 '14 at 14:07
  • What does `file.exists("substn-20100101.7z")` return? – Richie Cotton Feb 17 '14 at 14:32
  • In general, you should use the higher level `unzip` in the `utils` package over `unz`. – Richie Cotton Feb 17 '14 at 14:34
  • 3
    No one seems to have pointed out that `7z` != `zip`. I'm not aware of built-in support for 7z. – hadley Feb 17 '14 at 14:39
  • Richie: Hi Richie, the 'file.exists("substn-20100101.7z")' returns true. Thank you -------------- Hadley: actually I'm not sure about 7z. I just give it a try with .zip and get back to the post. Thank you – bosspe Feb 18 '14 at 15:57

3 Answers3

9

First of all if your problem is RAM, as you said each file has 22G, using compressed files won't resolve your problems. After read.table, for example, all file will be loaded in memory. If you are using these files to some kind of modeling i advise you to look at ff and bigmemory packages.

Another solution is use Revolutions R that has an academic licence and you can use for free. Revolutions R provides Big Data capabilities and you can manage this files easily with packages like revoscaleR.

Even another solution is using Postgres + MADLib + PivotalR. After ingesting data at Postgres, use PivotalR package to access that data and do models with MADLib library, directly from R console.

BUT, if you are planing something that be done with chunks of data, summary for example, you can use the package iterators. I will provide an use case to show how this can be done. Get Airlines data, 1988, and follow this code:

> install.packages('iterators')
> library(iterators)
> con <- bzfile('1988.csv.bz2', 'r')

OK, now you have a connection to your file. Let's create an iterator:

> it <- ireadLines(con, n=1) ## read just one line from the connection (n=1)

Just to test:

> nextElem(it)

and you will see something like:

1 "1988,1,9,6,1348,1331,1458,1435,PI,942,NA,70,64,NA,23,17,SYR,BWI,273,NA,NA,0,NA,0,NA,NA,NA,NA,NA"

> nextElem(it)

and you will see the next line, and so on. Be aware that you are reading a line at a time, so you are not loading all the file to RAM.

If you want to read line by line till the end of the file you can use 

> tryCatch(expr=nextElem(it), error=function(e) return(FALSE))

for example. When the file ends it return a logical FALSE.

Flavio Barros
  • 996
  • 1
  • 11
  • 29
1

If I understand the question correctly, at least on Windows OS, you could use 7-Zip Command-Line.

For the sake of simplicity put 7za.exe in your R working directory (and your 7zip files), create .bat file with the following text in it: 

"7za e *.7z -y"

...than in R you run the following code:

my_batch <- "your_bat_file_name.bat"
shell.exec(shQuote(paste(my_batch), type = "cmd"))

Than you just read.table()... It works for me.

Miha Trošt
  • 2,002
  • 22
  • 25
1

According to the readr package documentation, readr::read_csv and fellows will automatically unzip files ending in .gz, .bz2, .xz, or .zip. Although .7z is not mentioned, perhaps a solution is to change to one of these compression formats and then use readr (which also offers a number of other benefits). If your data is compressed with zip, your code would be:

library(readr)
data <- read_csv("substn-20100101.csv.zip", n_max=10)
James Brusey
  • 355
  • 3
  • 11