Read gzipped csv directly from a url in R

Question

I'm looking to download a gzipped csv and load it as an R object without saving it first to disk. I can do this with zipped files but can't seem to get it to work with gzfile or gzcon.

Example:

grabRemote <- function(url) {
    temp <- tempfile()
    download.file(url, temp)
    aap.file <- read.csv(gzfile(temp), as.is = TRUE)
    unlink(temp)
    return(aap.file)
}
grabRemote("http://dumps.wikimedia.org/other/articlefeedback/aa_combined-20110321.csv.gz")

That downloads a (small) gz compressed file containing Wikipedia article feedback data (not important, but just to indicate it isn't giant or nefarious).

The code I have works fine but I feel like I'm missing something very obvious by resorting to creating and destroying a temporary file.

score 25 · Accepted Answer · edited Mar 22 '19 at 21:45

25

I am almost certain I answered this question once before. The upshot is that Connections API of R (file(), url(), pipe(), ...) can do decompression on the fly, I do not think you can do it for remote http objects.

So do the very two-step you have described: use download.file() with a tempfile() result as second argument to fetch the compressed file, and then read from it. As tempfile() object, it will get cleaned up automatically at the end of your R session so the one minor fix I can suggest is to skip the unlink() (but then I like explicit cleanups, so you may as well keep it).

Edit: Got it:

con <- gzcon(url(paste("http://dumps.wikimedia.org/other/articlefeedback/",
                       "aa_combined-20110321.csv.gz", sep="")))
txt <- readLines(con)
dat <- read.csv(textConnection(txt))

dim(dat)
# [1] 1490   19

summary(dat[,1:3])
# aa_page_id       page_namespace                 page_title  
# Min.   :     324   Min.   :0      United_States        :  79  
# 1st Qu.:   88568   1st Qu.:0      2011_NBA_Playoffs    :  52  
# Median : 2445733   Median :0      IPad_2               :  43  
# Mean   : 8279600   Mean   :0      IPod_Touch           :  38  
# 3rd Qu.:16179920   3rd Qu.:0      True_Grit_(2010_film):  38  
# Max.   :31230028   Max.   :0      IPhone_4             :  26  
# (Other)              :1214

The key was the hint the gzcon help that it can put decompression around an existing stream. We then need the slight detour of readLines and reading via textConnection from that as read.csv wants to go back and forth in the data (to validate column width, I presume).

edited Mar 22 '19 at 21:45

Max Ghenis

14,783
16
84
132

answered Mar 03 '12 at 18:25

Dirk Eddelbuettel

360,940
56
644
725

Yeah I'm almost 70% sure you or JD have answered something similar. I actually got the `tempfile` idea from a previous answer of yours regarding zipped folders. But I can't find something w/ gzfile/gzcon, which seem to behave differently from some of the other file or connection related functions. – Adam Hyland Mar 03 '12 at 18:38
Can you distill this down? I actually used the same trick of 'streaming' out of a gzip`ed file way back in the early 1990s when disk space was scarcer and I kept simulation results gzip'ed. So the ability to transparently get the "gunzip" functionality into a C library fread is pretty old-school and standard. – Dirk Eddelbuettel Mar 03 '12 at 18:43
I'll mark this as the answer for now. I might come back and give myself a better answer after some fooling around w/ gzcon (which seems like the more promising angle). – Adam Hyland Mar 03 '12 at 22:52
Thanks for the `gzcon` hint. That helped to solve the riddle. – Dirk Eddelbuettel Mar 03 '12 at 23:00
Doesn't work with HTTPS:// links since `url()` does not support them. – akhmed May 26 '15 at 05:28
It does with `method="libcurl"` under R 3.2.0 if R is built that way. Else you can use `curl()` from the `curl` package as a drop-in replacement. See my `random` package. – Dirk Eddelbuettel May 26 '15 at 10:25
As of R version 3.4, gzcon has a new `text=TRUE` argument that can simplify the above by eliminating the need for textConnection. The gzcon can then be used as a textConnection: `dat <- read.csv(gzcon('filename.csv.gz',text=TRUE))` – bjerre Aug 31 '17 at 17:35
@MaxGhenis: I do not think that altering/editing somebody else's answer without stating so is a good idea. – Dirk Eddelbuettel Mar 22 '19 at 21:38
1

I undid the edit and added as a separate answer. The long URL in the answer here slowed me down, so thought this would be useful. – Max Ghenis Mar 22 '19 at 21:43
Are you sure noodling around answers that are 8 and 10 years old is the best use of our time? – Dirk Eddelbuettel Oct 16 '20 at 13:02
read.csv has issues parsing this example below. However zx8754 solution works. https://dumps.wikimedia.org/other/pageviews/2022/2022-06/pageviews-20220611-050000.gz – Scott Jun 11 '22 at 08:41

score 2 · Answer 2 · answered Oct 16 '20 at 06:33

Using data.table::fread :

x <- data.table::fread("http://dumps.wikimedia.org/other/articlefeedback/aa_combined-20110321.csv.gz")

dim(x)
[1] 1490   19

x[, 1:2]
#       aa_page_id page_namespace
#    1:   26224556              0
#    2:      31653              0
#    3:   26224556              0
#    4:   26224556              0
#    5:    1058990              0
#   ---                          
# 1486:     619464              0
# 1487:   19283361              0
# 1488:   19006979              0
# 1489:    5078775              0
# 1490:   30209619              0

score 1 · Answer 3 · answered Mar 22 '19 at 21:42

1

This function generalizes Dirk's answer:

R <- function(file_url) {
  con <- gzcon(url(file_url))
  txt <- readLines(con)
  return(read.csv(textConnection(txt)))
}

answered Mar 22 '19 at 21:42

Max Ghenis

14,783
16
84
132

Read gzipped csv directly from a url in R

3 Answers3

Linked

Related