4

I try to download a file, to get it from the server I need to send data at the same time. With curl on the command line it works fine:

curl "https://www.ishares.com/us/product-screener-download.dl" --data "productView=ishares&portfolios=239561-239855"

Unfortunately I don't get it to work with R. I tried with download.file, download.file with libcurl, curl_download and with httr. (download.file with curl or wget does not work as I'm on a window machine.)

What I tried and didn't work with curl:

library("curl")
handle <- new_handle()
handle_setopt(handle, customrequest = "POST")
handle_setform(handle, productView="ishares",portfolios="239561-239855")
curl_download("https://www.ishares.com/us/products/etf-product-list", "./data/ishares-us-etf.xls", handle=handle)

What I tried and didn't work with httr:

library(httr)
POST("https://www.ishares.com/us/products/etf-product-list", body = list(productView="ishares",portfolios="239561-239855"))
leo
  • 3,677
  • 7
  • 34
  • 46

3 Answers3

3

After findling a bit around with Fiddler I found out that I need to send the data with postfields and then everything works fine.

library("curl")
handle <- new_handle()
handle_setopt(handle, customrequest = "POST")
handle_setopt(handle, postfields='productView=ishares&portfolios=239561-239855')
curl_download("https://www.ishares.com/us/product-screener-download.dl", "./data/ishares-us-etf.xls", handle=handle)
leo
  • 3,677
  • 7
  • 34
  • 46
3

So you should use a right URL and encode = "form" in the httr::POST().

httr solution based on the @leo answer:

library(httr)
POST("https://www.ishares.com/us/product-screener-download.dl",
     body = list(productView = "ishares", portfolios = "239561-239855"),
     encode = "form", write_disk("/tmp/ishares-us-etf.xls"))
#> Response [https://www.ishares.com/us/product-screener-download.dl]
#>   Date: 2016-02-08 06:52
#>   Status: 200
#>   Content-Type: application/vnd.ms-excel;charset=UTF-8
#>   Size: 13.6 kB
#> <ON DISK>  /tmp/ishares-us-etf.xls
head(readLines(file_path), 5)
#>   [1] "<?xml version=\"1.0\"?>"
#>   [2] "<Workbook xmlns=\"urn:schemas-microsoft-com:office:spreadsheet\" xmlns:ss=\"urn:schemas-microsoft-com:office:spreadsheet\">"
#>   [3] "<Styles>"                          
#>   [4] "<Style ss:ID=\"Default\">"
#>   [5] "<Alignment Horizontal=\"Left\"/>"
Artem Klevtsov
  • 9,193
  • 6
  • 52
  • 57
-1

will this not do the job?

URL <- "https://www.ishares.com/us/products/etf-product-list"
values <- list(productView="ishares", portfolios="239561-239855")
POST(URL, body = values)
r <- GET(URL, query = values)
x <- content(r)
einar
  • 2,575
  • 1
  • 16
  • 14
  • thanks for your effort! Why should POST and then GET work? As I see both request will return a HTML error page, instead they should return an excel file (like the curl command on the bash). – leo Jan 19 '16 at 18:09