7

I've uploaded a .RData file (created using save()) to an ftp server, and I'm trying to use getURL() to download that same file. For all the examples and posts I've read, I can't seem to get this to work.

The .RData file was saved using:

save(results, file=RDataFilePath, compress="xz") #save object "results" w/ compression
#RDataFilePath is the location of the results.RData file on my harddrive

These data were uploaded using:

uploadURL <-"ftp://name:password@host/folder/results.RData" #name the url
ftpUpload(RDataFilePath, to=uploadURL, connecttimeout=120) #upload

This is how I try to download results.RData using getURL:

downloadURL <- "host/folder/results.RData"
load(getURL(downloadURL, userpwd="name:password", connecttimeout=120))

which gives the following error:

Error in curlPerform(curl = curl, .opts = opts, .encoding = .encoding) : 
  embedded nul in string: 'ý7zXZ'

When I paste the downloadURL string into my browser, the .RData file downloads immediately, so I know there isn't a typo. The error message suggests that the url can't get read b/c of the compression formatting; however, I get a similar error message when I use save() w/o compression.

I also get an error message when trying to download a .csv from the FTP:

read.csv(getURL(downloadURL1)) #downloadURL1 is similar to downloadURL, but points to the .csv file
Error in file(file, "rt") : cannot open the connection

and then a warning which states In addition: Warning message: In file(file, "rt") : cannot open file and then starts listing the contents of the .csv.

I've been trying to figure this out for the better part of the morning, and I feel like I must be missing something really basic. I'm guessing that I need to change some curl option so that it knows what type of file it is going to read. My syntax is probably a bit off, and I'm not using getURL correctly, but I'm not sure what I should be doing.

Any tips would be greatly appreciated.

p.s. My current approach is based on this Post

Community
  • 1
  • 1
rbatt
  • 4,677
  • 4
  • 23
  • 41

2 Answers2

9

You can try breaking it into two steps: first download the file, then load it.

download.file(downloadURL, "temp.rData")
load("temp.rData")

or sticking with rCurl you can try:

bin = getBinaryURL(downloadURL, ...yourOtherParams...) 
writeBin(bin, "temp.rData")  
load("temp.rData")
kith
  • 5,486
  • 1
  • 21
  • 21
  • 1
    This method works for the .csv file (I used `test=getURL( )` and `read.csv(text=test)` Thanks! However, I still get an error when trying to read the .RData file "Error in curlPerform(curl = curl, .opts = opts, .encoding = .encoding) : embedded nul in string: 'ý7zXZ'". When I try download.file, it says that it cannot open the URL. – rbatt Sep 16 '13 at 17:39
  • Are you on a unix system? If you have curl setup for command line use you can try something like system("curl ftp://xxxx temp.rData") from within R. If curl isn't working, maybe "wget" will work? – kith Sep 16 '13 at 18:20
  • I'm on OSX, my colleagues will be on Windows. I don't get very far with download.file, it won't connect to the server. I've been using the RCurl package. Using `getBinaryURL()` I can get in the file in binary form, but it's all gibberish and I can't figure out how to convert it. It's class is "raw", and save() then load() again does not convert it back to the original R object (or anything meaningful). – rbatt Sep 16 '13 at 18:52
  • This worked like a charm! I had just about given up... thank you so much! I searched for a solution to this for 6.5 hrs, and this is what got me there. I hope that other people trying to solve this problem find your answer :) It should also be noted that I did not have to pass any other special parameters to getBinaryURL, the defaults worked (except I had to pass my name:pass to `userpwd='). – rbatt Sep 16 '13 at 19:27
3

I spent quite a bit of time on this as well - hoping to use this in a Shiny app so I don't want to write to disk. 

library(RCurl)
url <- "ftp://F1World@aesius.ca/ALLF1Data.Rda"
userpwd <- "name:password"
bin = getBinaryURL(url, userpwd = userpwd, verbose = TRUE,
                   ftp.use.epsv = TRUE)

load(rawConnection(bin))

By Using rawConnection() I was able to avoid the write to disk step as it handled the RAW data type perfectly and avoided the error. FYI - This is my first post so I hope it's helpful

Wes Sauder
  • 39
  • 6
  • This scheme doesn't seem to work for me with a ".RData" file; R comes back with: `the input does not start with a magic number compatible with loading from a connection`, is this solution solely for a ".Rda" file? – Nick Aug 11 '21 at 18:53