0

Is it possible to load ftp files directly to the R workspace without downloading it? I have 700+ files each around 1.5 Gb and I want to extract approx 0.1 % of the information from every files and add them into a single dataframe.

I had a look at Download .RData and .csv files from FTP using RCurl (or any other method), could not get it to work.

Edit: After some reading, i managed to get the files into R

library(httr)
r <- GET("ftp://ftp.ais.dk/ais_data/aisdk_20141001.csv", write_memory())

when i try to read the body i use

content(r, "text")

but the output is gibberish. It might be because of the encoding, but how do i know which encoding the server uses. Any ideas on how to get the original data from the ftp?

Community
  • 1
  • 1
Jeppe Olsen
  • 968
  • 8
  • 19
  • the `curl` pkg and `httr` pkg (which uses the `curl` pkg) can "write to memory" vs disk. Time to read the help pages of those pkgs and experiment, esp since you've provided no code or URLs. – hrbrmstr Nov 16 '16 at 11:29

1 Answers1

3

I found a solution, which is very simple, but works nonetheless:

library(data.table)
r <- fread("ftp://ftp.ais.dk/ais_data/aisdk_20141001.csv")

This blog was helpfull

Jeppe Olsen
  • 968
  • 8
  • 19