3

Newb to python and working with APIs. My source create an ftp url where they are dumping files on a daily basis and I would like to grab file to perform engineering + analysis. My problem is, how do I specify username and password to pull the csv? 

import pandas as pd
data = pd.read_csv('http://site-ftp.site.com/test/cat/filename.csv)

How do I include credentials to this? PS- url is fake for the sake of an example.

Martin Evans
  • 45,791
  • 17
  • 81
  • 97
user8464180
  • 33
  • 1
  • 3
  • Why don't you consider downloading the file first through a standard python interface before running it through a specific library/system. Consider this: https://stackoverflow.com/questions/11768214/python-download-a-file-over-an-ftp-server – J. Martin Aug 14 '17 at 21:10

1 Answers1

3

With older versions of Pandas, you can use something like requests.get() to download the CSV data into memory. Then you could use StringIO to make the data "file like" so that pd.read_csv() can read it in. This approach avoids having to first write the data to a file.

import requests
import pandas as pd
from io import StringIO

csv = requests.get("http://site-ftp.site.com/test/cat/filename.csv", auth=HTTPBasicAuth('user', 'password'))
data = pd.read_csv(StringIO(csv.text))

print(data)

From pandas 0.19.2, the pd.read_csv() function now lets you just pass the URL directly. For example:

data = pd.read_csv('http://site-ftp.site.com/test/cat/filename.csv')
Martin Evans
  • 45,791
  • 17
  • 81
  • 97
  • Worked! Thanks Martin! Code for python 3: import pandas as pd import requests from io import StringIO csv = requests.get("http://site-ftp.site.com/test/cat/filename.csv", auth=HTTPBasicAuth('user', 'passwprd')) data = pd.read_csv(StringIO(csv.text)) print(data) – user8464180 Aug 15 '17 at 15:14