How do I scrape the text from a FTP server using Python?

Question

I am looking to extract all the information from this page:

I understand that requests lib wouldn't work for ftp, so I have resorted to using ftplib.

However, documentation seems to only explore the downloading of files in directories. How do I download this file without a "file type"

Thanks in advance.

what text ? ftp can give you only files or list of files in directores - if you see something more in browser or other program then it is extra information added by this program. Maybe add screenshot in question to show text which you heed — furas, Dec 19 '17 at 05:58
it is list of files in directory - you can get it using `ftp.retrlines('LIST')` see first [example in documentation](https://docs.python.org/3.6/library/ftplib.html) Web browser only reformat it in differen way. But you can't get the same result with ftplib. You could only use Selenium to control webb browser and then you could get text from browser. — furas, Dec 19 '17 at 06:09
If you want to download ftp://ftp.cmegroup.com/pub/settle/stleqt you download it with FTP. You don't really need Python or scraping for this. It's not clear what you mean with "file type" or why you think you need to know the file type, but this looks like just a text file. — tripleee, Dec 19 '17 at 06:13
it doesn't matter if file has extension or not. You can download it with Python using ftplib as any other file (maybe you should use binary mode instead of text mode). — furas, Dec 19 '17 at 06:15

score 1 · Answer 1 · answered Dec 19 '17 at 06:19

If you want to download a text file contents to memory, without using any temporary file, use retrlines like:

contents = ""
def collectLines(s):
    global contents
    contents += s + "\n"

ftp.retrlines("RETR " + filename, collectLines)

Or use an array:

lines = []
ftp.retrlines("RETR " + filename, lines.append)

1 Answers1