Downloading files with specific name from all directories on FTP server

Question

I would like to download all the txt files from a directory that is located an FTP server. The directory has a very complex structure, that consists in many sub-directories. Each subdirectory has a md5sum.txt file.

How can I write in script that recursively download the md5sum.txt from each sub-directory? To be more precise :

<> the directory is :
http://ftp.ebi.ac.uk/pub/databases/spot/eQTL/sumstats/

<> a sub-directory with a txt file :
http://ftp.ebi.ac.uk/pub/databases/spot/eQTL/sumstats/Alasoo_2018/exon/

score 1 · Answer 1 · answered May 29 '22 at 06:24

1

You should be able to use Python's os.walk to traverse this ftp server, although you may not be able to do it as you would a normal local directory. This GitHub repo implements os.walk for FTP servers. By what I understand, you can create a new object with the path of your FTP server, but you may have to modify Walk function to check if the file name is md5sum.txt, and if it is, then download it. You could use something like the python requests library to download the file.

answered May 29 '22 at 06:24

M B

2,700
2
15
20

You cannot *"use Python's `os.walk` to traverse this ftp server"*. You of course might use `os.walk`-like function. But not `os.walk`. – Martin Prikryl May 29 '22 at 09:42
Thank you. Is there a way please to do the same task by using shell scripts ? – Bogdan May 29 '22 at 20:01

score 1 · Answer 2 · answered May 29 '22 at 09:41

1

Start here: Downloading a directory tree with ftplib

Just add filter for your specific file name before the file download:

if file == "md5sum.txt":
    with open(os.path.join(destination,file),"wb") as f:
        ftp.retrbinary("RETR "+file, f.write)

answered May 29 '22 at 09:41

Martin Prikryl

188,800
56
490
992

Downloading files with specific name from all directories on FTP server

2 Answers2