Iterate through folders and subfolders to get to the files in each directory FTP Python

Question

Below I have included my current try at the code where I list the overall folder, run through files and download them. However, this method does not support iterating through subfolder and folder.

 def handleDownload(block):
    file.write(block)

ddir='U:/Test Folder'
filedestination = 'U:/SWEModelConstruction/UnmaskedData'
t1= []
t2= []
t3= []
os.chdir(ddir)
ftp = FTP('sidads.colorado.edu')
ftp.login()

print ('Logging in.')

directory = '/pub/DATASETS/NOAA/G02158/unmasked/'

print ('Changing to ' + directory)
ftp.cwd(directory)
ftp.retrlines('LIST')

print ('Accessing files')

filenames = ftp.nlst() # get filenames within the directory
print (filenames)

for filename in filenames:
    if filename not in ['.', '..']:
        #Parse values from filename to use in os.path.join
        for fname in filenames:
            t1 = fname[16:20]
            t2 = fname[20:22]
            t3 = fname[22:24]
            t4 = fname[16:24]

            if not t1: continue
        #use parsed values from filenamee to create folder and file pathss
        local_folder = os.path.join(filedestination, t1,t2,t3)
        local_filename = os.path.join(filedestination, t1,t2,t3,filename)
        local_dat = os.path.join(filedestination, t1,t2,t3,'zz_ssmv11034tS__T0001TTNATS'+t4+'05HP001.dat.gz')
        local_hdr = os.path.join(filedestination, t1,t2,t3,'zz_ssmv11034tS__T0001TTNATS'+t4+'05HP001.Hdr.gz')
        hdrfile = ('zz_ssmv11034tS__T0001TTNATS'+t4+'05HP001.Hdr')
        print (local_folder)
        print (local_filename)
        #check if folder for file exists and wether or not you have already it
        if os.path.exists(local_folder) and not os.path.isfile(local_filename):
            with open(local_filename, 'wb') as f_output:
                ftp.retrbinary('RETR '+ filename, f_output.write)
                ftp.quit()

do a function which calls itself recursively every time you find a directory — DevLounge, Apr 03 '17 at 20:12

score -1 · Answer 1 · answered Apr 03 '17 at 20:46

-1

I direct you to Alvin's Big Data Notebook. There you will find an example using pysftp and walktree. I do not know what FTP library you are using, but if you are using pysftp, this example should help. The key is understanding how to use walktree.

answered Apr 03 '17 at 20:46

aquil.abdullah

3,059
3
21
40

1

Do not post link-only answers + That article deals with SFTP (despite the title), so it's irrelevant - OP uses ftplib (FTP), not pysftp (SFTP) + Do not answer duplicate questions, vote to close them instead. – Martin Prikryl Apr 04 '17 at 06:28
@MartinPrikryl Do not tell me what to do! Just kidding, thank you for your advice, however, in the future you may want to consider offering with something like..."I don't know if you know this but you shouldn't do the following..." It will come across far more instructive. – aquil.abdullah Apr 04 '17 at 18:05

Iterate through folders and subfolders to get to the files in each directory FTP Python

1 Answers1