resumable FTP folder upload with ftplib

Question

I'm new to FTP. I put together a Python script using ftplib to recursively upload a large number of folders and files to an FTP server with TLS encryption drawing on Upload folders from local system to FTP using Python script. The script so far works until the connection to the server times out, which happens frequently. The script also reconnects to the server when it encounters the errors listed below. However, I can't figure out how to resume the interrupted FTP upload exactly where it left off in the complex folder structure upon reconnecting. While I've found solutions for resumable FTP uploads for individual files (How can I resume interrupted FTP upload in Python), I can't figure out how to resume the interrupted upload of files and folders within a complex and deeply nested folder structure.

The script needs to resume exactly where it left off in the folder structure with the upload when reconnecting without stepping back through all the uploaded directories again. Is there an efficient way to do this? Here is what I have so far (server, path, etc. are placeholders).

Thanks in advance for your help!

import ftplib
import os
import ssl
import time

class ReusedSslSocket(ssl.SSLSocket):
    def unwrap(self):
        pass

class MyFTP_TLS(ftplib.FTP_TLS):
    """Explicit FTPS, with shared TLS session"""
    def ntransfercmd(self, cmd, rest=None):
        conn, size = ftplib.FTP.ntransfercmd(self, cmd, rest)
        if self._prot_p:
            conn = self.context.wrap_socket(conn,
                                            server_hostname=self.host,
                                            session=self.sock.session)  # reuses TLS session            
            conn.__class__ = ReusedSslSocket  # we should not close reused ssl socket when file transfers finish
        return conn, size

session = MyFTP_TLS(server, username, password, timeout=None)
session.prot_p()

def uploadFolder(path):
    files = os.listdir(path)
    os.chdir(path)
    for f in files:
        if os.path.isfile(path + r'\{}'.format(f)):
            fh = open(f, 'rb')
            session.storbinary('STOR %s' % f, fh)
            fh.close()
        elif os.path.isdir(path + r'\{}'.format(f)):
            session.mkd(f)
            session.cwd(f)
            uploadFolder(path + r'\{}'.format(f))
    session.cwd('..')
    os.chdir('..')

def reset_connection(pwd=password):
    print("Attempting FTP reconnect")
    try:
        session.quit()
    except (ConnectionResetError, WindowsError, OSError) as e:
        print(e)
    time.sleep(2)
    session.connect(server, timeout=None)
    session.login(username, password)
    session.prot_p()
    uploadFolder(path)
try:
     uploadFolder(path)
except (ConnectionResetError, WindowsError, OSError) as e:
    print(e)
    reset_connection(password)

score 0 · Accepted Answer · answered Oct 20 '21 at 00:33

Add a try except where you upload the file and if something goes wrong, reconnect and upload the file again. You will need a loop to keep retrying:

    while True:
        try:
            fh = open(f, 'rb')
            session.storbinary('STOR %s' % f, fh)
            fh.close()
            break
        except:
            print(f'Error while uploading {path}'

Handle specific errors instead of all of them. You may want to add a retry counter after which it just crashes.

An other solution would be to keep track of the full path of each file that got successfully uploaded in a list and save the list upon crashing. You can then read the list from the file and restart the process. Prior to uploading each file check if it is in the list. Upon starting and reading the list or exiting and successfully completing all uploads,delete the file.

Thanks, @Tarik! This worked. I got it to work by tracking the full paths and restarting the process where it left off every time the connection broke until it successfully completed all uploads as you suggested. — ionosphere, Oct 21 '21 at 22:55

resumable FTP folder upload with ftplib

1 Answers1